Intelligent cutting plane selection method based on deep reinforcement learning
Through the intelligent cutting plane selection method based on deep reinforcement learning, the MILPs state information is captured using heterogeneous three-part graphs and deep learning models, the problem of insufficient cutting plane selection efficiency and stability in the existing technology is solved, and more efficient and stable MILPs solution is achieved.
Patent Information
- Application Number
- CN202411565139.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art fails to fully capture the mixed integer linear programming problem (MILPs) status information during the cutting plane selection process, and the sequence model ignores the position information of the input cutting plane sequence, resulting in insufficient solution efficiency and stability.
Using an intelligent cutting plane selection method based on deep reinforcement learning, the MILPs state is encoded into a heterogeneous three-part graph by constructing a neural network model, and the state information is captured using the heterogeneous graph Transformer and Attention mechanisms, ignoring the input order of the candidate cutting planes, and selecting the appropriate cutting plane sequence.
The performance and stability of MILPs solution are improved, significantly better than traditional heuristic algorithms and existing learning-based methods, especially in solving time and original-dual gap integrals.
Smart Images
Figure CN119514771A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of solving mixed integer linear programming problems, and in particular to an intelligent cutting plane selection method based on deep reinforcement learning. Background Art
[0002] Combinatorial optimization problems are a type of problem that involves finding the optimal object in a finite set of objects. Theories and methods related to combinatorial optimization problems have been widely developed in fields such as operations research, cybernetics, management science, and computer science, and have extremely important applications in many areas. Among different combinatorial optimization problems, mixed integer linear programming problems (MILPs) are a very important type. In actual production life, most combinatorial optimization problems can be equivalently converted into MILPs. With the continuous expansion of the scale of MILPs in the real world, how to solve MILPs accurately and quickly has become a meaningful research direction.
[0003] The core framework of modern advanced MILPs solvers such as Gurobi and SCIP is still generally the branch-and-bound algorithm. In order to further improve the solving efficiency, the solver usually integrates the cutting plane method into the branch-and-bound algorithm to become the branch-and-cut method. Therefore, how to add appropriate cutting planes to the subproblem has a significant impact on the solving efficiency of the solver. Unfortunately, the process of adding cutting planes still relies on a large number of manually designed heuristic algorithms, and heuristic methods often rely on specific problems or expert experience and cannot be applied to solving a wider range of problems. In addition, the most advanced learning-based methods formulate cutting plane selection as a sequence-to-sequence problem. However, the existing selection strategies formulated as sequence models have the following problems: (1) The model only captures the information of the cutting plane and ignores the importance of the MILPs state; (2) The sequence model extracts the potential position information of the input cutting plane sequence, which may affect the selection of the cutting plane. Therefore, how to more fully capture and utilize the potential state information of MILPs and ignore the influence of the position information of the input cutting plane sequence on the sequence model is a problem that current technology needs to solve. Summary of the invention
[0004] The technical problem to be solved by the present invention is to construct a more comprehensive MILPs state representation in the process of cutting plane selection and make the model ignore the potential position information of the input cutting plane sequence, thereby improving the solution performance of MILPs.
[0005] In order to achieve the above object, the present invention provides an intelligent cutting plane selection method based on deep reinforcement learning, comprising the following steps:
[0006] S1. Construct a neural network model and replace the cutting plane selection component of the mixed integer linear programming problem MILPs solver SCIP with the neural network model;
[0007] S2. Collect state information of mixed integer linear programming problems MILPs using solver SCIP;
[0008] S3. constructing the state information into a heterogeneous trigraph, and using the heterogeneous trigraph as an input of the neural network model to obtain a selected cutting plane sequence;
[0009] S4. Take the selected cutting plane sequence as action information and solve MILPs according to the selected cutting plane sequence to obtain reward information, and construct the state, action and reward information into a data set;
[0010] S5. Use the Actor-Critic algorithm to train the neural network model according to the data set, and finally obtain the trained neural network model as the cutting plane selection component of the solver SCIP;
[0011] Wherein, constructing a neural network model includes:
[0012] Construct an upper-layer policy network, which includes a heterogeneous graph Transformer module, an Attention module, and a multi-layer perceptron MLP module; the upper-layer policy network input first passes through the heterogeneous graph Transformer module, then passes through the Attention module, and finally passes through the MLP module to obtain the upper-layer policy network output;
[0013] Construct a lower-layer policy network, which includes a heterogeneous graph Transformer module and a decoder Decoder module of a pointer network; the lower-layer policy network input first passes through the heterogeneous graph Transformer module, and then passes through the decoder Decoder module of the pointer network to obtain the lower-layer policy network output.
[0014] Furthermore, the state information of the mixed integer linear programming problem MILPs in step S2 includes:
[0015] variable characteristics of the linear relaxation LP of the MILPs, the variable characteristics including regularized objective coefficient, variable type, whether the variable has an upper bound, whether the variable has a lower bound, regularized decreasing cost value, solution value, whether the solution value of the variable is within the upper bound, whether the solution value of the variable is within the lower bound, regularized age, and basis state;
[0016] Constraint characteristics of a linearly relaxed LP of MILPs, the constraint characteristics including constraint rank, number of regularized nonzero terms, regularized bias value, whether the constraint value is equal to the left side, whether the constraint value is equal to the right side, regularized dual solution, basis state, regularized age, number of LPs before regularization of the constraint was created, number of regularized integer columns, whether the active value of the constraint is always an integer in a feasible solution, whether the constraint is removable, and whether the constraint belongs to the current LP;
[0017] Candidate cut plane features of the linear relaxation LP of MILPs, the cut plane features including cut plane coefficient, target coefficient, similarity, validity, support, integer support and regularization violation.
[0018] Furthermore, step S3 constructs the state information into a heterogeneous trigraph, including:
[0019] Construct variable type, constraint type, and cutting plane type nodes as three Figure 3 Type nodes, where each variable of the linear relaxation LP is a variable type node, each constraint is a constraint type node, and each candidate cutting plane is a cutting plane type node;
[0020] Construct bidirectional directed edges between each constraint and the variables that appear in the constraint;
[0021] Construct bidirectional directed edges between each candidate cut plane and the variables that appear in the cut plane;
[0022] Build bidirectional directed edges between each candidate cut and each constraint.
[0023] Further, in step S5, the Actor-Critic algorithm is used to train the neural network model according to the data set, including:
[0024] Based on the policy gradient theorem, calculate the gradient of the objective function with respect to the upper policy network parameters
[0025]
[0026] Based on the policy gradient theorem, calculate the gradient of the objective function with respect to the underlying policy network parameters
[0027]
[0028] Among them, J([θ # ,θ2]) represents the objective function, E represents the mathematical expectation, represents the gradient, θ # represents the upper layer policy network parameters, θ2 represents the lower layer policy network parameters, s represents the MILPs state information, μ represents the initial state distribution, Represents the upper strategy network, ratio represents the ratio of the selected cutting plane, represents the lower layer policy network, a t represents the action selected by the policy network in state s, Q π (s,a t ) represents the action-value function;
[0029] Update the upper layer policy network parameters:
[0030]
[0031] Update the lower layer policy network parameters:
[0032]
[0033] Among them, α represents the learning rate.
[0034] The technical effects of the present invention are:
[0035] The present invention encodes the MILPs state into a heterogeneous trigraph and selects a heterogeneous graph network to fully utilize the heterogeneity, thereby more fully capturing the underlying information of MILPs and improving the performance of solving MILPs; the present invention also specifically improves the sequence model so that the sequence model can ignore the potential position information carried by the input order of the candidate cutting planes, thereby improving the stability when solving MILPs. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 A flowchart of a method for selecting an intelligent cutting plane based on deep reinforcement learning provided by an embodiment of the present invention;
[0037] Figure 2 The neural network model training process and neural network model architecture diagram provided by the embodiment of the present invention;
[0038] Figure 3 A schematic diagram of a heterogeneous trigraph provided by an embodiment of the present invention;
[0039] Figure 4 A schematic diagram of node relationships provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0040] The structural principle and working principle of the present invention are described in detail below in conjunction with the accompanying drawings and embodiments:
[0041] See also Figure 1 , Figure 1 This is a flow chart of a method according to an embodiment of the present invention. A method for selecting an intelligent cutting plane based on deep reinforcement learning in this embodiment includes the following steps:
[0042] Construct a neural network model and replace the cutting plane selection component of the mixed integer linear programming problem MILPs solver SCIP with the neural network model;
[0043] Specifically, see Figure 2 , the neural network model of the embodiment adopts a hierarchical reinforcement learning framework.
[0044] The upper policy network processes the input as follows: the input features encoded as heterogeneous trigraphs are first processed by the heterogeneous graph transformer, and the processed variable features and constraint features are discarded to obtain the candidate cutting plane features after message aggregation. Then, the attention mechanism is used to comprehensively consider the features of all candidate cutting planes. Finally, the mean and variance of the Gaussian distribution are obtained through the multi-layer perceptron MLP, and then the proportion of the selected cutting plane is obtained. The upper policy network can be formalized as:
[0045]
[0046] in, represents the upper layer policy network, θ represents the upper layer network parameters, tanh represents the hyperbolic tangent function, and They represent the mean and standard deviation of the output of the upper strategy network after inputting the state information s, The mean and standard deviation are and Gaussian distribution, K represents the result of random sampling from the Gaussian distribution.
[0047] The lower-layer policy network improves the traditional pointer network and adopts the heterogeneous graph Transformer as the Encoder part of the pointer network. The number of selected cutting planes is calculated according to the output of the upper-layer policy network, and the processed candidate cutting plane features are passed through the Decoder part of the pointer network to obtain the selected cutting plane sequence.
[0048] Use the SCIP solver to collect state information of mixed integer linear programming problems MILPs;
[0049] Specifically, the state information of MILPs consists of the variable features, constraint features, and candidate cutting plane features of the current linear relaxation LP.
[0050] Among them, the variable feature is a tensor with a dimension of n×17, where n represents the number of variables.
[0051] The variable characteristics include regularization target coefficient, variable type, whether the variable has an upper bound, whether the variable has a lower bound, regularization decreasing cost value, solution value, whether the solution value of the variable is in the upper bound, whether the solution value of the variable is in the lower bound, regularization age, and basis state.
[0052] The constraint feature is a tensor of dimension m×16, where m represents the number of primitive constraints.
[0053] Constraint features include constraint rank, number of regularized nonzero terms, regularized bias value, whether the constraint value is equal to the left side, whether the constraint value is equal to the right side, regularized dual solution, basis state, regularized age, number of LPs before the regularization constraint was created, number of regularized integer columns, whether the active value of the constraint is always an integer in a feasible solution, whether the constraint is removable, and whether the constraint belongs to the current LP.
[0054] The candidate cutting plane feature is a tensor of dimension c×13, where c represents the number of candidate cutting planes generated by the SCIP solver.
[0055] Cutting plane features include cutting plane coefficient, target coefficient, similarity, effectiveness, support, integer support, and regularization default.
[0056] The state information is constructed into a heterogeneous trigraph, and the heterogeneous trigraph is used as the input of the neural network model to obtain the selected cutting plane sequence;
[0057] Specifically, see Figure 3 ,In order to better characterize the state of MILPs and establish the intrinsic connection between constraints, variables and candidate cutting planes, the state is represented as a heterogeneous three-part graph.,Three types of nodes represent constraints, variables and candidate cutting planes, respectively.,The edges between nodes represent the relationship between constraints, variables and candidate cutting planes, and their eigenvalues are,the attention scores calculated by the attention mechanism.
[0058] In addition, the relationship between nodes is represented as a bidirectional directed graph, denoted as G = {V, E, A, R}, see Figure 4 , where V is the node set, E is the edge set, A represents the node type, and R represents the edge type. A directed edge e = (s, t) represents an edge from a source node s to a target node t. It is worth noting that the relationship from variables to constraints is not the same as the relationship from constraints to variables, and the relationship between other types of nodes also adopts a similar design.
[0059] The selected cutting plane sequence is used as action information and MILPs is solved according to the selected cutting plane sequence to obtain reward information, and the state, action and reward information are constructed as a data set;
[0060] Specifically, the SCIP solver encodes the extracted MILPs state information into a heterogeneous trigraph as the input of the neural network model. The neural network model obtains a return result after calculation. The SCIP solver selects the corresponding cutting plane sequence according to the return result and adds it to the LP relaxation, and then continues the subsequent steps of solving the MILPs. After the solution is completed, the reward information is obtained. In this embodiment, the reward information is set to the primal-dual integral obtained by the solution to guide the training process of the neural network model.
[0061] The Actor-Critic algorithm is used to train the neural network model according to the data set, and finally the trained neural network model is used as the cutting plane selection component of the SCIP solver.
[0062] Specifically, the neural network model parameter update process includes:
[0063] Based on the policy gradient theorem, calculate the gradient of the objective function with respect to the upper policy network parameters:
[0064]
[0065] Based on the policy gradient theorem, the gradient of the objective function with respect to the underlying policy network parameters is calculated:
[0066]
[0067] Among them, θ # represents the upper layer policy network parameters, θ2 represents the lower layer policy network parameters, s represents the MILPs state information, μ represents the initial state distribution, Represents the upper strategy network, ratio represents the ratio of the selected cutting plane, represents the lower layer policy network, a t represents the action selected by the policy network in state s;
[0068] Update the upper layer policy network parameters:
[0069]
[0070] Update the lower layer policy network parameters:
[0071]
[0072] Among them, α represents the learning rate.
[0073] In this embodiment, two groups of tests are conducted, and the embodiment is compared with the heuristic cutting plane selection strategy and the most advanced learning-based cutting plane selection strategy in terms of performance and stability in solving MILPs on six data sets to analyze the effect of the cutting plane selection method of the present invention.
[0074] Test 1: The performance of the embodiment, the heuristic cutting plane selection strategy, and the most advanced learning-based cutting plane selection strategy for solving MILPs is evaluated on six datasets. Different cutting plane selection strategies are used to solve the MILPs on the six datasets, and the arithmetic mean of the solution time of the MILPs on each dataset and the primal-dual integral are obtained for comparison.
[0075] Under the time limit T, the primal-dual integral calculation formula is:
[0076]
[0077] Where c is the coefficient vector of the objective function, is the best feasible solution found at time t, is the best dual bound for time t.
[0078] The time and primal-dual integral comparison results of solving MILPs using the above cutting plane selection strategy are shown in Tables 1 and 2:
[0079] Table 1
[0080]
[0081] Table 2
[0082]
[0083] Specifically, Nocuts, Default, Random, Nvscore, and Effscore represent five heuristic cutting plane selection strategies in the SCIP solver, HEM represents the most advanced learning-based cutting plane selection strategy, and HGTSM represents this embodiment. MIK, CORLAT, Load Balancing, Anonymous, MIPLIB mixed neos, and MIPLIB mixedsupportcase represent six types of mixed integer linear programming problems of different difficulty levels.
[0084] The data in Table 1 and Table 2 show that the embodiment is significantly better than other cutting plane selection strategies on six data sets. In particular, the embodiment has a significant advantage over the heuristic cutting plane selection strategy in the primal-dual gap integral. In addition, the embodiment reduces the solution time for several data sets. Even compared with the most advanced learning-based methods, the embodiment still maintains a significant advantage. Test 2: The stability of the embodiment and the most advanced learning-based cutting plane selection strategy in solving MILPs is evaluated on several data sets. Different cutting plane selection strategies are used to solve the MILPs on several data sets, and the solution stability index and stability improvement of the MILPs on each data set are obtained.
[0085] Stability calculation formula:
[0086]
[0087] Stability improvement calculation formula:
[0088]
[0089] S(·) indicates the stability of the method on a given dataset.
[0090] Where IR is the set of statistics for each MILP in the dataset after shuffling the order of the input cutting planes using the specified random seed, and IR(p) is the statistics for a specific problem p. ID is the set of statistics after the default cutting plane sequence generated by the SCIP solver is input into the model, and ID(p) is the statistics for a specific problem p. D is the specified MILP dataset. N is the number of problems in the dataset.
[0091] The stability comparison data results obtained by solving MILPs using the above cutting plane selection strategy are shown in Tables 3 and 4:
[0092] Table 3
[0093]
[0094] Table 4
[0095]
[0096] The data in Tables 3 and 4 show that the embodiments show significant stability improvements on multiple datasets compared to the state-of-the-art learning-based cutting plane selection strategies. Notably, on the CORLAT dataset, the improvements in solution time and primal-dual gap integral are both over 80%.
[0097] The content of the present invention is not limited to the above-mentioned specific embodiments. Without departing from the basic idea and core spirit of the present invention, those skilled in the art can make various modifications, replacements or improvements to the specific structure, form, steps or methods of the present invention according to their professional knowledge and actual needs. These modifications and deformations, whether partial or overall changes, should not affect the substantive content of the present invention and should be included in the scope of protection of the claims of the present invention. Therefore, the present invention is not limited to the specific examples described above, but covers all implementation methods and technical variations that conform to the idea of the present invention.
Claims
1. An intelligent cutting plane selection method based on deep reinforcement learning, characterized in that: The following steps are involved: S1. Construct a neural network model and replace the cutting plane selection component of the mixed integer linear programming problem MILPs solver SCIP with the neural network model; S2. Collect state information of mixed integer linear programming problems MILPs using solver SCIP; S3. constructing the state information into a heterogeneous trigraph, and using the heterogeneous trigraph as an input of the neural network model to obtain a selected cutting plane sequence; S4. Take the selected cutting plane sequence as action information and solve MILPs according to the selected cutting plane sequence to obtain reward information, and construct the state, action and reward information into a data set; S5. Using the Actor-Critic algorithm to train the neural network model according to the data set, and finally obtaining the trained neural network model as the cutting plane selection component of the solver SCIP; wherein, the construction of the neural network model includes: Construct an upper-layer policy network, which includes a heterogeneous graph Transformer module, an Attention module, and a multi-layer perceptron MLP module; the upper-layer policy network input first passes through the heterogeneous graph Transformer module, then passes through the Attention module, and finally passes through the MLP module to obtain the upper-layer policy network output; Construct a lower-layer policy network, which includes a heterogeneous graph Transformer module and a decoder Decoder module of a pointer network; the lower-layer policy network input first passes through the heterogeneous graph Transformer module, and then passes through the decoder Decoder module of the pointer network to obtain the lower-layer policy network output.
2. The intelligent cutting plane selection method based on deep reinforcement learning according to claim 1 is characterized in that: The state information of the mixed integer linear programming problem MILPs in step S2 includes: variable characteristics of the linear relaxation LP of the MILPs, the variable characteristics including regularized objective coefficient, variable type, whether the variable has an upper bound, whether the variable has a lower bound, regularized decreasing cost value, solution value, whether the solution value of the variable is within the upper bound, whether the solution value of the variable is within the lower bound, regularized age, and basis state; Constraint characteristics of a linearly relaxed LP of MILPs, the constraint characteristics including constraint rank, number of regularized nonzero terms, regularized bias value, whether the constraint value is equal to the left side, whether the constraint value is equal to the right side, regularized dual solution, basis state, regularized age, number of LPs before regularization of the constraint was created, number of regularized integer columns, whether the active value of the constraint is always an integer in a feasible solution, whether the constraint is removable, and whether the constraint belongs to the current LP; Candidate cut plane features of the linear relaxation LP of MILPs, the cut plane features including cut plane coefficient, target coefficient, similarity, validity, support, integer support and regularization violation.
3. The intelligent cutting plane selection method based on deep reinforcement learning according to claim 1 is characterized in that: Step S3 constructs the state information into a heterogeneous trigraph, including: Construct variable type, constraint type and cutting plane type nodes as three types of nodes in the trigraph, where each variable of the linear relaxation LP is a variable type node, each constraint is a constraint type node, and each candidate cutting plane is a cutting plane type node; construct bidirectional directed edges between each constraint and the variable appearing in the constraint; Construct bidirectional directed edges between each candidate cut plane and the variables that appear in the cut plane; Build bidirectional directed edges between each candidate cut and each constraint.
4. The intelligent cutting plane selection method based on deep reinforcement learning according to claim 1 is characterized in that: Step S5 uses the Actor-Critic algorithm to train the neural network model according to the data set, including: Based on the policy gradient theorem, calculate the gradient of the objective function with respect to the upper policy network parameters Based on the policy gradient theorem, calculate the gradient of the objective function with respect to the underlying policy network parameters Among them, J([θ # ,θ2]) represents the objective function, E represents the mathematical expectation, represents the gradient, θ # represents the upper layer policy network parameters, θ2 represents the lower layer policy network parameters, s represents the MILPs state information, μ represents the initial state distribution, Represents the upper strategy network, ratio represents the ratio of the selected cutting plane, represents the lower layer policy network, a t represents the action selected by the policy network in state s, Q π (s,a t ) represents the action-value function; Update the upper layer policy network parameters: Update the lower layer policy network parameters: Among them, α represents the learning rate.