A single-rolling unit hot rolling scheduling method, system, storage medium and computer device based on a DQN algorithm

Abstract

The application discloses a single-rolling unit hot rolling scheduling method based on a DQN algorithm, and comprises the following steps: S1, standardizing existing hot rolling planning data, designing hot rolling scheduling states, hot rolling scheduling actions and hot rolling scheduling rewards based on a reinforcement learning method, designing the structure of a neural network, and forming a deep Q learning framework; S2, based on the deep Q learning framework, designing a neural network training algorithm, and training the neural network Q net using the standardized hot rolling planning data; and S3, based on to-be-rolled slab data, applying the neural network Q net to compile and form a hot rolling plan. The application is aimed at a hot rolling production data set, aimed at a hot rolling plan scheduling problem in an actual hot rolling production process, trains a neural network based on a deep Q learning method and actual hot rolling planning data, and further attempts to compile a hot rolling production plan based on the trained network.

Description

Technical Field

[0001] This invention relates to information technology for hot rolling production, and more specifically, to a method, system, storage medium, and computer equipment for hot rolling scheduling in a single rolling unit based on the DQN algorithm. Background Technology

[0002] Hot rolling is a crucial metal processing technology, providing essential material support for various industrial applications. The scheduling of hot rolling projects is a critical factor affecting steel mill production costs and strip quality. A complete rolling unit consists of hot-rolled material and main material, with the rolling width of the main material varying from large to small. Furthermore, it's essential to ensure smooth transitions in rolling width, thickness, and hardness between adjacent slabs. The hot-rolled material portion is a small part of the hot rolling plan, and schedulers can usually determine it relatively easily. The quality of the hot rolling plan primarily depends on the rolling sequence of the slabs in the main material portion. The hot rolling plan discussed in this patent application represents the main material portion of the plan. Hot rolling scheduling refers to determining the rolling sequence of slabs to form the hot rolling plan, which is a combinatorial optimization problem and also an NP-hard problem. Classical methods for solving hot rolling scheduling problems include mathematical programming-based methods and heuristic methods such as intelligent optimization. However, mathematical programming-based methods often struggle to include all important factors affecting production, or the solution time increases dramatically with the problem size; intelligent optimization methods do not guarantee finding the optimal solution.

[0003] Deep Q-learning is an algorithm that combines reinforcement learning and neural networks. Reinforcement learning primarily studies how to enable an agent to learn optimal policies through interaction with the environment, thereby maximizing future rewards. In hot rolling scheduling, the agent learns the optimal policy by trying different rolling sequences of slabs, aiming to create an optimal hot rolling plan. Neural networks, as powerful function approximators, are ideal tools for fitting the optimal action-value function in deep Q-learning algorithms. Deep reinforcement learning can handle high-dimensional input data, learn optimal policies directly from raw data, has good generalization ability, can effectively balance exploration and exploitation, and can handle continuous and discrete action spaces. Furthermore, while an increasing number of researchers have proposed solutions to combinatorial optimization problems such as production operations and scheduling based on deep reinforcement learning methods, there is currently little research on solving the combinatorial optimization problem of hot rolling scheduling using deep reinforcement learning methods. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a hot rolling scheduling method, system, storage medium, and computer equipment for a single rolling unit based on the DQN algorithm. This invention addresses the hot rolling production dataset and the hot rolling scheduling problem in the actual hot rolling production process by training a neural network based on the deep Q-learning method and actual hot rolling plans. Furthermore, it attempts to compile a hot rolling production plan based on the trained network.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The first aspect of this invention provides a hot rolling scheduling method for a single rolling unit based on the DQN algorithm, comprising the following steps:

[0007] S1. Standardize the existing hot rolling plan preparation data, design hot rolling scheduling status, hot rolling scheduling actions and hot rolling scheduling rewards based on reinforcement learning methods, design the structure of the neural network, and form a deep Q-learning framework.

[0008] S2, Based on the aforementioned deep Q-learning framework, a training algorithm for the neural network is designed, and the neural network Q is trained using standardized hot rolling schedule data. net ;

[0009] S3, based on the data of the slab to be rolled, apply the aforementioned neural network Q. net Compile and formulate a hot rolling plan.

[0010] Preferably, in step S1, the existing hot rolling plan compilation data is standardized, including the standardization of the slab rolling width, rolling thickness, and hardness, and the calculation formula is as follows:

[0011]

[0012] Where, φ t (1) φ t (2) φ t (3) These represent the standardized rolling width, rolling thickness, and hardness, respectively; w t ,g t ,h t These represent the rolling width, rolling thickness, and hardness at time step t, respectively; w max ,g max ,h max These represent the maximum values ​​of rolling width, rolling thickness, and hardness in the entire dataset, respectively.

[0013] Preferably, in step S1, the hot rolling scheduling state s t ∈S is defined as all sequentially compiled slab sequences at time step t;

[0014] Where S represents the set of hot rolling schedule states.

[0015] Preferably, in step S1, the hot rolling scheduling action a t ∈A t It is defined as selecting a slab from those that have never been compiled and including it in the hot rolling plan;

[0016] Among them, A t This represents the set of hot rolling scheduling actions that can be selected at the current time t.

[0017] Preferably, in step S1, a hot rolling scheduling action a is performed. t The immediate reward after r t Defined as the opposite of the penalty for violating regulations;

[0018] At the current time t, the reward function for the jump in rolling width in the hot rolling plan is as follows:

[0019]

[0020] in, λ represents the immediate reward for a jump in the rolling width of the hot rolling plan; λ represents the penalty weight for a rebound in the rolling width.

[0021] The hot rolling scheduling bonuses for jumps in rolling thickness and hardness are as follows:

[0022]

[0023] in, These represent the immediate rewards for the jumps in rolling thickness and hardness at time step t, respectively; avg indicates that the data within the parentheses are averaged.

[0024] Instant total reward r at time step t t for:

[0025]

[0026] Where, ω W ω T ω H These represent the weights of the impact of the jump in rolling width, rolling thickness, and hardness on the immediate reward, respectively.

[0027] The total reward r for a hot rolling project is:

[0028]

[0029] Preferably, in step S2, the neural network has 3 nodes in the input layer, 4 fully connected hidden layers with 32 nodes in each layer, all nodes use ReLU activation functions, and the output layer has 1 node.

[0030] Preferably, the training algorithm for the neural network specifically includes the following steps:

[0031] S21, Initialization, capacity is N E The experience replay pool D, where all elements are null, is used to randomly generate the weights θ of the action value network Q, and the target action value network is also randomly generated. weight θ - Time step t = 1;

[0032] S22, let the selectable action space A t ={C0,C1,…,C n-1};

[0033] S23, if the action space A t If empty, proceed to step S22; otherwise, proceed to step S24.

[0034] S24. Randomly generate a real number ρ in the interval [0,1]. If ρ < ε, then randomly select an action a. t If not, then use the action-value network to select a. t =argmax a Q(φ t ,a;θ), that is, select the action a with the largest predicted Q value based on the output of the action value network. t Where a∈A t , φ t Characteristics representing the current state;

[0035] S25, Execute action a t Receive instant reward r t and the next state s t+1 ; Calculate s t+1 Feature φ t+1 From A t Remove a from t ;

[0036] S26, the empirical tuple (φ t ,a t ,r t ,φ t+1 ) is stored in the experience replay pool D, and sampled experience tuples (φ) with replacement are retrieved from the experience replay pool. j ,a j ,r j ,φ j+1 ), where j is the index of the sampled empirical tuple;

[0037] S27, if A t If it is empty, then y j =r j If not, then Among them, yj Indicates the target Q value;

[0038] S28, for the loss function ∑(y j -Q(φ j ,a j ;θ)) 2 Update the parameters θ of the action value neural network using the stochastic gradient descent algorithm; time step t = t + 1; if t % L = 0, then let θ - =θ, where "%" represents the remainder and L represents the interval for updating network parameters of the target action value;

[0039] S29. If time step t reaches the specified termination value, the algorithm terminates; otherwise, proceed to step S23.

[0040] Preferably, step S3 specifically includes the following steps:

[0041] S31, select one slab from n slabs as the first slab in the hot rolling plan;

[0042] S32, using the first slab of the hot-rolled plan as the current state, calculate the state characteristics;

[0043] S33, based on the current state characteristics, use the already trained neural network Q... net A hot rolling plan is prepared for each of the n slabs.

[0044] Preferably, step S31 specifically includes the following steps:

[0045] S311, Let set M be the set of n slabs. Select the batch of slabs with the largest rolling width from set M to form set M. W ;

[0046] S312, from set M W A batch of slabs with a rolling thickness equal to the mode was selected and formed into set M. T ;

[0047] S313, from set M T A batch of slabs with hardness equal to the mode was selected from the samples, forming set M. H ;

[0048] S314, choose M arbitrarily H One of the slabs in the process served as the starting point for the hot rolling program.

[0049] A second aspect of the present invention provides a hot rolling scheduling system for a single rolling unit based on the DQN algorithm, comprising:

[0050] The environment module describes the hot rolling scheduling process in the form of a Markov decision process, and includes the status, actions and rewards of the hot rolling schedule.

[0051] The intelligent agent module is used to interact with the environment to generate experience tuples, which ultimately realize the parameter update of the neural network, including the experience replay pool and the neural network.

[0052] Application module, using the Q-neuron trained by the application agent net The slab to be rolled is planned and scheduled to form a hot rolling plan;

[0053] The single-rolling-unit hot rolling scheduling system implements the single-rolling-unit hot rolling scheduling method based on the DQN algorithm provided in the first aspect of the present invention.

[0054] A third aspect of the present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the hot rolling scheduling method for a single rolling unit based on the DQN algorithm provided in the first aspect of the present invention.

[0055] A fourth aspect of the present invention provides a computer device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the hot rolling scheduling method for a single rolling unit based on the DQN algorithm provided in the first aspect of the present invention.

[0056] This invention provides a method, system, storage medium, and computer equipment for hot rolling scheduling in a single rolling unit based on the DQN algorithm. For the hot rolling scheduling process, it first standardizes existing hot rolling plan compilation data, designs hot rolling scheduling states, actions, and rewards based on reinforcement learning methods, designs the structure of a neural network, and forms a deep Q-learning framework. Based on the above deep Q-learning framework, it designs a training algorithm for the neural network and trains the neural network using the standardized hot rolling plan compilation data. net Based on the data of the slab to be rolled, a neural network Q is applied. net This invention allows for the creation of a hot rolling plan, forming a single-unit hot rolling plan. It not only makes hot rolling production scheduling more intelligent but also enables faster hot rolling plan creation compared to traditional methods, while maintaining the quality of the hot rolling plan. Attached Figure Description

[0057] Figure 1 This is a schematic flowchart of the hot rolling scheduling method for a single rolling unit according to the present invention;

[0058] Figure 2 This is a schematic diagram of the basic process of training the deep Q-network in the hot rolling scheduling method of a single rolling unit of the present invention;

[0059] Figure 3 This invention relates to a neural network Q used for generating hot rolling plans in a single-rolling-unit hot rolling scheduling method. net A schematic diagram illustrating the change of the loss function during training;

[0060] Figure 4 This is a schematic diagram of the structural framework of the hot rolling scheduling system for a single rolling unit according to the present invention. Detailed Implementation

[0061] To better understand the above-mentioned technical solutions of the present invention, the technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0062] Combination Figure 1 As shown, this invention provides a hot rolling scheduling method for a single rolling unit based on the DQN algorithm, comprising the following steps:

[0063] S1, for the hot rolling scheduling process, standardize the existing hot rolling plan preparation data, design hot rolling scheduling status, hot rolling scheduling action and hot rolling scheduling reward based on reinforcement learning method, design the structure of neural network, and form a deep Q learning framework;

[0064] S2, based on the deep Q-learning framework, designs a training algorithm for a neural network, using standardized hot rolling schedule data to train the neural network Q. net ;

[0065] S3, based on the data of the slab to be rolled, applies a neural network Q... net Compile and formulate a hot rolling plan.

[0066] In step S1 above, the existing hot rolling plan compilation data is standardized, including the standardization of the slab rolling width, rolling thickness, and hardness. The calculation formula is as follows:

[0067]

[0068] Where, φ t (1) φ t (2) φ t (3) These represent the standardized rolling width, rolling thickness, and hardness, respectively; w t ,g t ,h t These represent the rolling width, rolling thickness, and hardness at time step t, respectively; w max ,g max ,h max These represent the maximum values ​​of rolling width, rolling thickness, and hardness in the entire dataset, respectively.

[0069] In step S1 above, the hot rolling scheduling state s t∈S is defined as all sequentially compiled slab sequences at time step t;

[0070] Where S represents the set of hot rolling scheduling states; for example, s0 = Φ represents the initial state where no slabs have been scheduled; s1 = (C3) represents the state after C3 is selected as the first slab; s2 = (C3, C0) represents the state after slab C3 is scheduled and C0 is scheduled, and so on.

[0071] In step S1 above, the hot rolling scheduling action a t ∈A t It is defined as selecting a slab from those that have never been compiled and including it in the hot rolling plan;

[0072] Among them, A t This represents the set of selectable hot rolling scheduling actions at the current time t. The action space in hot rolling scheduling is denoted as A = {C0, C1, ..., C}. n-1 Let set A contain all the slabs to be fabricated. Assume the state at time step t is s. t If (C0, C1, C2, C3) is a given set of actions, then the set of actions that can be chosen is A. t ={C4,C5,…,C n-1}

[0073] In step S1 above, a hot rolling scheduling action a is performed. t The immediate reward after r t Defined as the negative of the penalty for violating regulations, the factors considered are the slab's rolling width, rolling thickness, and hardness. At the current time t, the reward function for the hot rolling plan's rolling width jump is as follows:

[0074]

[0075] in, λ represents the immediate reward for a jump in the rolling width of the hot rolling plan; λ represents the penalty weight for a rebound in the rolling width.

[0076] The hot rolling scheduling bonuses for jumps in rolling thickness and hardness are as follows:

[0077]

[0078] in, These represent the immediate rewards for the jumps in rolling thickness and hardness at time step t, respectively; avg indicates that the data within the parentheses are averaged.

[0079] Instant total reward r at time step t t for:

[0080]

[0081] Where, ωW ω T ω H These represent the weights of the impact of the jump in rolling width, rolling thickness, and hardness on the immediate reward, respectively.

[0082] The total reward r for a hot rolling project is:

[0083]

[0084] In step S2 above, the neural network has 3 nodes in the input layer, 4 fully connected hidden layers with 32 nodes in each layer, all nodes use the ReLU activation function, and the output layer has 1 node.

[0085] Combination Figure 2 As shown, in step S2 above, the neural network training algorithm specifically includes the following steps:

[0086] S21, Initialization, capacity is N E The experience replay pool D, where all elements are null, is used to randomly generate the weights θ of the action value network Q, and the target action value network is also randomly generated. weight θ - Time step t = 1;

[0087] S22, let the selectable action space A t ={C0,C1,…,C n-1};

[0088] S23, if the action space A t If empty, proceed to step S22; otherwise, proceed to step S24.

[0089] S24. Randomly generate a real number ρ in the interval [0,1]. If ρ < ε, then randomly select an action a. t If not, then use the action-value network to select a. t =argmax a Q(φ t ,a;θ), that is, select the action a with the largest predicted Q value based on the output of the action value network. t Where a∈A t , φ t Characteristics representing the current state;

[0090] S25, Execute action a t Receive instant reward r t and the next state s t+1 ; Calculate s t+1 Feature φ t+1 From A t Remove a from t ;

[0091] S26, the empirical tuple (φ t ,a t ,r t ,φ t+1 ) is stored in the experience replay pool D, and sampled experience tuples (φ) with replacement are retrieved from the experience replay pool. j ,a j ,r j ,φ j+1 ), where j is the index of the sampled empirical tuple;

[0092] S27, if A t If it is empty, then y j =r j If not, then Among them, y j Indicates the target Q value;

[0093] S28, for the loss function ∑(y j -Q(φ j ,a j ;θ)) 2 Update the parameters θ of the action value neural network using the stochastic gradient descent algorithm; time step t = t + 1; if t % L = 0, then let θ - =θ, where "%" represents the remainder and L represents the interval for updating network parameters of the target action value;

[0094] S29. If time step t reaches the specified termination value, the algorithm terminates; otherwise, proceed to step S23.

[0095] Step S3 above specifically includes the following steps:

[0096] S31, from n slabs (set M), select one slab as the first slab in the hot rolling plan;

[0097] S32, using the first slab of the hot-rolled plan as the current state, calculate the state characteristics;

[0098] S33, based on the current state characteristics, use the already trained neural network Q... net A hot rolling plan is generated by compiling a hot rolling plan for the slabs in set M.

[0099] Step S31 above specifically includes the following steps:

[0100] S311, Let set M be the set of n slabs. Select the batch of slabs with the largest rolling width from set M to form set M. W ;

[0101] S312, from set M W A batch of slabs with a rolling thickness equal to the mode was selected and formed into set M.T ;

[0102] S313, from set M T A batch of slabs with hardness equal to the mode was selected from the samples, forming set M. H ;

[0103] S314, choose M arbitrarily H One of the slabs in the process served as the starting point for the hot rolling program.

[0104] Combination Figure 4 As shown, the present invention also provides a hot rolling scheduling system for a single rolling unit based on the DQN algorithm, comprising:

[0105] The environment module describes the hot rolling scheduling process in the form of a Markov decision process, and includes the status, actions and rewards of the hot rolling schedule.

[0106] The intelligent agent module is used to interact with the environment to generate experience tuples, which ultimately realize the parameter update of the neural network, including the experience replay pool and the neural network.

[0107] Application module, using the Q-neuron trained by the application agent net The slab to be rolled is planned and scheduled to form a hot rolling plan;

[0108] The hot rolling scheduling system of this invention implements the hot rolling scheduling method of this invention.

[0109] Based on the above, Figure 1 The hot rolling scheduling method for a single rolling unit shown in the figure also includes a storage medium storing a computer program, which, when executed by a processor, implements the above-described method. Figure 1 The hot rolling scheduling method for a single rolling unit of the present invention is shown.

[0110] Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0111] Based on the above, Figure 1 To achieve the above objectives, this embodiment of the invention also provides a computer device, specifically a personal computer, server, network device, etc. This computer device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figure 1The hot rolling scheduling method for a single rolling unit of the present invention is shown.

[0112] The computer device may also include a user interface, network interface, camera, radio frequency (RF) circuitry, sensors, audio circuitry, Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB ports, card reader ports, etc. Optional network interfaces may include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0113] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0114] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of a computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software within the physical device.

[0115] Example

[0116] This embodiment provides a hot rolling scheduling method for a single rolling unit based on DQN, such as... Figure 1 As shown, the method includes:

[0117] S1, for the hot rolling scheduling process, standardize the existing hot rolling plan preparation data, design hot rolling scheduling status, hot rolling scheduling action and hot rolling scheduling reward based on reinforcement learning method, design the structure of neural network, and form a deep Q learning framework;

[0118] S2, based on the deep Q-learning framework, designs a training algorithm for a neural network, using standardized hot rolling schedule data to train the neural network Q. net ;

[0119] S3, based on the data of the slab to be rolled, applies a neural network Q... net Compile and formulate a hot rolling plan.

[0120] In step S1 above, the existing hot rolling plan compilation data is standardized, including the standardization of the slab rolling width, rolling thickness, and hardness. The calculation formula is as follows:

[0121]

[0122]

[0123] Where, φ t (1) φ t (2) φ t (3) These represent the standardized rolling width, rolling thickness, and hardness, respectively; w t ,g t ,h t These represent the rolling width, rolling thickness, and hardness at time step t, respectively; w max ,g max ,h max These represent the maximum values ​​of rolling width, rolling thickness, and hardness in the entire dataset, respectively.

[0124] In step S1 above, the hot rolling scheduling state s t ∈S is defined as all sequentially compiled slab sequences at time step t;

[0125] Where S represents the set of hot rolling scheduling states; for example, s0 = Φ represents the initial state where no slabs have been scheduled; s1 = (C3) represents the state after C3 is selected as the first slab; s2 = (C3, C0) represents the state after slab C3 is scheduled and C0 is scheduled, and so on.

[0126] In step S1 above, the hot rolling scheduling action a t ∈A t It is defined as selecting a slab from those that have never been compiled and including it in the hot rolling plan;

[0127] Among them, A t This represents the set of selectable hot rolling scheduling actions at the current time t. The action space in hot rolling scheduling is denoted as A = {C0, C1, ..., C}. n-1 Let set A contain all the slabs to be fabricated. Assume the state at time step t is s. t If (C0, C1, C2, C3) is a given set of actions, then the set of actions that can be chosen is A. t ={C4,C5,…,C n-1}

[0128] In step S1 above, a hot rolling scheduling action a is performed. t The immediate reward after r t Defined as the negative of the penalty for violating regulations, the factors considered are the slab's rolling width, rolling thickness, and hardness. At the current time t, the reward function for the hot rolling plan's rolling width jump is as follows:

[0129]

[0130] in, λ represents the immediate reward for a jump in the rolling width of the hot rolling plan; λ represents the penalty weight for a rebound in the rolling width.

[0131] The hot rolling scheduling bonuses for jumps in rolling thickness and hardness are as follows:

[0132]

[0133] in, These represent the immediate rewards for the jumps in rolling thickness and hardness at time step t, respectively; avg indicates that the data within the parentheses are averaged.

[0134] Instant total reward r at time step t t for:

[0135]

[0136] Where, ω W ω T ω H These represent the weights of the impact of the jump in rolling width, rolling thickness, and hardness on the immediate reward, respectively.

[0137] The total reward r for a hot rolling project is:

[0138]

[0139] In step S2 above, the neural network has 3 nodes in the input layer, 4 fully connected hidden layers with 32 nodes in each layer, all nodes use the ReLU activation function, and the output layer has 1 node.

[0140] like Figure 2 As shown, in step S2 above, the neural network training algorithm specifically includes the following steps:

[0141] S21, Initialization, capacity is N E The experience replay pool D, where all elements are null, is used to randomly generate the weights θ of the action value network Q, and the target action value network is also randomly generated. weight θ - Time step t = 1;

[0142] S22, let the selectable action space A t ={C0,C1,…,C n-1};

[0143] S23, if the action space A t If empty, proceed to step S22; otherwise, proceed to step S24.

[0144] S24. Randomly generate a real number ρ in the interval [0,1]. If ρ < ε, then randomly select an action a. t If not, then use the action-value network to select a. t =argmaxa Q(φ t ,a;θ), that is, select the action a with the largest predicted Q value based on the output of the action value network. t Where a∈A t , φ t Characteristics representing the current state;

[0145] S25, Execute action a t Receive instant reward r t and the next state s t+1 ; Calculate s t+1 Feature φ t+1 From A t Remove a from t ;

[0146] S26, the empirical tuple (φ t ,a t ,r t ,φ t+1 ) is stored in the experience replay pool D, and sampled experience tuples (φ) with replacement are retrieved from the experience replay pool. j ,a j ,r j ,φ j+1 ), where j is the index of the sampled empirical tuple;

[0147] S27, if A t If it is empty, then y j =r j If not, then Among them, y j Indicates the target Q value;

[0148] S28, for the loss function ∑(y j -Q(φ j ,a j ;θ)) 2 Update the parameters θ of the action value neural network using the stochastic gradient descent algorithm; time step t = t + 1; if t % L = 0, then let θ - =θ, where "%" represents the remainder and L represents the interval for updating network parameters of the target action value;

[0149] S29. If time step t reaches the specified termination value, the algorithm terminates; otherwise, proceed to step S23.

[0150] Table 1. Values ​​of parameters during neural network training.

[0151]

[0152]

[0153] The settings for some parameters during neural network training are shown in Table 1 above. Figure 3 The training Q mentioned net The change of the loss function over time, Q net The training loss decreased rapidly in the 0 to 1000 step range; after 1000 steps, the training loss stabilized at less than 0.2.

[0154] Step S3 above specifically includes the following steps:

[0155] S31, from n slabs (set M), select one slab as the first slab in the hot rolling plan;

[0156] S32, using the first slab of the hot-rolled plan as the current state, calculate the state characteristics;

[0157] S33, based on the current state characteristics, use the already trained neural network Q... net A hot rolling plan is generated by compiling a hot rolling plan for the slabs in set M.

[0158] Step S31 above specifically includes the following steps:

[0159] S311, Let set M be the set of n slabs. Select the batch of slabs with the largest rolling width from set M to form set M. W ;

[0160] S312, from set M W A batch of slabs with a rolling thickness equal to the mode was selected and formed into set M. T ;

[0161] S313, from set M T A batch of slabs with hardness equal to the mode was selected from the samples, forming set M. H ;

[0162] S314, choose M arbitrarily H One of the slabs in the process served as the starting point for the hot rolling program.

[0163] For the five independent rolling units in the test set, the hot rolling schedule was reconstructed based on the DQN algorithm and compared with the original schedule. The reward values ​​of the original schedule and the DQN schedule are shown in Table 2 below. Under the current reward system, the DQN schedule for these five rolling units generally achieved a higher evaluation than the original schedule, with an average improvement of 7.37%; however, the rewards for two hot rolling schedules were 9.58% and 1.34% lower than the original schedules, respectively.

[0164] Table 2 compares the rewards of the original plan and the plan developed using DQN for the dataset.

[0165]

[0166]

[0167] a. Improvement rate: The calculation formula is as follows This represents the relative difference between the rewards of the DQN plan and the rewards of the original plan, where r DQN This indicates the DQN program reward, r Ori Indicates the original planned reward; improvement rate The larger the value, the higher the DQN program reward is relative to the original program reward.

[0168] Those skilled in the art should recognize that the above embodiments are merely illustrative of the present invention and are not intended to limit the present invention. Any variations or modifications to the above embodiments that are within the spirit and essence of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A hot rolling scheduling method for a single rolling unit based on the DQN algorithm, characterized in that, Includes the following steps: S1. Standardize the existing hot rolling plan preparation data, design hot rolling scheduling status, hot rolling scheduling actions and hot rolling scheduling rewards based on reinforcement learning methods, design the structure of the neural network, and form a deep Q-learning framework. S2, Based on the aforementioned deep Q-learning framework, a training algorithm for the neural network is designed, and the neural network Q is trained using standardized hot rolling schedule data. net ; S3, based on the data of the slab to be rolled, apply the aforementioned neural network Q. net Compile and formulate a hot rolling plan.

2. The hot rolling scheduling method for a single rolling unit based on the DQN algorithm according to claim 1, characterized in that, In step S1, the existing hot rolling plan compilation data is standardized, including the standardization of the slab rolling width, rolling thickness, and hardness. The calculation formula is as follows: Where, φ t (1) φ t (2) φ t (3) These represent the standardized rolling width, rolling thickness, and hardness, respectively; w t ,g t ,h t These represent the rolling width, rolling thickness, and hardness at time step t, respectively; w max ,g max ,h max These represent the maximum values ​​of rolling width, rolling thickness, and hardness in the entire dataset, respectively.

3. The hot rolling scheduling method for a single rolling unit based on the DQN algorithm according to claim 1, characterized in that, In step S1, the hot rolling scheduling state s t ∈S is defined as all sequentially compiled slab sequences at time step t; Where S represents the set of hot rolling schedule states.

4. The hot rolling scheduling method for a single rolling unit based on the DQN algorithm according to claim 1, characterized in that, In step S1, the hot rolling scheduling action a t ∈A t It is defined as selecting a slab from those slabs that have never been compiled and including it in the hot rolling plan; Among them, A t This represents the set of hot rolling scheduling actions that can be selected at the current time t.

5. The hot rolling scheduling method for a single rolling unit based on the DQN algorithm according to claim 1, characterized in that, In step S1, a hot rolling scheduling action a is performed. t The immediate reward after r t Defined as the opposite of the penalty for violating regulations; At the current time t, the reward function for the jump in rolling width in the hot rolling plan is as follows: in, λ represents the immediate reward for a jump in the rolling width of the hot rolling plan; λ represents the penalty weight for a rebound in the rolling width. The hot rolling scheduling bonuses for jumps in rolling thickness and hardness are as follows: in, These represent the immediate rewards for the jumps in rolling thickness and hardness at time step t, respectively; avg indicates that the data within the parentheses are averaged. Instant total reward r at time step t t for: Where, ω W ω T ω H These represent the weights of the impact of the jump in rolling width, rolling thickness, and hardness on the immediate reward, respectively. The total reward r for a hot rolling project is:

6. The hot rolling scheduling method for a single rolling unit based on the DQN algorithm according to claim 1, characterized in that: In step S2, the neural network has 3 nodes in the input layer, 4 fully connected hidden layers with 32 nodes in each layer, all nodes use the ReLU activation function, and the output layer has 1 node.

7. The hot rolling scheduling method for a single rolling unit based on the DQN algorithm according to claim 6, characterized in that, The training algorithm for the neural network specifically includes the following steps: S21, Initialization, capacity is N E The experience replay pool D, where all elements are null, is used to randomly generate the weights θ of the action value network Q, and the target action value network is also randomly generated. weight θ - Time step t = 1; S22, let the selectable action space A t ={C0,C1,…,c n-1 }; S23, if the action space A t If empty, proceed to step S22; otherwise, proceed to step S24. S24. Randomly generate a real number ρ in the interval [0,1]. If ρ < ε, then randomly select an action a. t If not, then use the action-value network to select a. t =argmax a Q(φ t ,a;θ), that is, select the action a with the largest predicted Q value based on the output of the action value network. t Where a∈A t , φ t Characteristics representing the current state; S25, Execute action a t Receive instant reward r t and the next state s t+1 ; Calculate s t+1 Feature φ t+1 From A t Remove a from the middle t ; S26, the empirical tuple (φ t ,a t ,r t ,φ t+1 ) is stored in the experience replay pool D, and sampled experience tuples (φ) with replacement are retrieved from the experience replay pool. j ,a j ,r j ,φ j+1 ), where j is the index of the sampled empirical tuple; S27, if A t If it is empty, then y j =r j If not, then Among them, y j Indicates the target Q value; S28, for the loss function ∑(y j -Q(φ j ,a j ;θ)) 2 Update the parameters θ of the action value neural network using the stochastic gradient descent algorithm; time step t = t + 1; if t % L = 0, then let θ - =θ, where "%" represents the remainder and L represents the interval of updating network parameters for the target action value; S29. If time step t reaches the specified termination value, the algorithm terminates; otherwise, proceed to step S23.

8. The hot rolling scheduling method for a single rolling unit based on the DQN algorithm according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31, select one slab from n slabs as the first slab in the hot rolling plan; S32, using the first slab of the hot-rolled plan as the current state, calculate the state characteristics; S33, based on the current state characteristics, use the already trained neural network Q... net A hot rolling plan is prepared for each of the n slabs.

9. The hot rolling scheduling method for a single rolling unit based on the DQN algorithm according to claim 8, characterized in that, Step S31 specifically includes the following steps: S311, Let set M be the set of n slabs. Select the batch of slabs with the largest rolling width from set M to form set M. W ; S312, from set M W A batch of slabs with a rolling thickness equal to the mode was selected and formed into set M. T ; S313, from set M T A batch of slabs with hardness equal to the mode was selected from the samples, forming set M. H ; S314, choose M arbitrarily H One of the slabs in the process served as the starting point for the hot rolling program.

10. A hot rolling scheduling system for a single rolling unit based on the DQN algorithm, characterized in that, include: The environment module describes the hot rolling scheduling process in the form of a Markov decision process, and the content described includes the status, actions and rewards of the hot rolling schedule. The intelligent agent module is used to interact with the environment to generate experience tuples, which ultimately realize the parameter update of the neural network, including the experience replay pool and the neural network. Application module, using the Q-neuron trained by the application agent net The slab to be rolled is planned and scheduled to form a hot rolling plan; The single-rolling-unit hot rolling scheduling system implements the single-rolling-unit hot rolling scheduling method based on the DQN algorithm as described in any one of claims 1-9.

11. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the hot rolling scheduling method for a single rolling unit based on the DQN algorithm as described in any one of claims 1-9.

12. A computer device, characterized in that: It includes a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the hot rolling scheduling method for a single rolling unit based on the DQN algorithm as described in any one of claims 1-9.

Images