A continuous casting automatic casting control method based on deep reinforcement learning

Abstract

The application belongs to the field of metallurgical machinery and automation technology, and discloses a continuous steel casting automatic pouring control method based on deep reinforcement learning, which comprises the following steps: step 1, establishing a crystallizer simulation model; step 2, according to the crystallizer simulation model, establishing a crystallizer automatic pouring control model based on a Markov decision process; and step 3, according to the crystallizer automatic pouring control model, training a stopper learning control strategy by using a TD3 algorithm. The method has the characteristics of self-learning and high precision, can adapt to various steel grades, pouring section and other process parameters, and improves the intelligent level of continuous casting production.

Description

Technical Field

[0001] This invention belongs to the field of metallurgical machinery and automation technology, specifically relating to an automatic start-up control method for continuous steel casting based on deep reinforcement learning. Background Technology

[0002] The steel industry is a pillar industry in my country. The intelligent development of continuous casting production is an important link in smelting high-quality steel, and automatic crystallizer start-up control is a key technology for improving the level of intelligence and product quality in continuous casting production.

[0003] Traditional manual crystallizer start-up control relies heavily on human experience, resulting in poor product stability and adaptability. During the molten steel filling stage of the crystallizer, the high-speed molten steel stream impacts the inner wall of the crystallizer after flowing out from the side hole of the nozzle, forming upper and lower backflow. Excessive backflow can lead to problems such as slag entrapment and air bubble mixing in the molten steel, and in severe cases, it can even cause steel leakage accidents.

[0004] In recent years, with the development of artificial intelligence technology, reinforcement learning, as one of the key components of AI, has been increasingly widely applied in the field of industrial control. In reinforcement learning models, intelligent agents can make decisions and execute actions based on current environmental conditions to achieve a certain expected goal, and it is considered one of the core technologies leading to strong AI. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the existing technology, the purpose of this invention is to provide an automatic start-up control method for continuous steel casting based on deep reinforcement learning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An automatic start-up control method for continuous steel casting based on deep reinforcement learning includes the following steps:

[0008] Step 1: Establish a crystallizer simulation model;

[0009] Step 2: Based on the crystallizer simulation model, and using the Markov decision process, establish an automatic crystallizer start-up control model. The automatic crystallizer start-up control model is represented by a tuple consisting of a state space S, an action space A, a reward function R, and a discount factor γ. The crystallizer simulation model represents the state transition function of the automatic crystallizer start-up control model.

[0010] Step 3: Based on the automatic crystallizer start-up control model, use the TD3 algorithm to train the stopper rod to learn the control strategy.

[0011] Furthermore, the crystallizer simulation model, which is also the model of molten steel level changes within the crystallizer, is as follows:

[0012]

[0013] Among them, H act H represents the actual liquid level of molten steel in the crystallizer. set S1 represents the set liquid level of molten steel in the crystallizer, S1 represents the cross-sectional area of ​​the slab, k represents the correction factor, and W represents the set liquid level of molten steel in the crystallizer. i mid V represents the weight of the molten steel in the stopper rod at step i. i The volume of molten steel flowing into the crystallizer through the side hole of the nozzle at step i represents the volume of molten steel entering the crystallizer at step i; Herr represents the absolute value of the difference between the actual liquid level of molten steel in the crystallizer and the set liquid level; T represents the emergence time, that is, the total number of time steps from the start of the stopper rod to the end of the start of the straightening machine.

[0014] Furthermore, the volume of molten steel flowing into the crystallizer through the side orifice of the nozzle is obtained through a numerical relationship model of "stopper opening - molten steel flow rate", as follows:

[0015]

[0016] Wherein, P represents the stopper opening, P∈[0,30], and W big W represents the weight of the ladle of molten steel. mid The weight of the molten steel in the ladle is represented by V, the volume of molten steel flowing into the crystallizer through the side hole of the nozzle is represented by ρ, the density of the steel grade is represented by st, and the constraint condition is represented by st.

[0017] Furthermore, establishing an automatic start-up control model for the crystallizer specifically includes the following steps:

[0018] Step 2-1: Establish the state space:

[0019]

[0020] Step 2-2: Establish the action space:

[0021]

[0022] Step 2-3: Establish the reward function:

[0023] .

[0024] Furthermore, step 3 specifically includes the following steps:

[0025] Step 3-1: Initialize the Actor-Critic network framework, including two Critic networks, two target Critic networks, one Actor network, and one target Actor network; initialize the experience pool.

[0026] Step 3-2: Set the maximum number of rounds and the maximum number of steps the stick can move in each round. Perform the following steps in each round:

[0027] Step 3-2-a: The molten steel in the crystallizer is reset to the initial level. The stopper makes an action decision based on the current state and strategy, and adds random noise. The stopper executes the action, and the next state and the reward obtained are calculated by the state transition function. The current state S is then set. i Action Decision A i Reward R i Next state S i+1 Has the round ended? i The five parameters are stored in the experience pool;

[0028] Step 3-2-b: When the number of data entries in the experience pool exceeds the preset number, start updating the network. Randomly select N samples from the experience pool, calculate the target Q value through two target Critic networks, update the Critic network, perform delayed updates on the Actor network according to the deterministic policy gradient, and finally perform soft updates on all target networks.

[0029] Step 3-2-c: When the molten steel level in the crystallizer reaches the final set height or the number of steps reaches the maximum number of steps, the current round ends.

[0030] Step 3-3: After all rounds are completed, the training of the control strategy learned by the plug is finished.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) The present invention is a continuous casting automatic start-up control method based on deep reinforcement learning. It has the characteristics of self-learning and high precision, and can adapt to various steel grades, casting sections and other process parameters to improve the level of intelligence in continuous casting production.

[0033] (2) The present invention simulates and models the operating environment of the crystallizer, comprehensively considers the main factors affecting the change of liquid level in the crystallizer, and achieves rapid convergence of the reinforcement learning algorithm by using fewer manual control start-up data samples.

[0034] (3) The present invention models the automatic start-up control task of the crystallizer. After successful training, the model can adaptively generate control curves according to process parameters such as steel grade and casting section, guide the movement of stopper rod, and make the molten steel level in the crystallizer steadily rise to the set start-up level of the casting machine within a predetermined time. Attached Figure Description

[0035] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0036] Figure 1 This is a flowchart of the automatic pouring control method provided by the present invention.

[0037] Figure 2 This is a schematic diagram of the main components in the pouring scenario involved in this invention.

[0038] Figure 3 This is the fitting curve diagram when the correction coefficient k is calculated in this invention.

[0039] Figure 4 This is the overall structure diagram of the TD3 algorithm provided by the present invention.

[0040] Figure 5 This is a training result diagram based on the TD3 algorithm provided by the present invention.

[0041] Figure 6 This is a schematic diagram of the casting process for the present invention under the condition of X steel grade and slab specifications of 1540*230.

[0042] Figure 7 This is a schematic diagram of the casting process for the present invention under the condition of X steel grade and slab specifications of 1300*220.

[0043] Figure 8 This is a schematic diagram of the casting process for the present invention under the condition of X steel grade and slab size of 1100*200. Detailed Implementation

[0044] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0045] Example 1

[0046] Reference Figure 1 An automatic casting start control method based on deep reinforcement learning includes the following steps:

[0047] Step 1: Establish a crystallizer simulation model;

[0048] Specifically, the rate of change of molten steel level in the crystallizer is positively correlated with the opening of the stopper rod. Within the effective stroke of the stopper rod, the larger the opening, the faster the molten steel flows into the crystallizer through the side hole of the nozzle, resulting in more intense backflow. Since the fluidity of molten steel is affected by various factors such as steel grade and temperature, traditional fluid dynamics simulation experiments are difficult to accurately simulate the specific situations in engineering practice. Therefore, this invention uses a small amount of manual casting data as a basis to directly perform statistical modeling of the numerical relationship between the stopper rod opening and the nozzle flow rate.

[0049] Reference Figure 2 In general engineering practice, molten steel from the ladle flows uniformly into the tundish through a fixed-opening nozzle. Molten steel from the tundish then flows non-uniformly into the crystallizer via a stopper rod, with the outflow velocity from the ladle being greater than that from the tundish. Consequently, the molten steel level in the crystallizer gradually rises over time. In the manual pouring data, the sensor sampling interval is 0.5 seconds. To reduce the impact of sensor detection errors, this invention uses three data points as a group, with a sampling interval of 1.5 seconds, to extract the manual pouring data. Within each 1.5-second interval, the stopper rod opening is taken as the average of the three detection data points. According to the law of conservation of mass, the decrease in the weight of molten steel in the ladle minus the increase in the weight of molten steel in the tundish gives the weight of molten steel flowing from the tundish into the crystallizer at that stopper rod opening.

[0050] Step 1-1: Establish a numerical relationship model between "stopper opening degree and molten steel flow rate":

[0051]

[0052] Wherein, P represents the stopper opening, P∈[0,30], and W big W represents the weight of the ladle of molten steel. mid V represents the weight of molten steel in the ladle, V represents the volume of molten steel flowing into the crystallizer through the side hole of the nozzle, and ρ represents the density of the steel grade.

[0053] Considering that the weight of the molten steel in the ladle also affects the flow rate at the nozzle, the heavier the molten steel in the ladle, the faster the molten steel flows out at the same stopper opening. Therefore, the ladle weight is set as a parameter in the molten steel level change model in the crystallizer, and a correction coefficient k is added.

[0054] Step 1-2: Establish a model for the change in molten steel level within the crystallizer:

[0055] Among them, H act H represents the actual liquid level of molten steel in the crystallizer. set S1 represents the set liquid level of molten steel in the crystallizer, S1 represents the cross-sectional area of ​​the slab, and k represents the correction coefficient, the value of which is determined by fitting the data of manual casting. i mid V represents the weight of the molten steel in the stopper rod at step i. i The volume of molten steel flowing into the crystallizer through the side hole of the nozzle at step i represents the volume of molten steel entering the crystallizer at step i; Herr represents the absolute value of the difference between the actual liquid level of molten steel in the crystallizer and the set liquid level; T represents the emergence time, that is, the total number of time steps from the start of the stopper rod to the end of the start of the straightening machine.

[0056] Reference Figure 3The time frame is from the start of the stopper rod to the end of the start of the straightening machine. `Lv Detect` represents the molten steel level value detected by the sensor in a manually generated casting data set. Due to hardware limitations and the influence of high-temperature steam, the sensor can only accurately detect molten steel within a certain range downwards from the top of the crystallizer. `Lv Convert` represents the molten steel level value calculated by the crystallizer simulation model according to the present invention, with the correction coefficient `k` set to 306. In the sub-figure, the maximum level error is 3.95 mm, the minimum is 0.32 mm, and the average is 1.29 mm. The curve fitting results show that the crystallizer simulation model proposed in this invention performs well.

[0057] Step 2: Based on the crystallizer simulation model, and using a Markov decision process, establish an automatic crystallizer start-up control model. This model is represented by a tuple consisting of a state space S, an action space A, a reward function R, and a discount factor γ. The crystallizer simulation model represents the state transition function of the automatic crystallizer start-up control model.

[0058] Specifically, the reinforcement learning process is a process of continuous trial and error for the plug rod. Markov decision processes are often used as a model framework to describe reinforcement learning tasks. The automatic pouring control model for the crystallizer is represented by a quadruple [S,A,R,γ].

[0059] Step 2-1: Establish the state space, including the actual liquid level and the set liquid level in the crystallizer, as well as the weight of the intermediate package:

[0060]

[0061] Step 2-2: Establish the action space, i.e., the opening of the stopper rod, taking any value within 0 to 30 mm:

[0062]

[0063] Steps 2-3: Establish the reward function. Calculate the reward value obtained by the stopper rod based on the difference between the actual liquid level and the target liquid level in the crystallizer. The smaller the difference, the larger the reward value, which can better guide the stopper rod to make optimal action decisions.

[0064]

[0065] Step 3: Based on the automatic crystallizer start-up control model, use the TD3 algorithm to train the stopper rod to learn the control strategy.

[0066] Reference Figure 4 The TD3 algorithm is used to train the plug rod using a neural network. The optimal neural network parameters are obtained by iteratively converging the constructed policy gradient function through gradient descent. For example, at step i, the plug rod's current state space s is input. iTo the Actor network, the Actor network is based on state s i The opening a of the output stopper rod i To increase the exploratory nature of the stopper rod, for a i Add random noise. (The last part is incomplete and likely refers to a specific type of noise.) i With a i The input is fed into the crystallizer simulation model, and the reward value r is obtained through the state transition function. i and the next state s i+1 , will s i a i r i s i+1 d i Add the data to the experience pool. Once a certain number of samples are collected, randomly select N samples to update the network parameters. In practice, it is recommended that N be 64 or 128.

[0067] Step 3-1: Initialize the Actor-Critic network framework, including two Critic networks, two target Critic networks, one Actor network, and one target Actor network, and initialize the experience pool.

[0068] Step 3-2: Set the maximum number of rounds and the maximum number of steps the stick can move in each round. Perform the following steps in each round:

[0069] Step 3-2-a: The molten steel in the crystallizer is reset to the initial level. The stopper makes an action decision based on the current state and strategy, and adds random noise. The stopper executes the action, and the next state and the reward obtained are calculated by the state transition function. The current state S is then set. i Action Decision A i Reward R i Next state S i+1 Has the round ended? i The five parameters are stored in the experience pool.

[0070] Step 3-2-b: When the number of data entries in the experience pool exceeds the preset number, start updating the network. Randomly select N samples from the experience pool, calculate the target Q value through two target Critic networks, update the Critic network, perform delayed updates to the Actor network according to the deterministic policy gradient, and finally perform soft updates to all target networks.

[0071] Step 3-2-c: When the molten steel level in the crystallizer reaches the final set height or the number of steps reaches the maximum number of steps, the current round ends.

[0072] Step 3-3: After all rounds are completed, the training of the control strategy learned by the plug is finished.

[0073] To better illustrate this, a simulation experiment was conducted in this embodiment. The training simulation environment and algorithm program were programmed in Python, and a deep reinforcement learning framework was built based on TensorFlow. The neural networks in the algorithm all adopted a fully connected network architecture, and the activation function adopted was the rectified linear function (ReLU).

[0074] The initial height inside the crystallizer is set to 350mm. Before filling with molten steel, a stopper rod test is performed, which involves holding the stopper rod at 17mm, 8mm, and 12mm positions for two seconds each. This test is to check the flexibility of the stopper rod and prevent jamming during the actual pouring process. The actual filling process consists of five stages, and the duration and target height of each stage are shown in the table below:

[0075] Table 1 Simulation Parameter Table

[0076]

[0077] The relevant neural network parameters and training parameters are shown in Table 2 below:

[0078] Table 2 Neural Network and Training Parameters

[0079]

[0080] Based on the parameters in the table above, the stopcock was trained for 200 rounds using the TD3 algorithm, and the cumulative reward for each round was recorded. Figure 5 Experimental results show that as the number of training rounds increases, the plug learns from experience through exploration, trial and error, and eventually converges around 100 rounds.

[0081] This embodiment also simulated the casting process for a certain steel grade with three different specifications, referring to... Figures 6 to 8 The dashed line represents the stopper rod movement curve, while the solid line and dotted line represent the set liquid level and actual liquid level in the crystallizer, respectively. Using the error between the set liquid level and the actual liquid level in the crystallizer as an evaluation index, different error thresholds were set, and the accuracy of the automatic crystallizer start-up control model was calculated. The results are shown in Table 3. Experimental results show that when the slab specifications change significantly, the stopper rod can still effectively control the molten steel level in the crystallizer to gradually rise according to the set level.

[0082] Table 3. Accuracy of the model at different error thresholds

[0083]

[0084] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for automatic start-up control of continuous steel casting based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Establish a crystallizer simulation model; The crystallizer simulation model, also known as the model of molten steel level changes within the crystallizer, is as follows: ， Among them, H act H represents the actual liquid level of molten steel in the crystallizer. set S1 represents the set liquid level of molten steel in the crystallizer, S1 represents the cross-sectional area of ​​the slab, k represents the correction factor, and W represents the set liquid level of molten steel in the crystallizer. i mid V represents the weight of the molten steel in the stopper rod at step i. i Herr represents the volume of molten steel that flows into the crystallizer through the side hole of the nozzle at step i; Herr represents the absolute value of the difference between the actual liquid level of molten steel in the crystallizer and the set liquid level; T represents the emergence time, that is, the total number of time steps from the start of the stopper rod to the end of the start of the straightening machine. Step 2: Based on the crystallizer simulation model, and using the Markov decision process, establish an automatic crystallizer start-up control model. The automatic crystallizer start-up control model is represented by a tuple consisting of a state space S, an action space A, a reward function R, and a discount factor γ. The crystallizer simulation model represents the state transition function of the automatic crystallizer start-up control model. Step 3: Based on the automatic crystallizer start-up control model, use the TD3 algorithm to train the stopper rod to learn the control strategy.

2. The automatic start-up control method for continuous steel casting based on deep reinforcement learning according to claim 1, characterized in that, The volume of molten steel flowing into the crystallizer through the side orifice of the nozzle is obtained using a numerical relationship model of "stopper opening - molten steel flow rate", as follows: ， Wherein, P represents the stopper opening, P∈[0,30], and W big W represents the weight of the ladle of molten steel. mid The weight of the molten steel in the ladle is represented by V, the volume of molten steel flowing into the crystallizer through the side hole of the nozzle is represented by ρ, the density of the steel grade is represented by st, and the constraint condition is represented by st.

3. The automatic start-up control method for continuous steel casting based on deep reinforcement learning according to claim 2, characterized in that, The specific steps for establishing an automatic start-up control model for the crystallizer are as follows: Step 2-1: Establish the state space: ， Step 2-2: Establish the action space: ， Step 2-3: Establish the reward function: 。

Images