Actioner-reviewer learning agent providing autonomous operation of twin roll continuous casting machine

Abstract

A twin-roll continuous casting system includes counter-rotating casting rolls having a nip between the casting rolls and capable of transporting a casting strip downward from the nip, a casting roll controller configured to adjust at least one process control setpoint between the casting rolls in response to a control signal, a casting strip sensor configured to detect a process control setpoint between the casting rolls, and a controller configured to detect the process control setpoint between the casting rolls in response to the control signal. The cast strip sensor is configured to measure at least one parameter of the cast strip, and the controller is coupled to the cast strip sensor to receive a cast strip measurement signal from the cast strip sensor and to the cast roll controller to provide a control signal to the cast roll controller, the controller including a reinforcement learning (RL) proxy. The RL agent also includes a model-free actor-reviewer agent having a value function and a policy function, the RL agent having been trained on a plurality of continuous casting system operational data sets consisting of continuous casting runs performed by a plurality of different human operators.

Description

Background Art

[0001] Twin roll casting (TRC) is a near-net-shape manufacturing process that is used to produce strip of steel and other metals. In this process, molten metal is poured onto the surfaces of two casting rolls which simultaneously cool and solidify the metal into a strip close to its final thickness. The process is characterized by rapid thermomechanical dynamics that are difficult to control to achieve the desired properties of the final product. This is true not only for steady-state casting, but even more so during "start-up" (the casting transition period prior to steady-state casting). Strip metal produced during startup often contains an unacceptable amount of defects. For example, strip flutter is a phenomenon where the continuous caster vibrates around 35 Hz and 65 Hz. More specifically, the vibrations induce variations in the solidification process and lead to surface defects such as Figure 1A and 1B The chatter needs to be brought below an upper limit before commercially acceptable strip metal can be produced.

[0002] During the start-up and steady-state casting process, human operators are tasked with manually adjusting certain process control set points. During the start-up process, the operator's goal is to stabilize the production of steel strip as quickly as possible, including reducing chatter, in order to minimize the length of the start-up period while meeting specific strip quality metrics, thereby increasing product output by minimizing process start-up losses. They do this through a series of binary decisions (on / off switches) and continuous adjustment of multiple set points. In total, the operator controls more than twenty switches and set points; for the latter, the operator must determine when to adjust the set point and by how much.

[0003] Of the operator adjusted set points, the casting roll separation force set point (which will be referred to as the "force set point" from here on) is the set point that is most frequently adjusted during startup. It may be adjusted dozens of times in a period of about five minutes. The operator considers many factors when adjusting the force set point, but the most important is strip flutter, which is a strip defect caused by the natural frequency of the continuous caster.

[0004] Operators use various strategies to adjust the force set point. One is to consider a threshold for the chatter measurement; when the chatter value increases above the threshold, the operator will begin to reduce the force. However, individual operators use different thresholds based on their own experience and factors including the specific grade of steel or the width being cast. On the other hand, reducing the force too much may cause other quality problems in the steel strip; therefore, operators are usually trained to maintain the force as high as possible with reduced chatter.

[0005] Attempts have been made to improve various industrial processes, including twin-roll continuous casting. In recent years, man-in-the-loop control systems have become increasingly popular. Instead of considering people as exogenous signals (such as interference), man-in-the-loop systems consider people as part of the control system. Man-in-the-loop applications can be divided into three main categories: human control, human monitoring, and a mixture of the two. Human control is when a person directly controls the process, which can also be called direct control. Supervisory control is a hybrid approach in which a human operator adjusts specific set points and otherwise supervises the process of the primary automatic control. Supervisory control is commonly found in industry and has been the primary way to operate a twin-roll continuous caster to date. However, variations between human operators, such as changes in their personality traits, past experience, skill levels, or even their current moods, as well as varying, non-characteristic process factors, continue to lead to inconsistencies in process operation.

[0006] Modeling human behavior has been considered a black box problem. More specifically, researchers agree that system identification techniques can be used to simulate human behavior in man-in-the-loop control systems. These typically refer to predictive models of human behavior and subsequently to controller design based on the identified model. The effectiveness of this approach of first identifying a model of human behavior and then designing a controller based on the model depends on the available data. Disadvantageously, if the human data contains multiple different operator behaviors, any identified model will likely underfit the data and result in a poorly performing controller due to significant variations between different operators.

[0007] Furthermore, the proposed approaches aim to characterize the role of the human operator as a feedback controller in the system, however, rather than modeling the behavior of the human operator, the proposed approaches identify the optimal control strategy based on a system model. In other words, they do not learn directly from the strategies used by experienced human operators. In some industrial applications, particularly during highly transient periods of operation such as process startup, system modeling can be extremely difficult, and not all control objectives can be quantified. Therefore, it is not trivial to automate such processes using a model-based approach; instead, a method is needed to determine the optimal operating strategy based on both explicit control objectives and implicit control objectives revealed by the human operator's behavior. Summary of the invention

[0008] The twin-roll continuous casting system includes a pair of counter-rotating casting rolls, a casting roll controller, a casting strip sensor, and a controller, wherein the pair of counter-rotating casting rolls have a roll gap between the casting rolls and are capable of conveying a casting strip downward from the roll gap, the casting roll controller is configured to adjust at least one process control set point between the casting rolls in response to a control signal, the casting strip sensor is capable of measuring at least one parameter of the casting strip, the controller is coupled to the casting strip sensor to receive the casting strip measurement signal from the casting strip sensor, and is coupled to the casting roll controller to provide a control signal to the casting roll controller, and the controller includes a reinforcement learning (RL) agent. The RL agent also includes a model-free actor-critic agent having a value function and a policy function, and the RL agent has been trained on a plurality of continuous casting system operation data sets consisting of continuous casting runs performed by a plurality of different human operators.

[0009] In some embodiments, the RL agent further comprises an advantage function that calculates an advantage value of a selected action as an immediate reward value of the selected action plus a discounted value of a subsequent state of the selected action minus a value of the current state; and the advantage value is used to train a policy function. In some embodiments, the policy function is configured to evaluate the advantage function in the following manner: assigning a value to actions from the plurality of continuous casting system operation data sets that have a negative advantage value relative to actions that are not found in the plurality of continuous casting system operation data sets.

[0010] In some embodiments, the RL agent also includes an advantage function that calculates an advantage value of a selected action as an immediate reward value of the selected action plus a discounted value of a subsequent state of the selected action minus a value of the current state; and a natural exponent of the advantage value is used to train the policy function.

[0011] The cast strip sensor may include a thickness gauge that measures the thickness of the cast strip at intervals across the width of the cast strip.The process control set point may include a force set point between the casting rolls, and the parameter of the cast strip may include chatter.

[0012] In some embodiments, the RL agent further comprises a reward function that calculates an immediate reward as a weighted piecewise defined reward function based on user defined thresholds for chatter and edge spike parameters. In some embodiments, the RL agent further comprises an advantage function that calculates an advantage value as an immediate reward value for a selected action plus a discounted value of a subsequent state of the selected action minus a value of the current state.

[0013] The at least one parameter of the cast strip may include flutter and at least one strip profile parameter.The at least one strip profile parameter may be selected from the group consisting of edge protrusion, edge ridge, maximum peak, and high edge flag.

[0014] The policy function may include a stochastic policy function. The policy function may also include dependencies on actions of previous steps.

[0015] The data in the operational dataset can be augmented. In this embodiment, for each step in the operational dataset, the recurrence from the previous step is embedded to improve the actor training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A is the belt profile without chatter defects.

[0017] Figure 1B is a belt profile with chatter defects.

[0018] Figure 2 is a diagrammatic representation of a twin roll continuous caster in accordance with at least one aspect of the present invention.

[0019] Figure 3 yes Figure 2 Illustration of a detail of a twin-roll continuous caster shown in FIG.

[0020] Figure 4 is a plot of the mean force trajectories of the clusters of the training dataset.

[0021] Figure 5A yes Figure 4 Figure 2 shows an example of the force trajectory of cluster 1 in Figure 2.

[0022] Figure 5B yes Figure 4 Figure 2 shows an example of the force trajectory of cluster 2 in Figure 2.

[0023] Figure 5C yes Figure 4 Figure 2 shows an example of the force trajectory of cluster 3 in Figure 2.

[0024] Fig. 6A yes Figure 4 Plot of the maximum chatter amplitude spectrum of cluster 1 in Figure 2.

[0025] Figure 6B yes Figure 4 Plot of the maximum chatter amplitude spectrum for cluster 2 in Figure 2.

[0026] Figure 6C yes Figure 4 Figure 2. Plot of the maximum chatter amplitude spectrum of cluster 3 in Figure 2.

[0027] Figure 7 is a plot of the RL agent’s force setpoint value trajectory and the associated chatter trajectory.

[0028] Figure 8 is a plot comparing the force setpoint value trajectories and the associated chatter trajectories of the two RL agents.

[0029] Fig. 9 is a second plot comparing the force setpoint value trajectories and associated chatter trajectories of the two RL agents.

[0030] Fig.10 is a third plot comparing the force setpoint value trajectories and associated chatter trajectories of the two RL agents.

[0031] Fig.11 is a fourth graph comparing the force setpoint value trajectories and the associated chatter trajectories of the two RL agents.

[0032] Fig.12 is a plot comparing the RL agent's force setpoint value trajectory with the operator's force setpoint value trajectory and the associated chatter trajectory.

[0033] Fig.13 is a second plot comparing the RL agent's force setpoint value trajectory to the operator's force setpoint value trajectory and the associated chatter trajectory.

[0034] Fig.14A is a graphical representation of the thickness variation along the length of the cast strip.

[0035] Fig. 14B is an illustration of a waviness defect in a cast strip profile, including edge peaks.

[0036] Fig.15 is a schematic diagram of a cross section of a cast strip depicting the edge peak parameters.

[0037] Fig.16 is a plot of the average silhouette width versus the number of clusters for the operational dataset.

[0038] Fig.17 The contour width of each sample under different clustering settings is shown.

[0039] Fig.18A An example of the force trajectories in the first cluster is shown.

[0040] Fig.18B An example of force trajectories in the second cluster is shown.

[0041] Fig.19A RL agent verification in the first case of an initial edge spike is shown.

[0042] Fig.19B The RL agent verification in the second case of the initial edge spike is shown, the second case having lower edge spikes than the first case.

[0043] Fig. 20A RL agent verification in the third case of an initial edge spike is shown.

[0044] Fig. 20B RL agent verification in a fourth case of an initial edge spike is shown, the fourth case having a similar edge spike as the third case.

[0045] Fig.21 is a simplified diagram of a twin-roll continuous caster showing the relationship between the roll separation force and the cast strip.

[0046] Fig. 22 The trajectories of the forces and control targets of the human operator and the RL agent under the operating conditions of the fifth case, without the augmented dataset control, are shown.

[0047] Fig.23 Trajectories of force and thickness with corresponding losses of a human operator and a RL agent under operating conditions of the fifth case, without augmented dataset control, are shown.

[0048] Fig.24 The trajectories of the forces and control targets of the human operator and the RL agent under the fifth case operating condition with augmented dataset control are shown.

[0049] Fig.25 Trajectories of force and thickness with corresponding losses of a human operator and a RL agent under operating conditions of the fifth case with augmented dataset control are shown. DETAILED DESCRIPTION

[0050] refer to Figure 2 and Figure 3 , a twin roll continuous caster is generally indicated at 11, which produces a thin cast steel strip 12, which passes through a guide table 13 into a transition path to a pinch roll stand 14. After leaving the pinch roll stand 14, the thin cast strip 12 enters and passes through a hot rolling mill 16, which includes a backup roll 16B and upper and lower work rolls 16A, where the thickness of the strip is reduced. When leaving the rolling mill 15, the strip 12 is transferred to a delivery table 17, where the strip 12 can be force-cooled by water (or water / air) jets 18, and then the strip 12 passes through a pinch roll stand 20 including a pair of pinch rolls 20A and reaches a coiler 19.

[0051] The twin roll continuous casting machine 11 includes a main frame 21 supporting a pair of laterally positioned casting rolls 22 having casting surfaces 22A and forming a nip 27 therebetween. Molten metal is supplied from a ladle (not shown) to a tundish 23 during the casting activity, through a refractory shield 24 to a removable tundish 25 (also referred to as a distributor container or transition piece), and then through a metal delivery nozzle 26 (also referred to as a core nozzle) above the nip 27 between the casting rolls 22. Molten steel is introduced from the tundish 23 into the removable tundish 25 via an outlet of the shield 24. The tundish 23 is equipped with a sliding gate valve (not shown) to selectively open and close the outlet 24 and effectively control the flow of molten metal from the tundish 23 to the continuous casting machine. The molten metal flows from the removable tundish 25 through the outlet, and optionally flows to and through the core nozzle 26.

[0052] The molten metal thus delivered to the casting rolls 22 forms a casting pool 30 supported by the casting roll surface 22A above the roll gap 27. The casting pool is limited at the ends of the rolls by a pair of side dams or plates 28, which are applied to the ends of the rolls by a pair of pushers (not shown) comprising hydraulic cylinder units connected to the side dams. The upper surface of the casting pool 30 (commonly referred to as the "meniscus" level) can be raised above the lower end of the delivery nozzle 26 so that the lower end of the delivery nozzle 26 is immersed in the casting pool.

[0053] The casting rolls 22 are water cooled internally by a coolant supply (not shown) and are driven in counter-rotational directions by drives (not shown) so that the shell solidifies on the moving casting roll surfaces and is brought together at the roll nip 27 to produce the thin cast strip 12 which is conveyed downwardly from the roll nip between the casting rolls.

[0054] Below the twin roll caster 11, the cast steel strip 12 is conveyed within the sealed enclosure 10 to a guide table 13, which guides the strip through an X-ray machine for measuring the strip profile to a pinch roll stand 14, through which the strip leaves the sealed enclosure 10. The seal of the enclosure 10 may not be complete, but is suitable for allowing controlled atmosphere within the enclosure and controlled oxygen access to the cast strip within the enclosure. After leaving the sealed enclosure 10, the strip may pass through another sealed enclosure (not shown) after the pinch roll stand 14.

[0055] The casting roll controller 94 is coupled to actuators that control all casting roll operating functions. One of the controls is the force set point adjustment. This determines the amount of force applied to the strip as it is cast and solidifies between the casting rolls. Oscillations in the feedback from the force actuators indicate chatter. The force actuator feedback can be provided to the casting roll controller or recorded by a separate device / software.

[0056] The controller 92 comprises a trained RL agent that is coupled to the casting roll controller 94 via, for example, a computer network. The controller 92 provides force actuator control inputs and receives force actuator feedback to the casting roll controller 94. The force actuator feedback may come from commercially available data logging software or the casting roll controller 94.

[0057] In some embodiments, a transverse thickness profile is obtained by the thickness gauge 44 and transmitted to the controller 92 before the strip enters the hot rolling stand.

[0058] The present invention avoids the shortcomings of known control systems by employing a model-free reinforcement learning engine such as a Deep Q Network (DQN) that has been trained on metrics from the manual control process, including operator actions and caster responses, as an RL agent in the controller 92. The DQN is a neural network that approximates the action value of each state-action pair.

[0059] In the first embodiment provided below, configuration and training of an RL agent with one action and a reward function is provided, the reward function having one caster quality metric. However, this is for clarity of the present disclosure, and additional actions and caster feedback responses may be incorporated into the RL agent. Additional actions include rolling mill control. Additional metrics may include, for example, cast strip profile measurement and flatness measurement. Furthermore, while various embodiments disclosed herein use RL agents as examples, other model-free adaptive and / or learning agents may also be suitable and therefore may be substituted in any disclosed embodiment.

[0060] In the first embodiment, DQN is a function that maps states to action values ​​for all actions in an action set, as shown in Equation 1, where Q is a neural network, S is the state information of a sample, and The action value corresponding to the N elements in the action set.

[0061]

[0062] In some embodiments, the state at time step t is defined as S t =[C t ,δC t ,F t ,δF t ], where C and δC are the chatter and the change in chatter over one time step, respectively, and F and δF are the force and the change in force over one time step, respectively. In some embodiments, casting data is recorded at 10 Hz. Force setpoint adjustments made by the operator may be downsampled to 0.2 Hz, based on the observation that the operator typically does not adjust the force setpoint more frequently than 0.2 Hz. Given the noisy nature of the chatter signal, every 50 consecutive samples may be averaged (i.e., the chatter averaged over a 5 second period) to obtain Ct In some embodiments, non-overlapping 5 second blocks are used. Two index subscripts represent data samples, namely t and k. The time index t represents a time step within a single cast sequence. The sample index k represents the unique index of a sample in a data set that contains samples from all cast sequences.

[0063] In some embodiments, an action is defined as a change in the force setpoint value between the current time step and the next time step. Unlike a state, which is a continuous value, an action is a i ,{i=1,2,…,N}. In the problem considered here, N=4; there are three commonly used force reduction rates, and the last action represents keeping the force value unchanged.

[0064] In reinforcement learning (RL), rewards reflect what the user values ​​and what the user avoids. In the context of using RL to design a policy for regulating a process setpoint, two types of information can be used: 1) the behavior of an "expert" operator, and 2) a performance measure that is well defined in terms of states. Each plays a different role in defining the reward function that incentivizes the desired behavior.

[0065] Given that a human operator may control the process based on general rules of thumb and their personal experience with the process, a reward function designed to simulate the operator's behavior is a way to capture their expertise without requiring a model of their decision making. On the other hand, if the reward function is designed to only simulate their behavior, the trained RL agent will not necessarily be able to improve on the operator's actions. To achieve the latter, it is useful to consider a second component of the reward function, which places value on an explicit performance metric. For example, in the force setpoint regulation problem solved in this first embodiment, the desired performance goal is to be below some upper limit T su The short start-up time and below a certain upper limit C ub The low chatter level is as described below.

[0066] In some embodiments, the implicit characterization of the performance target includes the following: To better characterize different force set point adjustment behaviors, a k-means clustering algorithm may be applied to cluster over 200 separate casting sequences, all representing the same metal grade and strip width, based on the force set point trajectory implemented by the operator during each casting for a given metal grade and strip width, to ensure that the differences identified by clustering are a function of the behavior of the human operator working during each casting campaign for that grade and width.

[0067] Additional levels and widths may be characterized in a similar manner. Alternatively, additional levels and widths may use the same trained RL agent, but with different starting points assigned to the different levels and widths.

[0068] In the examples herein, the force set point adjustment behavior is characterized by the force set point trajectory for a 500 second period after the initial automatic adjustment. In one example, in the available casting data sequence, the behavior of a total of 6 different operators is represented. During a given casting, the process is operated by a crew of 2 operators, one of whom is responsible for the force set point adjustment. In order to account for the different force set point adjustment behaviors of different crews, the training data set is clustered and the preferred behaviors are identified. In some embodiments, for the k-means algorithm, k={3,4,5,6}. In this example, for the data set, the clustering results are most stable for k=3. Only 2% of the casting sequences remain offset from one cluster to another. Other values ​​of k may be applicable to other data sets. Figure 4 The average force trace is shown, which is calculated by averaging the values ​​at each time step in the force trace of each cluster respectively. Figures 5(a)-5(c) show examples from each of the three clusters. Figures 6(a)-6(c) show histograms of the chatter amplitude for each of the three clusters. According to Table I, cluster 3 has the shortest average start-up time but does not have the smallest start-up time variation; cluster 1 has the smallest start-up time variation but does not have the shortest average start-up time.

[0069] Cluster 3 is also characterized by the most aggressive set point adjustment behavior, both in terms of the rate at which the force set point is reduced and in terms of the overall magnitude of the force set point reduction. Another characteristic of the casting sequences belonging to cluster 3 is that they cover a wider range of force set point values ​​due to the aggressive adjustment of the set point. Cluster 3 is preferred because it has the shortest average start-up time and the lowest overall chatter level among the three force behavior clusters.

[0070] Table I: Scaled temporal performance statistics for force clusters; mean actuation times and standard deviations are normalized to cluster 2.

[0071]

[0072]

[0073] In addition to rewarding the simulation of certain operator setpoint adjustment behaviors, the reward function should also explicitly incentivize the desired performance metric. su, it is important to reward or penalize each time step equally, since it is unknown whether a decision made near the start of a casting results in a short startup time. To emphasize that casting sequences with different startup times should be rewarded differently, in some embodiments, the time reward for each step is Where T su is the start-up time, and T ub It is the upper limit of the startup time that the user considers acceptable. su As the upper limit is approached, the exponential function results in an increasing penalty rate.

[0074] In this embodiment, the second performance goal is to maintain the jitter value below a certain user-defined threshold. ub The maximum acceptable vibration value is indicated; if the vibration value is lower than C ub , then there is no flutter penalty assigned to this step. Mathematically, the flutter reward can be expressed as [min(0,C ub -C t )]. Reducing the force too much may cause other quality problems of the steel strip at the expense of reducing chatter. Therefore, the lower limit of the acceptable force F lb It is also mandatory.

[0075] The total reward function is shown in Equation 2:

[0076]

[0077] In addition to the implicit and explicit performance goals above, using R t The first term of applies a constant reward at each sample. Based on the casting activity records, it can be observed that the operator generally avoids reducing the force setpoint at a given time step when the chatter value and the start-up time at a given sample are within acceptable levels. In order to incentivize the RL agent to learn from this behavior, a constant reward is assigned to each sample obtained from the operator's casting records. If for a sample, the sum of both the time and chatter penalties (negative rewards) is less than this constant, the net reward for this sample is still positive. In addition, in order to emphasize that there are specific types of behaviors that the RL agent is expected to learn from, additional constant rewards can be assigned to samples in the casting sequence from the cluster of preferred force behaviors, and the net reward for each of these samples will be positive. Associated with the modified training algorithm below, these positive net rewards incentivize the RL agent to follow the operator's behavior in certain situations.

[0078] In a typical DQN training process, the RL agent performs additional trials based on the updated value function and collects more data from the new trials. However, the expense of operating an actual twin-roll strip caster, including the materials considered and produced, makes it infeasible to train the RL agent to perform trials on an actual caster. In this case, all available samples from operator-controlled casting activities are collected from the casting to train the value function Q at each training step. Training can be continued on an actual operating caster.

[0079] In some embodiments, the MATLAB deep learning toolbox is used to initialize and train the DQN. However, other reinforcement learning networks and tools can be used. Specifically, as shown in Algorithm 1, the train() function is used, and the state S of all samples is K As the network input, and its corresponding action value q K The set of parameters used as the training value function .

[0080] Algorithm 1 Pseudocode of the deep Q-network learning process (modified version)

[0081] 1: Initialize the discount factor γ

[0082] 2: Initialize parameter set And create a neural network

[0083] 3: Initialize the action value q of each sample k

[0084] 4: Training with all samples

[0085] 5: for each iteration do

[0086] 6: Update q K :

[0087] 7: Training with all samples

[0088] 8: end for each q k convergence

[0089] Another modification during training is the action value q k Updates. k is a 1-by-N vector, and each entry represents the action value of an action option. As shown in the following equation 3:

[0090]

[0091] Among them, onehot(Ak ) is action A k is a one-hot encoding of the selected action (a 1-by-N vector with 1s for the selected action and 0s for the rest), d is a binary indicator of whether the current state is a terminal of the trajectory, ones is a 1-by-N vector with all elements set to 1, and S k′ In the current state S k The equation updates the action-value of the selected action to be the sum of the immediate reward and the discounted maximum of the state at the next time step. However, for those actions that were not selected, instead of approximating their action-values ​​by using the value function from the previous iteration, their action-values ​​are set to zero plus the discounted maximum of the next state. k The update works more like a labeling process for a classification problem. If the immediate reward is positive, the trained RL agent is more likely to act like the operator, and increasing the immediate reward increases the likelihood of emulating the operator's behavior. Conversely, if the immediate reward is negative, the action selected by the operator is less likely to be selected than the other N-1 actions that were not selected. In addition, the likelihood of selecting each of the N-1 actions increases equally.

[0092] By combining DQN with a greedy policy and selecting the most valuable action in each given state, the trained RL agent can adjust the force set point. The RL agent is required to provide force set point adjustment based on the available casting sequence data and record the force set point trajectory for each casting sequence in the validation set. A more specific testing process is shown in Algorithm 2.

[0093] Algorithm 2 Pseudocode for proxy checking

[0094] 1: Get F1, C1, C0 from casting sequence data

[0095] 2: Initialize δ(F1) = 0

[0096] 3: Calculate δ(C1) = C1-C0

[0097] 4: Form the first state: S1 = [F1, δ(F1), C1, δ(C1)]

[0098] 5: Import the trained action-value function Q

[0099] 6: Initialize time step t = 1

[0100] 7: for each time step t do

[0101] 8: Calculate the action value in the current state:

[0102]

[0103] 9: Select actions based on action value:

[0104] 10: Get C from the casting sequence t+1 .

[0105] 11: Calculate δ(C t+1 )←C t+1 -C t

[0106] 12:ifF t ≥F lb then

[0107] 13: Update δ(F t+1 )←A t

[0108] 14: Calculate F t+1 ←F t +A t

[0109] 15: else

[0110] 16: Update δ(F t+1 )←0

[0111] 17: Calculate F t+1 ←F t

[0112] 18: end if

[0113] 19: Form the next state: S t+1 ←[F t+1 ,δ(F t+1 ),C t+1 ,δ(C t+1 )]

[0114] 20: Update t←t+1

[0115] 21: end for until the casting sequence ends

[0116] Algorithm 2 is used to calculate and collect the force decision trajectories of each RL agent under different chatter scenarios. Figure 7 Contains the force setpoint value trajectory of the RL agent and the associated flutter trajectory under which the force setpoint value is calculated for T ub =500, C ub = 0.5, and prefers the operator behavior described by Cluster 3. When the chatter exceeds a specified threshold and / or the chatter has an increasing trend, the RL agent begins to reduce the force setpoint; similarly, as the chatter decreases below the threshold and / or the chatter shows a decreasing trend, the RL agent stops further reductions in the force setpoint. As expected, these results are consistent with the design of the reward function.

[0117] To demonstrate the sensitivity of the trained RL agent to the operator data used for training, two different preferred clusters were created. The first contained only casting sequences from the most aggressive cluster (cluster 3 from the k-means clustering results), while the second contained casting sequences from both the most aggressive cluster (cluster 3) and the moderate cluster (cluster 2). Both RL agents were trained with the same data set but different preference cluster settings. Casting sequences belonging to cluster 3 were considered preferred in both training settings because these data included system operations across the entire range of possible force state values, while data belonging to clusters 1 and 2 were not.

[0118] Figure 8 and Fig. 9 Examples of RL agent responses under different chatter scenarios are given. RL agent A, trained with a reward function that favors the most aggressive operator actions, chooses to reduce the force setpoint faster than RL agent B, which is trained with a reward function that favors both moderate and aggressive operator actions. These results are consistent with the design of the reward function and demonstrate how the choice of operator actions used for training affects each RL agent.

[0119] To demonstrate the sensitivity of the reward function to changes in performance specifications, the other parameters in the reward function can be fixed, but the maximum acceptable jitter value C can be varied. ub And train two RL agents. Table II shows the details of the reward function settings for the two RL agents.

[0120] Table II: Agent C and D parameter settings

[0121] acting <![CDATA[flutter value threshold C ub > <![CDATA[Startup time threshold T ub > Preferred Clusters C 0.5 500 3 D 1 500 3

[0122] Fig.10 and Fig.11 Examples of RL agent responses under different chatter scenarios are provided. RL agent C, trained with a lower maximum acceptable chatter value, shows more aggressive force regulation behavior than RL agent D, trained with a higher maximum acceptable chatter value. This is again consistent with the design of the reward function and shows how performance specification can affect the behavior of each RL agent even when the same data is used to train each.

[0123] Ultimately, the purpose of training an RL agent to automatically adjust the force set point is to improve the performance and consistency of the twin-roll strip casting process (or other processes that may be applicable). In order to verify the trained RL agent before implementing it on a running twin-roll continuous casting machine, the behavior of the trained RL agent was directly compared with the behavior of different human operators. Because the RL agent was not implemented on an online continuous casting machine for verification purposes, the comparison was made between the operator's past actions (where their decisions affected the force state and thus the chatter) and the actions that the RL agent would do given those specific force and chatter measurements. Nevertheless, this provides some basis for evaluating the differences between human operators and machine RL agents.

[0124] In one example, the RL agent C is compared with the human operator behavior in two different castings. Fig.12 In the example above, the operator does not reduce the force set point even though the chatter shows a strong increasing trend. Fig.13 In the , the operator began to reduce the force before the chatter began to increase. Engineers with expertise in twin-roll strip casting evaluated these comparisons and concluded that the RL agent’s behavior was superior to that of the human operator. However, it is important to note that in each case, the human operator may have considered other factors that affect the quality of the strip besides chatter, which may explain their decision making during these castings.

[0125] In some embodiments, additional caster responses are added to the reward function. For example, in some embodiments, the strip profile is measured by a gauge 44 and provided to the RL agent 92. The gauge 44 may be located between the casting rolls and the hot rolling mill 16. The strip profile parameters may include edge protrusions, edge ridges, maximum peaks relative to 100 mm, and high edge marks. Each of these may be assigned an upper limit. As with the flutter reward function, the reward function for the profile parameters is designed to assign negative rewards when the measured parameters approach their respective upper limits. These reward functions can be scaled, for example, to assign equal weights to each parameter and then summed. The sum can be scaled to ensure that the flutter reward term dominates at least during startup. An example of such a reward function is shown in Equation 4:

[0126]

[0127] Where C is chatter, bg is edge ridge, rg is edge ridge, mp is the maximum peak relative to 100 mm, and fg is the high edge flag. This results in a reward function with a chatter score and a profile score. Additional profile parameters that can be measured and included in the reward function include total thickness profile, profile crown, and repetitive periodic disturbances related to the rotational frequency of the casting rolls.

[0128] In another embodiment, each of the above embodiments can be extended to operate the continuous casting machine under steady state conditions after the startup time has passed. In some embodiments, the reward function is modified, for example, to eliminate the startup time term. For example, in an embodiment with both the chatter term and the profile term provided above, the reward function can be modified as shown in Equation 5:

[0129]

[0130] The relative weights of the flutter and contour reward functions can also be tuned.

[0131] In other embodiments, different reward functions are developed for steady-state operation, and different RL agents are trained for steady-state operation. In other embodiments, a model-based AI agent is developed and trained for steady-state operation. In some embodiments, one or more model-based controllers are operated simultaneously with a trained model-free RL agent. For example, an iterative learning controller can control a wedge to reduce periodic disturbances, as described in WO 2019 / 060717 A1, which is incorporated herein by reference, and any RL agent described herein can implement actions to reduce chatter and / or contour defects.

[0132] In the Deep Q-Network RL agent above, it was shown that the trained RL agent was able to independently adjust one set point based on a single target signal. However, one might expect to scale the RL agent to multiple target signals and reward functions containing multiple time-varying targets, as determining and applying offsets might be impractical. Furthermore, since the training process only uses a limited dataset from human recordings, an unbalanced dataset might also negatively affect the agent's behavior.

[0133] Therefore, in another embodiment of the RL agent, a modified actor-critic algorithm is provided for a control problem in which multiple control objectives are defined. Similar to the modified DQN algorithm above, the modified actor-critic algorithm trains the RL agent using only human records. It is also expected that the trained agent takes the most rewarding actions completed by some operators in similar situations. However, instead of applying an offset to the reward function, an actor-critic algorithm is adopted, which trains the policy function as a multi-class classification problem, so that a cost-sensitive method can be applied to update the policy function based on both the reward and action distributions in the data set. In addition, the method is applied to learn the set point control strategy in a twin-roll continuous casting process, and it is shown that the trained agent is able to independently make reasonable and consistent set point adjustments under given scenarios.

[0134] The actor-critic algorithm is discussed following the nomenclature provided in Table III below.

[0135] Table III: Nomenclature

[0136] Tag Description

[0137] S status

[0138] A Action

[0139] Δ(.) The difference (.) between two consecutive steps

[0140] D Training database

[0141] F Roller separation force set point value

[0142] R Instant Reward

[0143] N is the number of samples in the training database

[0144] Parameter set of the Φ-valued function

[0145] Ψ is the set of parameters of the policy function

[0146] Description of subscript / superscript

[0147] (.) k Discrete Time Index

[0148] (.) i Sample index in the database

[0149] (.) lb The lower limit of (.)

[0150] An RL agent using the actor-critic algorithm consists of two main functions, a value function and a policy function. The value (critic) function V maps a state to its value, which is defined as the expected long-term reward starting from a given state; i.e. The policy (actor) function π maps state-action pairs to probability values ​​between 0 and 1, which represent the likelihood of taking action A given a state S under the policy. The RL agent interacts with the real or simulated environment according to the policy function π and collects the current state S, action A, the state S+1 at the next time step, and the immediate reward R to update both the value function and the policy function. The immediate reward R can be calculated as shown in Equations 2, 4, or 5 above, the segmented defined reward function of Equation 9 below, or other suitable reward functions. Given a limited training data set, the value function can be evaluated as shown in Algorithm 3.

[0151] Algorithm 3: Pseudocode of the value (critic) function training process

[0152] 1: Initialization

[0153] 2: Form a training data set with samples: d i ∈D,d i ={S ki ,Aki ,S (k+1)i ,R ki},i=1,2,…N,

[0154] 3:

[0155] 4: forf = 1: number of iterations do

[0156] 5: Ford i ∈D do

[0157] 6: Calculate v i =R ki +γ(1-β i )V(R (k+1)i |Φ f-1 )

[0158] (*β i Is a binary indicator indicating the state S Ki Is it the end state of the sequence?

[0159] 7: end for

[0160] 8:

[0161] 9: end for

[0162] If any new observations are collected, it is always possible to include them into the dataset D and increase the training iterations. However, in this example, a finite training set is used and the converged value function V will be fixed and used to train the policy function. The training process of the policy function involves updating the likelihood of choosing a specific action in a given state based on the advantage value a. As shown in the advantage function in Equation 6, if the immediate reward R and the discounted value γV(S (k+1)i ) is greater than the value of the current state V(S ki ), then the advantage value a i is positive, and action A ki is considered valuable and its ki The probability of the case should increase based on how big the advantage is. However, if the advantage value is a i is negative, then the updated policy function will be ki Less likely to choose A ki When free exploration in a real or simulated environment is not available, negative advantage values ​​may increase the likelihood that the policy will choose an action that is not represented in the dataset. In other words, the consequences of that action in terms of the resulting state are unknown. To mitigate this problem, To determine the increase in π(A ki |S ki ) is likely to happen. is always positive, so actions with smaller values ​​observed in the dataset will still have a higher chance of being chosen compared to those actions that were never taken given a particular state.

[0163] Furthermore, limited training datasets may have an uneven distribution in terms of the actions taken by human agents. To effectively learn from imbalanced datasets, researchers have developed methods such as resampling, random forests, and cost-sensitive methods. When free exploration is available, resampling is not a challenge because the agent is able to interact with the environment and upsample those actions that are more uncommon. However, when free exploration is not possible, cost-sensitive methods are an effective way to implement in policy function update scenarios. We can write η(A ki ) is defined as action A ki The probability of appearing in the training data set D. The loss function depends on η(A ki )and Both. As shown in Equation 7, if an action is frequently taken in the training dataset and has a small or negative advantage value, its weight in the loss function will be low. The training process of the policy function is shown in Algorithm 4.

[0164] Algorithm 4: Pseudocode of the policy (actor) function training process

[0165] 1: Initialize learning rate a, discount factor γ

[0166] 2: Form a training data set with samples: d i ∈D,d i ={S ki ,A ki ,S (k+1)i ,R ki},i=1,2,…,N

[0167] 3: Input Φ*, the parameters of the value function from Algorithm 1

[0168] 4: Randomly initialize Ψ0, the parameter set of the policy function

[0169] 5: forf = 1: number of iterations do

[0170] 6: Loss = 0

[0171] 7: for d∈D do

[0172] 8: Calculate the advantage value

[0173] a i =R ki +γ(1-β i )V(S (k+1)i |Φ* )-V(S ki |Φ * ) (6)

[0174] (*β i is a binary indicator indicating whether state S is the end state of the sequence)

[0175] 9: Update loss

[0176]

[0177] 10: end for

[0178] 11:

[0179] 12: end for

[0180] During the startup process, the casting roll separation force set point (which will be referred to as the "force set point") is the most frequently adjusted set point. The operator adjusts the force set point in response to different profile issues as described above. Strip flutter (C), a non-negative value representing thickness variation along the casting length, is the primary factor in adjusting the force set point. In addition, the operator can adjust the force set point in response to another type of profile defect, edge spike. Unlike flutter, which describes a profile defect along the casting length, edge spike is a profile defect that exists along a cross-section of the strip. Four parameters are used to characterize different edge spike issues:

[0181] (1) Edge bulge (bg): The thickness range from the peak to the nearest minimum value in the direction away from the outer end in the edge region of 0 mm to 25 mm from the outer end. It is a non-negative value.

[0182] (2) Edge ridge (eg): The thickness range from the peak to the nearest minimum value in the direction away from the outer end in the edge region 25 mm to 50 mm from the outer end. It is a non-negative value.

[0183] (3) Maximum peak value (mp): The maximum thickness between the edge protrusion position and the edge ridge position relative to the inner end of the edge region. It is a real value.

[0184] (4) High edge flag (fg): A binary value indicating whether any edge region is thicker than the center thickness of the cross section. Fig.15 A scene is shown where the edge regions are thicker than the center region.

[0185] For illustrations of edge bumps, edge ridges, and maximum peaks, see Figures 14 and Fig.15 .

[0186] Generally, increasing the force set point increases the force applied to the belt surface and reduces the amount of semi-solid material (also called "mushy" material) between the solidified shells, which alleviates some of the edge spike problem. However, the mushy material acts as a damper, which reduces the vibration of the belt. Therefore, the reduction of mushy material results in less damping and more vibration in the belt, which in turn worsens the flutter problem. Therefore, there is a trade-off between reducing flutter and reducing the edge spike problem.

[0187] Given that modeling system dynamics during the startup process can be difficult, the reinforcement learning agent considered here is designed to learn solely by observing records of human actions and then suggest optimal set-point adjustments (values ​​and times) to the human operator. The state at time step k consists of:

[0188] S k ={C k ,ΔC k , F k ,ΔF k , bg k ,Δbg k , e.g. k ,Δeg k , fg k ,Δfg k , mp k ,Δmp k}, (8)

[0189] Where Δ(.)=(.) k -(.) k-1 is the difference between the value of the current time step and the value of the previous time step. The casting data was recorded at 1 Hz and smoothed with a 10 second moving average filter. In addition, based on the observation that human operators do not adjust the force set point more frequently than 0.2 Hz, the data can be further downsampled to 0.2 Hz to accommodate the force set point adjustment frequency used by human operators.

[0190] It is also observed that the operator usually adjusts the force set point by one of eight fixed values. Therefore, at time step k, the agent is allowed to adjust the force set point by one of these eight values ​​A k ∈{a j ,j=1,2,...,8} to adjust the force set point. Of these actions, three represent decreasing the force set point, four represent increasing the force set point, and one is defined as keeping the force set point constant. A challenging aspect of the particular problem considered is that when a human operator holds the force set point constant, it is not known whether the action was taken intentionally or whether it represents a more passive behavior due to the operator being distracted by other operational tasks. How this ambiguity is resolved is described in more detail below.

[0191] The reward function explicitly motivates the desired performance metric. Edge spikes and chatter are the main issues that can be addressed by force setpoint adjustment during the startup process. The chatter problem is characterized by the chatter parameter value and the edge spikes are characterized by the edge bump, edge ridge, and maximum peak parameters. The high edge flag parameter is not used to characterize the edge spike problem because it is a binary value and is not comparable to the other three parameters related to edge spikes. However, the high edge flag information is embedded in the state vector to provide the agent with additional information to make decisions. It is desirable to have low values ​​of chatter, edge bump, edge ridge, maximum peak and a decreasing trend of these parameters. However, once the value of a parameter decreases below a user-defined threshold, there is no need to continue reducing its value. Based on these observations, the edge spike parameter is defined as P k =max(bg k ,eg k ,mp k ), and the reward function defined for the performance objective segmentation is:

[0192]

[0193] Where W Δ(.) is the weight used to scale Δ(.) in the range [-1,1], W (.) is the weight used to scale (.) in the range [-2,2], and C lb and P lb are user defined thresholds for chatter and edge spike parameters.

[0194] In order to classify the different force setpoint regulation behaviors, a k-means clustering algorithm was used to cluster the 95 individual casting sequences in the training dataset. The start-up process of each sequence was operated by one of the six human operators. All casting sequences represented the same steel grade and strip width and were collected from the same continuous caster to prevent any behavioral variations caused by differences in casting conditions.

[0195] The force setpoint regulation behavior is characterized by the force setpoint trajectory for 500 seconds after the start of the manual mode of the force setpoint. Since there are 6 operators in the dataset of this example, the clustering is evaluated as a result of k={2,3,...,6}. The average silhouette width shows that both k=2 and k=3 have an average silhouette width above 0.5. According to Fig.17 , there is no big difference between k = 2 and k = 3. Therefore, for simplicity, the clustering results of k = 2 are used. Fig.17 a also shows the uneven distribution in the clusters. Fig.18A (Cluster 1) and Fig.18BExample of force trajectories shown in (Cluster 2), more than 70% of the sequences have Cluster 1 force behavior, which is less aggressive in both force modulation range and frequency. In addition, more than 90% of the samples in the training dataset have actions with zero force change.

[0196] Both the value function and the policy function are represented as neural networks. The choice of the neural network architecture is heuristic and is shown in Table IV. In one example, the value function has 701 learnable parameters and the policy function has 848 learnable parameters. The total number of samples used to train these two neural networks is 4594.

[0197] Table IV: Neural network architectures for value and policy functions

[0198] Value function Strategy function

[0199] Fully connected layer (12→20) Fully connected layer (12→20)

[0200] Hyperbolic tangent activation layer Leaky ReLU activation layer

[0201] Fully connected layer (20→20) Fully connected layer (20→20)

[0202] Hyperbolic tangent activation layer Leaky ReLU activation layer

[0203] Fully connected output layer (20→1) Fully connected output layer (20→8)

[0204] Softmax activation layer

[0205] During the testing, ninety-five casting sequences were used to train the reinforcement learning agent, and another eight casting sequences with the same metal grade and width conditions were used for testing. In addition to the force setpoint values ​​F, ΔF selected by the operator, the agent is provided with other defined states at each time step. At the initial time step, the agent observes the initial force setpoint value and is asked to adjust it based on the state information; the decision made by the agent affects the force setpoint value of subsequent steps. The goal of this test is to verify whether the trained agent reacts to the twin-roll continuous casting process in an intuitive way in the presence of specific defects in the steel strip. Figures 19 and 20 show two pairs (Case 1 and Case 2) of test sequence comparisons. The action force (blue curve) represents the actual force trajectory of the human operator, and the force prediction (black "+" curve) represents the force trajectory of the agent.

[0206] These comparisons demonstrate two important points. The first is that Fig.19A(Case 1) and 19B (Case 2). Case 1 exhibits a higher number of edge spikes compared to Case 2. Because the process behaves differently between the two castings, the RL agent makes different setpoint decisions; this is expected and expected. In contrast, despite how the process behaves differently, the underlying human operator trajectory is similar. The second point is that Fig. 20A (Case 3) and 20B (Case 4). When the objectively relevant parameters between the two castings were similar, the agent also made consistent decisions in both castings. This is in contrast to what a human operator would do in the actual casting, where different force setpoint decisions would be made despite similar process performance. Although these results do not represent a closed-loop interaction between the agent and the twin-roll casting process, they provide valuable insights into how the agent behaves in different casting scenarios.

[0207] In one aspect of the invention, the actor-critic algorithm is modified to better accommodate learning from multiple human experts, given the following constraints on the class of settings considered:

[0208] 1) During the algorithm training phase, the only available data is generated by human experts.

[0209] 2) The data of multiple experts are mixed in the dataset. All experts are able to stabilize the closed-loop supervisory control system.

[0210] 3) When evaluating the performance of experts based on a given criterion, the performances may not be equally preferred.

[0211] Given that the reinforcement learning agent is trained only from human data (and has no process model), the following holds:

[0212] 1) If the human expert behaves very consistently, such that the state-action mapping is 1-to-1, then the reinforcement learning agent should learn this mapping exactly.

[0213] 2) If there is an inconsistency such that multiple actions are observed to be taken in a particular state,

[0214] Then the agent should learn to pick the most preferred one.

[0215] By using the natural exponential exp(α i ) Substitution advantage α i To temporarily disable the exploration nature of the reinforcement learning algorithm, because negative α i Causes the action taken by the policy function to deviate from the action taken by the human expert a i The function exp(α i ) has the same i Same monotonicity. Therefore, if a sample has a high positive advantage, then the corresponding exp(αi ) is also high, so the sample is considered to be preferred. On the contrary, if the sample has a low positive advantage or negative advantage, its corresponding exp(α i ) becomes lower, and the sample is considered less preferred.

[0216] In addition, a deterministic policy function is expected, but due to concerns about inconsistency in the training datasets generated by multiple experts, a random policy function π(a i |s i ,Ψ h ) to characterize the conditional distribution of actions. This policy function plays the role of a sensitivity weight for handling unbalanced training datasets. The modified loss function is shown in Equation 10.

[0217]

[0218] Embed the recurrence from the previous step to improve the actor training process. Reconstruct the samples in the training dataset D so that each sample d i ={a i (-1) ,s i ,a i ,α i}, where a i (-1) is the action taken in the previous step, α i is an advantage. Since this data reconstruction is mainly used for actor training, it is considered that the fixed And the corresponding advantages have been calculated.

[0219] The policy function is also redesigned to depend on the actions at the previous step, such that:

[0220]

[0221] in, is the action taken by the policy function π under the given conditions. This is sufficient if only the teacher forcing technique is considered. However, the agent is also expected to behave more robustly, which means that the agent should also be able to tolerate the mistakes it made in the previous steps. Therefore, the augmented data is constructed as follows. Assume that sample d i is not the last step of the trajectory, then its corresponding enhanced sample is

[0222]

[0223] In each iteration, the training process first determines based on Equation 11 And based on equation 12, Then, it determines and updates the parameter set Ψ, which satisfies Equation 13. The policy function training process using the augmented dataset is shown in Algorithm 5.

[0224]

[0225] Algorithm 5: Pseudocode of the training process

[0226]

[0227]

[0228] In this example, the focus is on two set points: roll separation force and in-gauge thickness. Fig.21 As shown, the roll separation force set point directly affects the force applied to the rolls and therefore to the strip. The in-gauge thickness set point affects the casting speed; the smaller the set point, the faster the rolls. In the following, these set points are referred to as the "force" and "thickness" set points.

[0229] Surface quality and thickness profile uniformity are two major issues in steel strip manufacturing. This includes chatter, surface defects and edge spikes, and thickness profile non-uniformity. Chatter, such as Figure 1B Shown is the thickness variation along the casting length. Based on the vibration frequency, the chatter is separated into high frequency chatter and medium frequency chatter.

[0230] Edge spikes characterize thickness defects along the cross section of the ribbon, e.g. Fig.15 As shown in Figure 2. Four quantities are used to characterize the edge spike problem. They are:

[0231] 1) Edge bulge (bg): The thickness range from the peak to the nearest minimum value in the direction away from the outer end in the edge region of 0 to 25 mm from the outer end. It is a non-negative value.

[0232] 2) Edge ridge (eg): The thickness range from the peak to the nearest minimum value in the direction away from the outer end in the edge region 25 mm to 50 mm from the outer end. It is a non-negative value.

[0233] 3) Maximum peak value (mp): The maximum thickness between the edge protrusion position and the edge ridge position relative to the inner end of the edge area. It is a real value.

[0234] 4) High edge flag (fg): A binary value indicating whether any edge region is thicker than the center thickness of the cross section.

[0235] In some embodiments, the state, action, and reward functions are constructed as follows.

[0236] State: With a fixed number of state elements, we prefer to encode more information about the dynamics. Therefore, the state vector is defined as

[0237]

[0238] Among them, Ch i and Cm i is sample d i High frequency and medium frequency vibration, is the minimum thickness allowed, t i is the time relative to the time at which a human operator can begin adjusting the set point and for any element is the difference between two consecutive steps. Time and the minimum thickness allowed are also included in the state vector, since the desired strip thickness is part of the final product requirements. Any decision that results in a thickness set point less than the minimum thickness allowed should result in a penalty. As time increases, the penalty also increases.

[0239] Action: The action is simply defined as the force (F) and thickness (Th) setpoint values ​​at the next time step:

[0240]

[0241] Reward: The reward function is a function of all control objectives in the state vector, including all control objectives except t and For each element other than is considered separately below. In addition, the reward is a weighted sum of changes in all control objectives such that

[0242] r i =R(s i )=-W(s i ) T s i , (16)

[0243] Among them, W(s i ) is a piecewise linear weighted function of the state vector. When the control target xi in the state vector is lower than its threshold, the corresponding target and its change Δ(x) i The weights of both decrease. The weighting function is always non-negative, so the negative sign in front of it makes the value lower, and the decreasing trend of the control objective leads to higher rewards.

[0244] The time-dependent thickness penalty is directly encoded in the loss function as

[0245]

[0246] Notice

[0247]

[0248] in, (Ψ) is the thickness set point adjustment determined by the policy function π. Then, the thickness penalty loss J is used Th To determine the parameter set Ψ, this is simply done by replacing J in Eq. m Substitute with J defined in Equation 19 to determine.

[0249] J(d i ,Ψ)=J m (d i ,Ψ)+J Th (d i ,Ψ) (19)

[0250] As mentioned above, the training process relies solely on data generated by human experts, since no simulator is available due to the complexity of the system. However, we still want to evaluate and compare the trained agents before actual implementation. Therefore, a method is provided to evaluate the performance of the agent without a simulator. Then, the agent trained with the recursive augmentation dataset can be compared with the agent without the augmentation dataset.

[0251] Similar to sequence-to-sequence RNN, the policy function is required to be based on the K-step state trajectory {s (+1) ,s (+2) ,...,s (+K) Generate setpoint trajectory Since the policy function has recurrence from previous outputs as shown in Equation 11, the action taken by the agent at time step k should be

[0252]

[0253] And when k=1, the initial action a is given (0) .

[0254] Assume that the K-step state trajectory is generated by a set point trajectory {a (+1) ,a (+2) ,...,a (+K)} is generated. As mentioned before, if all human experts share the same consistent control policy, the agent should perfectly learn the policy, and the set point trajectories generated by human experts or agents should also be similar. However, if there is policy inconsistency, which may lead to imperfect imitation of the expert's control policy, the agent should prioritize learning from samples with higher advantages. Therefore, for each time step k, the advantage α(k) can be calculated based on Equation 22, where β (k) is a binary indicator showing whether step k is the end of the sequence. The validation loss is defined as

[0255]

[0256] This paper compares the two agents using eight unseen test sequences. The trajectory plot for one sequence is shown in detail, and the loss statistics for all eight sequences are shown and discussed. Fig. 22 The force trajectory of the agent without the augmented dataset is shown. The presented casting sequence has an increasing edge spike from the beginning of the casting sequence. Accordingly, the human expert increases the force setpoint. After about 100 seconds, the edge spike starts to decrease. The force trajectory determined by the agent starts to deviate from the actual force trajectory selected by the human expert at about 50 seconds, and the difference between the two trajectories increases with time. Accordingly, at Fig.23 The loss of force tracking increases with the sequence. The thickness determined by the agent follows the thickness chosen by the human very well, so the loss of thickness tracking remains low.

[0257] Fig.24 The force trajectory of the agent with the augmented dataset is shown. Although the agent with the augmented dataset also keeps the force set point constant at the beginning of the casting, at about 75 seconds, as the edge spikes above 1 and continues to increase, the agent begins to increase the force set point. Fig.25 , the force tracking loss still increases over time, even though the difference between the force determined by the agent and the actual force has not increased. This is because according to Equation 21, the loss is the weighted difference between the force setpoint determined by the agent and the true force. In this sequence, the advantage α (+k) increases with the increase of k. Therefore, although the tracking error remains the same, the loss increases. Table IV shows the loss statistics for all test sequences. By training with augmented data, the losses corresponding to force and thickness tracking are improved in most of the test sequences.

[0258] Table IV

[0259] Loss statistics for test sequences

[0260]

[0261] In this embodiment, recursive features are embedded to improve the performance of reinforcement learning controllers for complex supervisory control scenarios. As in other embodiments, the problem setting considers that no system model is available and the reinforcement learning algorithm should evaluate, select and learn from data of multiple human experts. An augmented dataset is iteratively constructed to perturb the output recursion, thereby enhancing the robustness of the action learning process in subsequent steps in the sequence. In the context of a supervisory control problem with a twin-roll continuous casting example, an agent trained with a recursive augmented dataset performs better in tracking favorable actions on a test sequence than an agent trained without any recursive augmented dataset.

[0262] Additional actions may also be assigned to the RL agent. For example, the RL agent may be trained to reduce periodic disturbances by controlling the wedge control of the casting rolls. Some embodiments include local temperature control of the casting rolls to control the casting roll shape and thereby the cast strip profile. See, for example, WO 2019 / 217700, which is incorporated herein by reference. In some embodiments, strip profile measurements are used in the reward function, so the RL agent may control local heating and / or cooling of the casting rolls to control the strip profile.

[0263] Actions may also be extended to other parts of the twin roll casting process equipment, including the control of the hot rolling mill 16 and the water jets 18. For example, various controls have been developed for shaping the work rolls of the hot rolling mill to reduce flatness defects. For example, work roll bending jacks have been provided to affect symmetrical changes in the center area of ​​the roll gap profile of the work roll relative to the area adjacent to the edge. Roll bending can correct symmetrical shape defects common to the center area and both edges of the strip. Moreover, force cylinders can affect asymmetrical changes in the roll gap profile on one side relative to the other side. Roll force cylinders can deflect or tilt the roll gap profile to correct shape defects in the strip that occur asymmetrically at either side of the strip, where one side is tighter than the average tensile stress on the strip and the other side is looser than the average tensile stress on the strip. In some embodiments, the RL agent is trained to provide actions to each of these controls in response to measurements of the cast strip before and / or after hot rolling the strip to reduce thickness.

[0264] Another method of controlling the shape of the work rolls (and therefore the elongation of the cast strip passing between the work rolls) is through localized segmental cooling of the work rolls. See, for example, U.S. Pat. No. 7,181,822, which is incorporated herein by reference. By controlling the local cooling of the working surfaces of the work rolls, the upper and lower work roll profiles can be controlled by the thermal expansion or contraction of the work rolls to reduce shape defects and local buckling. Specifically, the control of local cooling can be achieved by increasing the relative volume or velocity of the coolant sprayed onto the work roll surface through the nozzle, sprayed into one or more sections of the belt shape buckling area observed, resulting in a contraction of the work roll diameter of any one or both work rolls in the area, increasing the roll gap profile, and effectively reducing the elongation in the section. Conversely, by reducing the relative volume or velocity of the coolant sprayed onto the working surface of the work roll by the nozzle, the work roll diameter in the area is enlarged, the roll gap profile is reduced, and the elongation is effectively increased. Alternatively or in combination, the control of local cooling can be achieved by locally controlling the temperature or volume of water flowing through the work roll near the working surface, and internally controlling the cooling of the working surface of the work roll in the section on the work roll. In some embodiments, the RL agent is trained to provide actions to provide localized segmental cooling of the work rolls in response to a continuous casting mill metric (eg, flatness defect).

[0265] In some embodiments, the RL agent in any of the above embodiments receives reinforcement learning not only from the casting activities manually controlled by the operator, but also from the RL agent's own operation of the physical continuous casting machine. That is, in operation, the RL agent continues to learn through reinforcement learning including real-time continuous casting machine metrics in response to the RL agent's control actions, thereby improving the performance of the RL agent and the continuous casting machine.

[0266] In some embodiments, smart alarms are included to alert operators to intervene when necessary. For example, the RL agent may instruct a step change but receive an unexpected response. This may occur, for example, if a sensor fails or an actuator fails.

[0267] The functional features that enable the RL agent to effectively drive all process set points and also perform process and machine condition monitoring constitute an autonomously driven twin-roll continuous caster where the operator needs to intervene only in the presence of machine component failure or process emergency (e.g. failure of a critical refractory element).

[0268] It should be understood that any method described herein using any reinforcement learning agent as described or expected and any associated algorithm can be performed using one or more controllers, wherein the reinforcement learning agent is stored as instructions on any memory storage device. The instructions are configured to be performed (executed) using one or more processors in conjunction with a twin-roll continuous caster to control the formation of thin metal strips by twin-roll continuous casting. Any such controller and any processor and memory storage device can be arranged to communicate operably with any component of the twin-roll continuous caster as needed, including being arranged to communicate operably with any sensor and actuator. A sensor as used herein can generate a signal that can be stored in a memory storage device and used by a processor to control a specific operation of a twin-roll continuous caster as described herein. An actuator as used herein can receive a signal from a controller, a processor, or a memory storage device to adjust or change any part of a twin-roll continuous caster as described herein.

[0269] To the extent used, the terms "comprises," "comprising," and "having," or any variation thereof, as used in the claims and / or description herein, shall be deemed to indicate an open group that may include unspecified other elements. The terms "one," "an," and the singular form of a word shall be deemed to include the plural form of the same word, such that these terms mean to provide one or more somethings. The terms "at least one" and "one or more" are used interchangeably. The term "single" shall be used to refer to one and only one something. Similarly, when a specific number of things is expected, other specific integer values ​​are used, such as "two." The terms "preferably," "preferred," "preferably," "optionally," "may," and similar terms are used to indicate that the things, conditions, or steps mentioned are optional (i.e., not required) features of an embodiment. Unless otherwise stated, the range described as "between a and b" includes the values ​​of "a" and "b."

[0270] Although various improvements have been described herein with reference to specific embodiments thereof, it should be understood that such description is illustrative only and should not be construed as limiting the scope of any claimed invention. In addition, it should be understood that, unless otherwise stated, the features of any specific embodiment discussed herein may be combined with one or more features of any one or more embodiments otherwise discussed or contemplated herein.

Claims

1. A twin-roll continuous casting system, comprising: a pair of counter-rotating casting rolls having a nip therebetween and capable of conveying a cast strip downwardly from the nip; a casting roll controller configured to adjust at least one process control set point between the casting rolls in response to a control signal; a casting belt sensor capable of measuring at least one parameter of the casting belt; and a controller coupled to the strip sensor to receive strip measurement signals from the strip sensor and coupled to the casting roll controller to provide control signals to the casting roll controller, the controller comprising a reinforcement learning (RL) agent; The RL agent also includes a model-free actor-critic agent having a value function and a policy function, the RL agent having been trained on a plurality of continuous casting system operation data sets consisting of continuous casting runs performed by a plurality of different human operators.

2. The twin-roll continuous casting system according to claim 1, wherein: The RL agent also includes an advantage function that calculates an advantage value of a selected action as an immediate reward value of the selected action plus a discounted value of a subsequent state of the selected action minus a value of the current state; and The advantage value is used to train the strategy function.

3. The twin-roll continuous casting system according to claim 2, wherein: The strategy function is configured to evaluate the advantage function in such a manner that actions from the plurality of continuous casting system operation data sets having negative advantage values ​​are assigned relative to actions not found in the plurality of continuous casting system operation data sets.

4. The twin-roll continuous casting system according to claim 1, wherein: The RL agent also includes an advantage function that calculates an advantage value of a selected action as an immediate reward value of the selected action plus a discounted value of a subsequent state of the selected action minus a value of the current state; and Wherein, the natural exponent of the advantage value is used to train the policy function.

5. The twin-roll continuous casting system according to claim 1, wherein: The cast strip sensor includes a thickness gauge that measures the thickness of the cast strip at intervals across the width of the cast strip.

6. The twin-roll continuous casting system according to claim 1, wherein: The process control set points include a force set point between the casting rolls; and Wherein, the parameters of the casting belt include flutter.

7. The twin-roll continuous casting system according to claim 1, wherein: The RL agent also includes a reward function that computes an immediate reward as a piecewise defined reward function: Among them, W Δ(.) is the weight used to scale Δ(.) in the range [-1,1], W (.) is the weight used to scale (.) in the range [-2,2], and C lb and P lb are user defined thresholds for chatter and edge spike parameters.

8. The twin-roll continuous casting system of claim 1, further comprising an advantage function, wherein the advantage function calculates the advantage value as the immediate reward value of the selected action plus the discounted value of the subsequent state of the selected action minus the value of the current state; in, The immediate reward is calculated by a reward function that calculates the immediate reward as a weighted segment-defined reward function based on user-defined thresholds for the chatter and edge spike parameters.

9. The twin-roll continuous casting system according to claim 1, wherein: The at least one parameter of the cast strip includes flutter and at least one strip profile parameter.

10. The twin-roll continuous casting system according to claim 9, wherein: The at least one belt profile parameter is selected from the group consisting of edge protrusions, edge ridges, maximum peaks, and high edge marks.

11. The twin-roll continuous casting system according to claim 1, wherein: The policy function comprises a random policy function.

12. The twin-roll continuous casting system according to claim 1, wherein: The policy function includes dependencies on actions of previous steps.

13. The twin-roll continuous casting system according to claim 1, wherein: For each step in the operational dataset, the recurrence from the previous steps is embedded to improve the actor training process.

Images