A method for constructing a hot rolling scheduling hierarchical reinforcement learning model

Abstract

The application belongs to the technical field of steel production and artificial intelligence, and relates to a construction method of a hot rolling scheduling hierarchical reinforcement learning model, comprising the following steps: obtaining a single product order data set according to hot rolling production daily plan order data, and constructing a hot rolling production scheduling sample data set based on the data set; presetting a daily rolling total number, constructing a hot rolling scheduling hierarchical reinforcement learning model corresponding to all rolling processes, training the model by using a training sample data set, and generating an overall rolling order scheduling sequence; and verifying the trained model by using a verification sample order data set to obtain a final hot rolling scheduling hierarchical reinforcement learning model; the application improves model decision efficiency and training stability, effectively adapts to order structure changes and process parameter updates, ensures that the generated scheduling scheme meets process feasibility and production stability requirements, and significantly reduces the calculation cost caused by invalid action exploration.

Description

Technical Field

[0001] This invention relates to the fields of steel production technology and artificial intelligence technology, and in particular to a method for constructing a hierarchical reinforcement learning model for hot rolling scheduling. Background Technology

[0002] As a key intermediate product in the steel industry, the production planning and scheduling quality of hot-rolled slabs directly affects the stability of raw material supply and the efficiency of production cost control in downstream manufacturing industries. Against the backdrop of manufacturing transformation and upgrading, traditional rolling planning and scheduling methods dominated by manual experience are no longer adequate for modern production needs. Specifically, this manifests in several ways: Existing hot-rolling scheduling in engineering applications often relies on manual experience or programmed scheduling based on fixed rules. When dealing with large order volumes, complex order structures, and dynamic situations such as order insertions, cancellations, and temporary specification changes, these methods often require repeated manual trial scheduling and backtracking modifications, resulting in low efficiency in scheduling generation and adjustment. Particularly when handling complex process constraints such as width jumps, thickness jumps, and hardness jumps, manual scheduling struggles to achieve dynamic collaborative optimization across multiple objectives. Furthermore, hot-rolling scheduling exhibits significant strong constraints, making the scheduling problem a combination of high-dimensional combinations and strong constraints, increasing the difficulty of solving the scheduling problem.

[0003] To improve the automation level of scheduling, existing research and engineering practices have proposed various intelligent optimization approaches, including heuristic search, rule-driven optimization, and data-based adaptive strategy learning. However, existing methods still tend to suffer from problems such as excessively large action space, unstable constraint satisfaction, and insufficient dynamic adjustment response when facing strongly constrained hot rolling scheduling. These methods struggle to consistently output rolling schedules that meet process constraints and have high scheduling quality in production environments with frequently changing orders. Therefore, there is a need to provide an automated decision-making method or model for hot rolling scheduling that can effectively coordinate the main rolling material organization and transition processes while satisfying the aforementioned process constraints, enabling rapid generation and dynamic adjustment of scheduling schemes, and facilitating integration with existing hot rolling scheduling systems. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a method for constructing a hierarchical reinforcement learning model for hot rolling scheduling, in order to solve the problems of excessively large action space, unstable constraint satisfaction, insufficient dynamic adjustment response, and difficulty in generating efficient and high-quality rolling schedules in the prior art.

[0005] On one hand, embodiments of the present invention provide a method for constructing a hierarchical reinforcement learning model for hot rolling scheduling, comprising:

[0006] Based on the hot-rolled production daily planned order data, a single product order dataset is obtained; Based on the single product order dataset, a hot rolling production scheduling sample dataset is constructed, which includes an open billet order dataset, a transition material order dataset, and a main rolled material order dataset; the hot rolling production scheduling sample dataset is divided into a training sample order dataset and a validation sample order dataset according to a preset ratio. The total number of daily rolling strokes is preset, and a hierarchical reinforcement learning model for hot rolling scheduling corresponding to all rolling strokes is constructed. The training sample order dataset is used to train the hierarchical reinforcement learning model for hot rolling scheduling, and an overall rolling stroke order schedule containing open billet orders, transition material orders and main rolling material orders is generated. The trained hot rolling scheduling hierarchical reinforcement learning model was validated using the aforementioned validation sample order dataset to obtain the final hot rolling scheduling hierarchical reinforcement learning model.

[0007] Furthermore, the process of obtaining a single-product order dataset based on the hot-rolled production day plan order data includes: The hot-rolled production daily planned order data is categorized and its features are extracted to obtain a hot-rolled production daily planned order dataset; the hot-rolled production daily planned order dataset is then split into multiple single-product orders to form a single-product order dataset.

[0008] Furthermore, the hot rolling scheduling hierarchical reinforcement learning model includes at least: The main rolling mill decision unit is used to select the next main rolling mill order group from the main rolling mill order group dataset of the training sample order dataset to form the main layout of the rolling mill. The transition material coordination unit is used to select a transition material order from the transition material order dataset of the training sample order dataset or output a transition completion flag after receiving the connection request from the main rolling material decision unit, so as to achieve smooth connection between main rolling material order groups. The billet determination unit is used to determine the first rolling order for each rolling pass based on preset non-learning rules and the billet order dataset in the training sample order dataset.

[0009] Furthermore, the construction process of the main rolling mill decision unit includes: Based on the main rolling mill order group dataset in the training sample order dataset, candidate main rolling mill order group information is determined; a main rolling mill status information set is constructed based on the candidate main rolling mill order group information, the scheduling progress information during the rolling mill construction process, the reverse width control information, and the current sequence end status information. Based on the main rolling material status information set, the decision output of the main rolling material decision unit is defined as a main rolling material order group selection action, and a main rolling material action candidate set consisting of the selection actions of the current main rolling material order group is formed. Construct evaluation rules for main rolled materials; the evaluation rules for main rolled materials include: after each main rolled material order group is added to the current rolling process, calculating the evaluation signal within the rolling process based on the rolling process status after the addition, and calculating the global evaluation signal based on the overall scheduling results after all rolling processes planned for the day have been constructed; The main rolling mill decision unit is configured to select actions from the main rolling mill action candidate set by processing the main rolling mill status information set, and to receive feedback based on the main rolling mill evaluation rules to generate a rolling mill order schedule for the main rolling mill order group.

[0010] Furthermore, the main rolled material state information set is represented in the following form: ; in, This represents the set of information on the status of the main rolled material; The attribute information representing the first state of the main rolled material state information set; The attribute information representing the second state of the main rolled material state information set; The first part of the main rolled material status information set The attribute information of each state; Indicates the first One main rolled product order group; The first part of the main rolled material status information set The attribute information of each state specifically includes the main rolled product order group. The first order information Last order information and the total number of orders for the main rolled products order group ; Used to uniformly characterize main rolled product order groups The first and last order information in the database. Indicates the order sequence number; 、 、 Used to represent respectively The width, thickness, and hardness information of the corresponding main rolled product order group; Indicates the number of rolling cycles already scheduled; This indicates the number of times reverse width occurs between the current rolling mill main rolling order groups; This is a switch indicating whether reverse width is allowed between main rolled material order groups. This indicates that inverse width is allowed. This indicates that reverse bandwidth is prohibited; This indicates the width, thickness, and hardness information of the previous scheduled main rolled material order group; The candidate set of main rolling actions is represented in the following form: ; in, This represents the candidate set of main rolling actions; , and These respectively represent the selection of main rolled product order groups. , , Actions for the selected master rolled product order group; The evaluation rules for the main rolled material are expressed in the following form: ; in, This indicates the evaluation rules for main rolled materials. j Indicates the rolling sequence number; S This indicates the preset total daily rolling volume; Indicates the first j Reward and penalty functions for each rolling process; This represents a function that penalizes global resource waste.

[0011] Furthermore, the construction process of the transition material coordination unit includes: Based on the transition material order dataset in the training sample order dataset, candidate transition material information is determined; a transition material status information set is constructed based on the candidate transition material information, the candidate main rolling material order group information, the transition material resource consumption information during the rolling process construction, the inverse width related aggregation information, the inverse width allowable marker information, and the current sequence end status information. Based on the transition material status information set, the decision output of the transition material coordination unit is defined as a transition material order selection action, and a transition material action candidate set consisting of the selection actions of the current transition material order is formed. Construct transition material evaluation rules; the transition material evaluation rules include: generating connection evaluation signals between main rolling order groups based on the parameter differences between the selected transition material and the current sequence end order, the compliance with reverse width process constraints, and the consumed transition material resources; The transition material coordination unit is configured to respond to a connection request from the main rolling material decision unit by processing the transition material status information set to select an action from the transition material action candidate set and receiving feedback based on the transition material evaluation rules to generate a transition connection order sequence between main rolling material order groups.

[0012] Furthermore, the transition material state information set is represented in the following form: ; in, This represents the set of transition material status information; The attribute information representing the first state of the transition material state information set; The attribute information representing the second state of the transition material state information set; The first part represents the state information set of the transition material. The attribute information of each state; The transition material state information set represents the information from the first state to the second state. The attribute information of each state; Indicates the location number of the transition material; , , They represent the first Information on the width, thickness, and hardness of each transition material order product; Indicates the quantity of transitional materials already used in the order; This indicates the number of times reverse width occurs between the current rolling mill main rolling order groups; This represents the sum of the number of times reverse width occurs between the current rolling mill transition material and the main rolling material order group, and between transition materials; This is a switch indicating whether reverse width is allowed between main rolled material order groups; This is a switch quantity indicating whether reverse width is allowed between the order groups of transition materials and main rolled materials, or between transition materials themselves. This indicates that inverse width is allowed. This indicates that reverse bandwidth is prohibited; This indicates the width, thickness, and hardness information of the previous order for transitional materials. This indicates the width, thickness, and hardness information of the first order in the next scheduling target order group determined by the current main rolling mill decision unit; e represents an integer. The candidate set of transition material actions is represented in the following form: ; in, This represents the candidate set of transitional material actions; , , These represent the selection of the first... , , The action of a transitional material order; The evaluation rules for the transition material are expressed in the following form: ; in, Indicates the evaluation rules for transition materials; This indicates the upper limit for the number of orders that continuously use transitional materials; , Represents the reward factor, and , ; This indicates the quantity of transitional materials currently in use.

[0013] Furthermore, when making action selections, both the main rolling mill decision unit and the transition material coordination unit perform constraint screening on their respective action candidate sets according to the action screening rules, so as to obtain the rolling order scheduling sequence of the main rolling mill order group and the transition connection order sequence between the main rolling mill order groups that meet the process and scheduling constraints. The action filtering rules include at least the following: Uniqueness constraints are used to prevent main rolled material order groups or transition material orders from being repeatedly scheduled within the same scheduling cycle. Reverse width constraint is used to restrict or prohibit the arrangement in which the width of the slab for the next order is greater than the width of the slab for the previous order within the same rolling stroke.

[0014] Furthermore, the method also includes: Based on the first rolling order of each rolling cycle, the rolling order schedule of the main rolling material order group obtained after processing by the action filtering rules, and the transitional order sequence between the main rolling material order groups, an overall rolling order schedule containing the initial rolling billet order, transitional material order, and main rolling material order group is generated.

[0015] Furthermore, the training process of the hot rolling scheduling hierarchical reinforcement learning model includes: A training round is constructed based on the preset daily rolling volume; Value assessment models are configured for the main rolling material decision unit and the transition material coordination unit, respectively; in each training round, the scheduling environment and model parameters of each value assessment model are initialized, and the initial decision state is constructed using the training sample order dataset; Based on the alternating decision-making and interaction between the main rolling mill decision unit and the transition material coordination unit, interaction sample data is collected and stored in the experience cache unit corresponding to the main rolling mill decision unit and the transition material coordination unit. Based on the interaction sample data accumulated by each of the experience caching units, the training control parameters of the corresponding value assessment model are updated. Based on the scheduling decisions of the main rolling mill decision unit and the transition material coordination unit, the training control parameters of each value assessment model are configured and optimized to obtain the trained hot rolling scheduling hierarchical reinforcement learning model.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: First, unlike related technologies which suffer from low decision-making efficiency and poor training stability, this invention uses a hierarchical reinforcement learning model for hot rolling scheduling based on a two-layer collaborative decision-making mechanism. This model decomposes the complex scheduling problem into two levels: "selection of main rolling material orders" and "connection of transition materials." Through state transfer and reward feedback, it conducts collaborative and Markov decision-making, reducing the computational pressure caused by the excessively high dimensionality of the state space under single-layer decision-making, and improving the model's decision-making efficiency and training stability.

[0017] Secondly, unlike the problem of insufficient dynamic adjustment response in related technologies, the hot rolling scheduling hierarchical reinforcement learning model constructed in this invention can be trained offline based on historical sample data, and continuously adaptively learn and adjust parameters online through real-time interactive feedback. This adaptive dynamic learning and adjustment mechanism enables the hot rolling scheduling hierarchical reinforcement learning model to actively adapt to order fluctuations and process changes in the production environment, significantly improving the adaptive capability and robustness of the scheduling system, thereby achieving better and more stable long-term production scheduling performance.

[0018] Third, unlike related technologies where the action space is too large and it is difficult to generate high-quality rolling schedules, this invention uses an action space constraint mechanism based on action screening rules. In the decision-making process of the hierarchical reinforcement learning model for hot rolling scheduling, it dynamically applies action masks driven by process constraints such as uniqueness constraints and inverse width constraints to filter candidate actions in real time. This strictly limits the decision search range to the action space that meets the process requirements. As a result, it not only ensures that the final generated rolling order schedule meets the requirements of process feasibility and production stability, but also significantly reduces invalid exploration, improves the model decision efficiency and overall computational performance, and reduces computational costs.

[0019] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from the description and drawings, which are particularly pointed out. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 is a flowchart of the method for constructing a hierarchical reinforcement learning model for hot rolling scheduling according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the training process of the hot rolling scheduling hierarchical reinforcement learning model according to an embodiment of the present invention. Detailed Implementation

[0021] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0022] A specific embodiment of the present invention discloses a method for constructing a hierarchical reinforcement learning model for hot rolling scheduling, such as... Figure 1 As shown, the steps S1 to S4 are as follows: Step S1: Obtain a single product order dataset based on the hot-rolled production daily planned order data.

[0023] Hot-rolled production day planned order data refers to the data set received from the upstream production planning system to guide the specific production tasks of the hot-rolling workshop on that day. It records the complete requirements of all customer orders or internal production instructions planned to be rolled on that day. Hot-rolled production day planned order data includes: the identification number of each order, the steel grade, target width, thickness and hardness grade and other specifications required by the order, the slab number corresponding to the order and its original size and material information, the order's delivery priority or production urgency, the total weight or total length required by the order, and relevant production process routes or special quality requirements and other order data.

[0024] In practice, obtaining a single product order dataset specifically includes: first, classifying and extracting features from the hot-rolled production day plan order data. For example, features include the steel type, thickness, width, and hardness attributes required by the order, as well as the steel type, thickness, width, and hardness attributes of the slab corresponding to the order, to obtain the hot-rolled production day plan order dataset.

[0025] Secondly, the hot-rolled production day planned order dataset is split into orders. In actual production, each order consists of one or more product orders. For ease of analysis, all orders in the hot-rolled production day planned order dataset can be split into orders containing only a single product, thus forming a single-product order dataset containing multiple single-product orders. The total number of orders is denoted as […]. This provides a well-structured data foundation for the subsequent order classification process.

[0026] Step S2: Based on the single product order dataset, construct a hot rolling production scheduling sample dataset including an open billet order dataset, a transition material order dataset, and a main rolled material order dataset; divide the hot rolling production scheduling sample dataset into a training sample order dataset and a validation sample order dataset according to a preset ratio.

[0027] Specifically, it includes two stages: order classification and building a sample dataset of hot rolling production scheduling.

[0028] During implementation, in the order classification stage, all orders in the single product order dataset can be divided into three categories based on the hot rolling production daily plan requirements and the steel grade, width, thickness and hardness attributes of each order product: open-rolled billet orders, transition material orders and main rolled material orders. Furthermore, the main rolled material orders can be divided into several main rolled material order groups according to the steel grade and product attribute information of the main rolled material order products.

[0029] The initial rolling order is the first order rolled in each rolling cycle. Its correct selection can ensure the stability of the initial rolling process of each rolling cycle. The transition material order refers to the order used to connect two different main rolling material order groups, such as the order for the steel grade with the lowest hardness level. The main rolling material order refers to the rolling material order that constitutes the main body of the rolling cycle, representing the main production tasks in the daily plan.

[0030] Specifically, based on preset steel grade selection rules, orders with open-rolled billet steel grade attributes can be selected from the single-product order dataset to form an open-rolled billet order dataset. Simultaneously, orders with transitional steel grade attributes can be selected from the single-product order dataset to form a transitional steel grade order dataset. The steel grade selection rules include: pre-setting an open-rolled billet steel grade set and a transitional steel grade set; if the steel grade attribute of an order belongs to the open-rolled billet steel grade set, the order is determined to be an open-rolled billet order; if the steel grade attribute of an order belongs to the transitional steel grade set, the order is determined to be a transitional steel grade order.

[0031] When dividing the main rolled product orders, the remaining orders in the single product order dataset, excluding billet orders and transition material orders, can be regarded as main rolled product orders. They can be grouped according to their steel grade, width, thickness and hardness attributes to form a main rolled product order group dataset that includes multiple main rolled product order groups, so as to achieve efficient and continuous batch rolling in the scheduling.

[0032] The steel grades for open-rolled billet orders and transitional steel orders are typically fixed. For example, Q195 is set as the open-rolled billet steel grade, and Q235B is set as the transitional steel grade. Furthermore, the filtering of the open-rolled billet order dataset should have a width restriction; that is, an order is only included in the open-rolled billet order dataset if its width parameter falls within a preset range for open-rolled billet width. Therefore, through the above steel grade filtering rules and width restrictions, open-rolled billet orders and transitional steel orders can be distinguished and determined within a single product order dataset, thus obtaining the main rolled steel order group. The mathematical representations of the above three types of orders are as follows: The open-rolled billet order dataset is represented as follows: (1) , (2) in, This represents the dataset of open-rolled billet orders; Represented as the 1st and 2nd respectively , No. and the The attribute information for each open-rolled billet order includes at least width, thickness, and hardness information; This indicates the total number of orders for rolled billets. They represent the first Information on the width, thickness, and hardness of each rolled billet order.

[0033] The transition material order dataset is represented as follows: (3) , (4) in, This represents the transition material order dataset; , , , Represented as the 1st, 2nd, and 3rd respectively and the The attribute information of each transition material order, including width, thickness, and hardness information; This indicates the total number of orders for transitional materials; They represent the first Information on the width, thickness, and hardness of each transition material order.

[0034] The dataset of main rolled product order groups is represented as follows: (5) in, Indicates the main rolled product order group sequence; , , They are respectively represented as the 1st, 2nd and 3rd One main rolled product order group This indicates the total number of main rolled product order groups.

[0035] Therefore, in rolling process j, the main rolled product order group order sequence It can be described in the following form: , (6) , , (7) in, This indicates the main rolled product order group in rolling process j. All order information; and These represent the main rolled product order groups. The first, second, and last order information; Indicates main rolled product order group The total number of orders in the country , This indicates the total number of main rolled product order groups; y represents the order within the main rolled product order group. The location of the middle, It is the main rolled product order group The first in Order attribute information for each location, As the value of y changes You can describe the main rolled product order group. Order information at any position in the data; j represents the j-th rolling stroke. S represents the preset total daily rolling stroke; These represent orders. Information on width, thickness, and hardness.

[0036] It can be understood that the rolling stroke in the embodiments of the present invention refers to the set of continuous rolling objects corresponding to a single rolling plan under the premise of meeting the rolling process constraints and equipment conditions; for example, in the existing hot rolling plan, rolling objects are often divided according to the roll change boundary, and the rolling objects between the two work rolls before and after the change can constitute a rolling plan unit, which is the rolling stroke.

[0037] In the stage of constructing the hot-rolled production scheduling sample dataset, the three types of order data obtained above can be used to construct a hot-rolled production scheduling sample dataset containing the initial rolling billet order dataset, the transition material order dataset, and the main rolling material order group dataset, and the total number of orders is denoted as... Each order type in this dataset has width, thickness, and hardness attributes. Furthermore, this hot-rolled production scheduling sample dataset can be divided into a training sample order dataset and a validation sample order dataset according to a preset ratio (e.g., 8:2).

[0038] Therefore, the hot rolling production scheduling sample dataset constructed and divided in the above manner can provide high-quality, structured input data for subsequent hierarchical reinforcement learning models, effectively supporting the models to learn scheduling patterns during the training phase and evaluate their generalization ability and scheduling effect during the validation phase, thus laying a reliable data foundation for the practical application of the models.

[0039] Step S3: Preset the total number of daily rolling strokes, construct a hierarchical reinforcement learning model for hot rolling scheduling corresponding to all rolling strokes, and train the hierarchical reinforcement learning model for hot rolling scheduling using the training sample order dataset to generate an overall rolling stroke order schedule that includes open billet orders, transition material orders, and main rolling material order groups.

[0040] Before constructing the hierarchical reinforcement learning model for hot rolling scheduling, it is necessary to preset the total number of rolling cycles that can be completed each day, i.e., the total number of rolling cycles S per day. The total number S is determined based on the hot rolling production scheduling sample dataset and by taking into account production constraints such as the time required to change rolls and the maximum daily rolling tonnage.

[0041] It is understandable that the core of the hot rolling scheduling problem lies in rationally arranging discrete groups of open-rolling billet orders, transition material orders, and main rolling material orders into several rolling strokes, while satisfying various production process constraints and daily capacity limits, and forming an executable production sequence. Therefore, the optimization objective of this invention is to maximize the total number of orders scheduled into the rolling strokes while satisfying production constraints.

[0042] The hot rolling scheduling hierarchical reinforcement learning model constructed in this embodiment of the invention has the following input configuration: the training sample order dataset containing the open billet order dataset, the transition material order dataset, and the main rolled material order group dataset, and a preset daily total number of rolling strokes S; its output configuration is: the order scheduling sequence for each rolling stroke dynamically and autonomously generated by the value evaluation architecture based on hierarchical reinforcement learning, that is, the overall rolling stroke order scheduling sequence including open billet orders, transition material orders, and main rolled material order groups. Each output sequence corresponds to the rolling order sequence of orders within a rolling stroke, which can be directly used to guide the sequential execution of subsequent actual production.

[0043] The following details the construction process of the hierarchical reinforcement learning model for hot rolling scheduling in this invention.

[0044] Specifically, the hot rolling scheduling hierarchical reinforcement learning model includes at least: The main rolling mill decision unit is used to select the next main rolling mill order group from the main rolling mill order group dataset of the training sample order dataset, that is, to select the next main rolling mill order group from the currently available set of main rolling mill order groups, so as to form the main layout of the rolling mill. The transition material coordination unit, upon receiving a connection request from the main rolling mill decision unit, selects a transition material order from the transition material order dataset of the training sample order dataset or outputs a transition completion flag. In other words, it selects a transition material order that meets the process constraints from the currently available set of transition material orders or outputs a transition completion flag to achieve smooth connection between main rolling mill order groups; for example, smooth connection between the current end-of-rolling-cycle order and the first order of the target main rolling mill order group. This unit allows for the configuration of transition materials between main rolling mill order groups, thereby achieving process connection between adjacent main rolling mill order groups or between the unrolled billet and the main rolling mill order group, reducing reverse width risk, and controlling the consumption of transition material resources.

[0045] Preferably, the main rolling mill decision unit and the transition material coordination unit are linked through the transmission of target information and feedback of evaluation signals: the evaluation signal of the main rolling mill decision unit is used to comprehensively reflect the layout effect of the main rolling mill in a single rolling stroke and the overall utilization level of the main rolling mill; the evaluation signal of the transition material coordination unit is used to reflect the process compliance of a single transition connection and the consumption of transition material resources.

[0046] Furthermore, the hierarchical reinforcement learning model for hot rolling scheduling also includes: The billet determination unit is used to determine the first rolling order for each rolling pass based on preset non-learning rules and the billet order dataset in the training sample order dataset.

[0047] Since initial rolling orders typically meet the basic process requirements of the first rolling mill of a rolling cycle, and their key dimensional parameters are relatively stable with a limited range of options, the first rolling mill can be determined using fixed rules. Therefore, to reduce model complexity and minimize invalid exploration during training, this invention selects the first rolling order from the initial rolling order dataset using non-learning deterministic rules. These deterministic rules can be: selection based on arrival time, order number, or a preset priority table, and the selection result is written to the first rolling position of the current rolling cycle. Thus, after the first rolling mill is determined, in subsequent steps, the main rolling material decision unit and the transition material coordination unit, based on constraint feasibility screening, jointly schedule the selection of the main rolling material order group and the transition connection to obtain an overall rolling cycle order scheduling sequence that includes initial rolling orders, transition material orders, and main rolling material order groups.

[0048] Preferably, the construction process of the main rolling mill decision unit includes: Based on the main rolling mill order group dataset in the training sample order dataset, candidate main rolling mill order group information is determined; a main rolling mill status information set is constructed based on the candidate main rolling mill order group information, the scheduling progress information during the rolling mill construction process, the reverse width control information, and the current sequence end status information. Based on the main rolling material status information set, the decision output of the main rolling material decision unit is defined as a main rolling material order group selection action, and a main rolling material action candidate set consisting of the selection actions of the current main rolling material order group is formed. Construct evaluation rules for main rolled materials; the evaluation rules for main rolled materials include: after each main rolled material order group is added to the current rolling process, calculating the evaluation signal within the rolling process based on the rolling process status after the addition, and calculating the global evaluation signal based on the overall scheduling results after all rolling processes planned for the day have been constructed; The main rolling mill decision unit is configured to select actions from the main rolling mill action candidate set by processing the main rolling mill status information set, and to receive feedback based on the main rolling mill evaluation rules to generate a rolling mill order schedule for the main rolling mill order group.

[0049] The following will detail the construction process of the main rolling material status information set, main rolling material action candidate set, and main rolling material evaluation rules in the main rolling material decision unit.

[0050] First, construct a set of main rolled material status information.

[0051] To enable the main rolling mill decision-making unit to comprehensively assess the selectivity of main rolling mill order groups, scheduling progress, and reverse width risk during the rolling mill construction process, this invention organizes key process and planning information related to rolling mill construction into a main rolling mill status information set, which serves as the input to this unit. The main rolling mill status information set includes at least the following: (1) Candidate Main Rolled Material Order Group Information: This represents the set of main rolled material order groups that can still be selected and their key attributes. Each candidate main rolled material order group can correspond to a set of feature fields, which at least cover the number of orders in the group and the width, thickness, and hardness parameters of the first and last orders in the group. The candidate main rolled material order group information can form a candidate list according to a preset sorting rule, and select the first K items as input according to a fixed number K; if the number of candidates is less than K, placeholders can be used to fill in the gaps to ensure interface consistency.

[0052] (2) Scheduling progress information: used to indicate the stage position of the current daily plan. The progress information may include the number of scheduled rolling cycles, the number of orders scheduled for the current rolling cycle, or the cumulative length, so as to support the adoption of differentiated selection tendencies at different stages and avoid stage mismatch.

[0053] (3) Reverse width control information: used to reflect the current status of reverse width related process restrictions. The reverse width control information includes at least the cumulative number of reverse widths and the switch flag indicating whether reverse widths are allowed; if necessary, it may also include the remaining number of reverse widths allowed or the reverse width section flag, which is used to constrain the selection of subsequent main rolling order groups.

[0054] (4) Current sequence end status information: used to describe the end order attributes of the current rolling sequence. The end status information includes at least the width, thickness and hardness parameters of the previous order, so as to determine the feasibility of connecting with the end order and the transition cost when selecting the next main rolling material order group.

[0055] It is understood that the above candidate main rolling order group information is determined based on the main rolling order group dataset in the training sample order dataset, and the above scheduling progress information, reverse width control information and current sequence end status information are derived from the model running information during the rolling process construction.

[0056] The aforementioned main rolling material status information set can be used to simultaneously provide optional set descriptions, progress contexts, inverse width constraint statuses, and end-to-end connection criteria, thereby supporting the main rolling material decision-making unit to form stable main rolling material order group selection results under the condition of meeting process constraints.

[0057] Preferably, the main rolled material state information set is represented in the following form: (8) in, This represents the set of information on the status of the main rolled material; The attribute information representing the first state of the main rolled material state information set; The attribute information representing the second state of the main rolled material state information set; The first part of the main rolled material status information set The attribute information of each state; Indicates the first One main rolled product order group; The first part of the main rolled material status information set The attribute information of each state specifically includes the main rolled product order group. The first order information Last order information and the total number of orders for the main rolled products order group ; Used to uniformly characterize main rolled product order groups The first and last order information in the database. Indicates the order number. ; 、 、 Used to represent respectively The width, thickness, and hardness information of the corresponding main rolled product order group; Indicates the number of rolling cycles already scheduled; This indicates the number of times reverse width occurs between the current rolling mill main rolling order groups; This is a switch indicating whether reverse width is allowed between main rolled material order groups. This indicates that inverse width is allowed. This indicates that reverse bandwidth is prohibited; This indicates the width, thickness, and hardness information of the previous scheduled main rolled material order group; It is understandable that in actual production processes, a rolling stroke typically includes 2 to 3 main rolling material order groups. To ensure the scientific and reasonable setting of the allowance parameters, when designing the state space of the main rolling material decision unit, the number of main rolling material order groups can be set to twice the total number of daily rolling strokes S, depending on the actual situation. If the number of order groups is insufficient, consistency can be maintained by filling in null values.

[0058] Second, construct a candidate set of main rolling actions.

[0059] The decision output of the main rolling mill decision unit is the selection operation of each main rolling mill order group; the main rolling mill order group selection operation aims to determine the next main rolling mill order group to be arranged from the current main rolling mill action candidate set, and use it as the next segment arrangement object of the main rolling process skeleton.

[0060] The candidate set of main rolling actions is represented in the following form: (9) in, This represents the candidate set of main rolling mill actions. When an action is selected, it means that the main rolling mill order group is added to the scheduling queue as the next main rolling mill order group for the current rolling mill, and the scheduling status is updated. , and These respectively represent the selection of main rolled product order groups. , , Actions for the selected master rolling mill order group.

[0061] Preferably, to ensure that the selection of actions for outputting the main rolled material meets the process feasibility constraints, the main rolled material decision-making unit can first screen candidate main rolled material order groups using pre-set feasible action screening rules to form a list of feasible actions, and then make the final selection within the list of feasible actions. The relevant content regarding the feasible action screening rules will be explained later.

[0062] Third, establish evaluation rules for main rolled materials.

[0063] To guide the main rolling mill decision-making unit to improve the utilization level of the main rolling mill and reduce unscheduled orders while meeting the scheduling process constraints, this invention sets evaluation signals corresponding to the decision-making process for the main rolling mill decision-making unit. These evaluation signals mainly include two parts: in-rolling-stroke evaluation and global evaluation.

[0064] (1) In-stroke evaluation: During the construction of each rolling stroke, after determining that a certain main rolled material order group will be added to the current rolling stroke, the in-stroke evaluation is calculated based on the rolling stroke status after the addition. The in-stroke evaluation aims to encourage the formation of rolling strokes that meet rolling stroke constraints and have sufficient main load, and can be constructed according to the following rules: If the current rolling mill's main rolling material order quantity and cumulative length are within the preset target range, a positive evaluation will be given; If adding the main rolling mill to the main rolling mill order group causes the current rolling process to violate the relevant process constraints of the main rolling mill, or causes the cumulative quantity or length to exceed the upper limit and make it impossible to continue building, a negative evaluation will be given; Based on meeting the constraints, priority should be given to layout schemes that have a larger number of main rolling material orders and a cumulative length closer to the target value during the rolling process.

[0065] (2) Global Evaluation: After the rolling mill construction within the daily plan is completed, a global evaluation is calculated based on the overall scheduling results. The global evaluation aims to avoid situations where "the local rolling mill construction is in good condition, but there are still a large number of unscheduled orders remaining overall." Specifically, it can be constructed according to the following rules: If there are still unscheduled main rolling material order groups at the end of the scheduling period, corresponding penalties will be imposed based on the number of unscheduled orders. Based on achieving the daily production or total output target, the smaller the non-displacement, the less severe the penalty; when the non-displacement is zero or within the allowable threshold range, no penalty may be imposed, or a slight positive evaluation may be given.

[0066] During the evaluation process, to facilitate the calculation of constraints such as the cumulative length and quantity of orders for a single rolling stroke, the following sequence is used to characterize the order ranking of rolling stroke j: (10) (11) in, It is the rolling process The order sequence in It is the first Any order information during the rolling process, Indicates that the order is in the rolling process The sorting position in , Indicates rolling process The total number of orders in the country They represent the first Location orders Information on width, thickness, and hardness.

[0067] It is understandable that the above rolling sequence is a complete and executable order sequence. That is to say, the first rolling order and its related information, as well as the orders selected by the subsequent transition material coordination unit and their related information, will also be added to rolling sequence j.

[0068] The evaluation rules for the main rolled material are expressed in the following form: (12) , (13) , (14) (15) in, This indicates the evaluation rules for main rolled materials. j Indicates the rolling sequence number; S This indicates the preset total daily rolling volume; Indicates the first j Reward and penalty functions for each rolling process; This represents a function that penalizes global resource waste. Indicates the reward factor. Indicates the penalty factor; This indicates the total length of the products in the scheduled orders for rolling process j. It is the minimum value of the total length of the order that can be scheduled for rolling stroke j. It is the maximum value of the total length of the order that can be scheduled in rolling stroke j; This indicates the total number of orders that can be scheduled for rolling mill j. , It is the minimum number of orders that can be scheduled for rolling mill j. It is the maximum number of orders that can be scheduled for rolling process j; This indicates the number of scheduled orders; This indicates the order's position in rolling mill j; These represent orders. The corresponding slab width, thickness, and hardness information. Indicates order Width and thickness information; Indicates the penalty factor. This represents the total number of orders in the hot-rolled production scheduling sample dataset obtained after splitting.

[0069] Preferably, the construction process of the transition material coordination unit includes: Based on the transition material order dataset in the training sample order dataset, candidate transition material information is determined; a transition material status information set is constructed based on the candidate transition material information, the candidate main rolling material order group information, the transition material resource consumption information during the rolling process construction, the inverse width related aggregation information, the inverse width allowable marker information, and the current sequence end status information. Based on the transition material status information set, the decision output of the transition material coordination unit is defined as a transition material order selection action, and a transition material action candidate set consisting of the selection actions of the current transition material order is formed. Construct transition material evaluation rules; the transition material evaluation rules include: generating connection evaluation signals between main rolling order groups based on the parameter differences between the selected transition material and the current sequence end order, the compliance with reverse width process constraints, and the consumed transition material resources; The transition material coordination unit is configured to respond to a connection request from the main rolling material decision unit by processing the transition material status information set to select an action from the transition material action candidate set and receiving feedback based on the transition material evaluation rules to generate a transition connection order sequence between main rolling material order groups.

[0070] The following will detail the construction process of the transition material state information set, transition material action candidate set, and transition material evaluation rules in the transition material coordination unit.

[0071] First, construct a transition material state information set.

[0072] Given the relatively limited number of transition material orders and the impact of transition configuration on the transition effect between main rolling mill order groups, the transition material coordination unit, in addition to collecting information on the transition material itself, also incorporates information on scheduled rolling processes, the status information of main rolling mill order groups, and the next step connection target information defined by the upper-level main rolling mill decision-making unit. Simultaneously, key process and planning information related to "transition material selectivity judgment, connection between main rolling mill order groups, reverse width risk control, and transition material resource consumption" are integrated into a status information set to characterize the current transition connection context. The transition material status information set includes at least the following: (1) Candidate transition material information: This information is determined based on the transition material order dataset in the training sample order dataset. It is used to characterize the transition material order dataset that still has options and its key attributes, so that the transition material coordination unit can know the currently available transition materials and their characteristics. In terms of implementation, the key parameters such as the width, thickness, and hardness of each candidate transition material can be expanded into a vector according to a preset order; when the number of candidates is insufficient, placeholder padding can be used to ensure that the input dimension remains consistent.

[0073] (2) Candidate main rolling order group information: This information is determined by the main rolling order group dataset, and its function is to characterize the main rolling order group dataset that can still be selected and its key attributes; this information is consistent with the definition of candidate main rolling order group information in the aforementioned main rolling decision unit status information.

[0074] (3) Connecting target information: It is used to explicitly represent the output of the main rolling material decision unit, namely "the main rolling material order group corresponding to this transition request", and then transmit the decision result of the main rolling material decision unit to the transition material coordination unit to ensure that the transition material configuration is consistent with the main rolling material skeleton and form inter-layer collaboration.

[0075] (4) Transitional material resource consumption information: used to depict the scale of transitional material usage. This information should include at least the quantity of transitional material used in order to avoid excessive consumption of transitional material and improve overall utilization.

[0076] (5) Reverse width related aggregation information: used to quantify the cumulative status of reverse width during the connection of main rolling order groups. This information includes at least the number of reverse widths that have occurred between main rolling order groups, the reverse width related counter of the current rolling stroke or its aggregation amount, so as to continuously control the reverse width risk in subsequent transitions.

[0077] (6) Reverse width allowance flag information: As a process constraint switch quantity (Boolean quantity or 0 / 1), it is used to characterize whether reverse width is allowed under the current strategy or equipment conditions; this information can cover the global reverse width allowance switch quantity and the reverse width allowance switch quantity in the scenario of transition material and main rolling material connection.

[0078] (7) Current sequence end status information: This information describes the end status of the current rolling mill sequence. It includes at least the width, thickness, and hardness parameters of the previous scheduled order to support the assessment of the feasibility of transitioning between the candidate transition material and the current end order. This information is consistent with the definition of the current sequence end status information in the aforementioned main rolling mill decision unit status information.

[0079] Preferably, the transition material state information set is represented in the following form: (16) in, This represents the set of transition material status information; The attribute information representing the first state of the transition material state information set; The attribute information representing the second state of the transition material state information set; The first part represents the state information set of the transition material. The attribute information of each state; The transition material state information set represents the information from the first state to the second state. The attribute information of each state; Indicates the location number of the transition material; , , They represent the first Information on the width, thickness, and hardness of each transition material order product; Indicates the quantity of transitional materials already used in the order; This indicates the number of times reverse width occurs between the current rolling mill main rolling order groups; This represents the sum of the number of times reverse width occurs between the current rolling mill transition material and the main rolling material order group, and between transition materials; This is a switch indicating whether reverse width is allowed between main rolled material order groups; This is a switch quantity indicating whether reverse width is allowed between the order groups of transition materials and main rolled materials, or between transition materials themselves. This indicates that inverse width is allowed. This indicates that reverse bandwidth is prohibited; This indicates the width, thickness, and hardness information of the previous order for transitional materials. This indicates the width, thickness, and hardness information of the first order in the next scheduling target order group determined by the current main rolling mill decision unit; 'e' represents an integer; its value range is... .

[0080] Second, construct a candidate set of transitional material actions.

[0081] The decision output of the transition material coordination unit is defined as the transition material selection action; the transition material selection action is used to determine the transition material to be used for connection from the current transition material action candidate set.

[0082] The candidate set of transition material actions is represented in the following form: (17) in, This represents the candidate set of transition material actions. When a certain action is selected, it means that the transition material is added to the scheduling sequence as the next transition material for the current rolling stroke, and a status update is triggered. , , These represent the selection of the first... , , The action of a transitional material order; available Describe any first The action of a transitional material order, .

[0083] Third, establish evaluation rules for transitional materials.

[0084] The transition material evaluation rule is used to quantitatively evaluate the insertion effect of candidate transition materials and, based on this, determine the selection result of the transition material for the connection between main rolling order groups from the transition material action candidate set. Preferably, the transition material evaluation rule is related to at least the following factors: parameter differences between the candidate transition material and the current sequence end order, reverse width allowable conditions and reverse width risks, and transition material resource consumption, thereby reducing the use of transition materials and improving resource utilization efficiency while meeting process constraints.

[0085] In some implementations, the transition material coordination unit adopts a segmented evaluation mechanism: when the amount of transition material used is within a preset reasonable range, a positive evaluation is given; when the amount of transition material used exceeds a preset upper limit or causes adverse connection situations such as reverse width, a penalty is introduced to suppress excessive consumption of transition material and reduce connection risks.

[0086] The evaluation rules for the transition material are expressed in the following form: (18) in, Indicates the evaluation rules for transition materials; This indicates the upper limit for the number of orders that continuously use transitional materials; , Represents the reward factor, and , ; This indicates the quantity of transitional materials currently in use.

[0087] Preferably, when the main rolling mill decision unit and the transition material coordination unit make action selections, they both perform constraint screening on their respective action candidate sets according to the action screening rules, so as to obtain the rolling order scheduling sequence of the main rolling mill order group and the transition connection order sequence between the main rolling mill order groups that meet the process and scheduling constraints. It is understood that the aforementioned action selection rule refers to the filtering rule dynamically generated and applied to the action candidate set based on real-time process constraints during the decision-making process of the hot rolling scheduling hierarchical reinforcement learning model. Based on uniqueness constraints, inverse width constraints, and other production limitations, this rule evaluates and selects candidate actions in real time at each decision step, dynamically filtering out all actions that do not meet the constraints, thereby forming a list of actionable actions at that moment. This process ensures that the model explores and optimizes only within the feasible and effective decision space.

[0088] The action filtering rules include at least the following: (1) Uniqueness constraint, used to prevent main rolling mill order groups or transition mill orders from being repeatedly scheduled within the same scheduling cycle; for example, each order is only allowed to be scheduled once within the same scheduling cycle, that is, once an order (or order group) is selected and inserted into a certain rolling position, the order (or order group) is not allowed to be selected again in subsequent decision-making processes. The specific formula is as follows: For the main rolling mill decision-making unit, in the ranking During a single rolling stroke, if its motion space is selected... That is, selection When it is assigned to the next master rolled product order group, its status is changed to avoid being scheduled repeatedly. Rewritten as: (19) For the transition material coordination unit, in the first row During a single rolling stroke, if its motion space is selected... That is, selection As the next transitional material, in order to avoid it being scheduled repeatedly, The content is rewritten as follows: (20) (2) Reverse width constraint, used to restrict or prohibit the arrangement in the same rolling stroke where the width of the slab of the next order is greater than the width of the slab of the previous order.

[0089] During implementation, both the main rolling mill decision unit and the transition material coordination unit must satisfy the reverse width constraint. It can be understood that the "reverse width" in this embodiment of the invention refers to the fact that if the width of the later order is greater than the width of the earlier order in two adjacent orders in the same rolling stroke, then a reverse width can be said to have occurred.

[0090] The inverse width occurrence event in this embodiment can be represented by the following formula: (twenty one) (twenty two) in, Indicates a switch quantity, i.e. This indicates that a reverse bandwidth event has occurred. This indicates that no reverse sizing event has occurred; Indicates the position of the order in rolling stroke j; Indicates the first One order width, Indicates the first One order The width; This indicates the width variation margin between two adjacent orders in rolling stroke j; This represents a pre-defined, sufficiently large positive number used to linearize logical constraints; This indicates the lower limit of the width variation between two adjacent orders. This indicates the upper limit of the width variation between two adjacent orders; It is a sufficiently small positive number used to represent strict inequality relationships. The specific value of this parameter can be determined based on the slab size difference corresponding to the order. For example, if the minimum slab size difference is 1mm, then... Can be taken as .

[0091] When the rolling mill j is scheduled to reach the... When there are multiple orders, the cumulative order length is calculated using the following formula: , (twenty three) in, This indicates the scheduling of rolling mill j up to the [number]th [stage]. The order length when there are multiple orders.

[0092] Therefore, whether a reverse width event is allowed must meet the following constraints: (twenty four) in, This indicates the maximum cumulative length of orders that are allowed to have "reverse width".

[0093] The number of times a reverse width event occurs must satisfy the following constraint: (25) in, This indicates the number of reverse width events that have occurred between the main rolling mill order groups in rolling process j. This represents the sum of the number of reverse widths occurring between the transition material and the main rolled material order group, and between transition materials themselves, in rolling process j. It is the maximum number of reverse width events allowed to occur per rolling stroke.

[0094] (3) Other process constraints In addition to the aforementioned constraints, to ensure the stability of the rolling process and product quality, each rolling pass must also meet other process constraints during scheduling, such as thickness jump thresholds, hardness jump thresholds, and temperature jump thresholds. These constraints apply only to the transition material coordination unit. Within the transition material's movement space, transition material movements that do not meet the following constraints will be prohibited. The specific forms of these other process constraints are shown below: Adjacent order thickness jump constraint for rolling stroke j: (26) in, and These represent the first and second strokes of rolling process j, respectively. Orders and the The thickness of the order, It is the threshold for thickness jump between adjacent orders.

[0095] Hardness jump constraint between adjacent orders in rolling stroke j: (27) in, and These represent the first and second strokes of rolling process j, respectively. Orders and the The hardness of the order, It is the threshold for hardness jump between adjacent orders.

[0096] Temperature jump constraint between adjacent orders in rolling stroke j: (28) in, and These represent the first and second strokes of rolling process j, respectively. Orders and the The temperature of the order. It is the temperature jump threshold between adjacent orders.

[0097] To facilitate understanding of the construction process of the hot rolling scheduling hierarchical reinforcement learning model of the present invention, a specific embodiment is provided below.

[0098] For example, the main rolling mill order group dataset in this embodiment contains 10 groups, and a state space model of the main rolling mill decision unit is constructed; the main rolling mill state information set is represented in the following form: (29) Its main rolling action candidate set It shall be represented in the following form: (30) The evaluation rules for the main rolled material are set, specifically including: (31) , (32) (33) Among them, setting =1, =3, =1, 50km 100km 50, 100; The transition material order dataset in this embodiment contains 15 transition material orders, and the transition material status information set is constructed as follows: (34) The transition material action candidate set is constructed as follows: (35) The evaluation rules for transition materials are set as follows: (36) Among them, setting =1, =3, =2.

[0099] At the same time, set the relevant constraint parameters in formulas (22) to (25) in the action mask part, where , Set a safety margin ,therefore Set a limit on the number of reverse width operations. The maximum cumulative length of scheduled orders that allow for reverse width .

[0100] The thickness, hardness, and temperature jump constraints are entered into the system action mask limits. Taking an order with a thickness of 3.5mm and a hardness grade of 2 as an example, the width jump must be within 300mm, the thickness jump must be within 1.5mm, the hardness grade jump must be within 2, and the furnace exit temperature of adjacent products cannot exceed 30 degrees Celsius. This collaboratively generates the final rolling order scheduling program for this embodiment.

[0101] Furthermore, the method also includes: generating an overall rolling order schedule containing open billet orders, transitional material orders, and main rolling material order groups based on the first rolling order of each rolling stroke, the rolling order schedule of the main rolling material order group obtained after processing by the action filtering rules, and the transitional connection order sequence between the main rolling material order groups.

[0102] Specifically, the hot rolling scheduling hierarchical reinforcement learning model generates a complete rolling order scheduling sequence through the following collaborative mechanism: First, the initial rolling order is determined by the billet determination unit; then, the main rolling material decision unit and the transition material coordination unit make decisions alternately based on the corresponding state information sets. The main rolling material decision unit outputs the main rolling material order group selection action and its sequence, and the transition material coordination unit outputs the transition material sequence when process connection is required. The two iteratively advance through a closed-loop process of "receiving state input, outputting selection action, and receiving evaluation feedback" until the current rolling process is completed. Finally, the initial rolling order, each main rolling material order group and the transition connection sequence between them are automatically combined in sequence to form an overall rolling order scheduling sequence that includes the initial rolling billet order, the transition material order, and the main rolling material order group.

[0103] Therefore, by introducing the action mask corresponding to the above-mentioned process constraints during the action selection process, the present invention effectively reduces the action search space while ensuring that the generated scheduling sequence always meets the actual production process requirements in terms of width, thickness, hardness, and temperature jump, thereby improving the model training efficiency and ensuring the executability and production stability of the generated scheduling scheme.

[0104] Preferably, the training process of the hot rolling scheduling hierarchical reinforcement learning model includes: A training round is constructed based on the preset daily rolling volume; Value assessment models are configured for the main rolling material decision unit and the transition material coordination unit, respectively; in each training round, the scheduling environment and model parameters of each value assessment model are initialized, and the initial decision state is constructed using the training sample order dataset; Based on the alternating decision-making and interaction between the main rolling mill decision unit and the transition material coordination unit, interaction sample data is collected and stored in the experience cache unit corresponding to the main rolling mill decision unit and the transition material coordination unit. Based on the interaction sample data accumulated by each of the experience caching units, the training control parameters of the corresponding value assessment model are updated. Based on the scheduling decisions of the main rolling mill decision unit and the transition material coordination unit, the training control parameters of each value assessment model are configured and optimized to obtain the trained hot rolling scheduling hierarchical reinforcement learning model.

[0105] The following details the construction process of the hierarchical reinforcement learning model for hot rolling scheduling in this invention.

[0106] During the model training phase, the parameters of the main rolling mill decision unit and the transition material coordination unit can be iteratively updated, enabling these two units to output stable and feasible scheduling results under the constraints of feasible action selection rules. The training objectives include: improving the daily planned order coverage and reducing transition connection resource consumption and connection risks, while meeting process constraints such as width, thickness, and hardness, as well as reverse width control requirements.

[0107] In some implementations, the training process employs a model update method based on interactive samples. The main rolling mill decision unit and the transition mill coordination unit are each configured with a corresponding value assessment model and an experience caching unit; the two units can share some feature encoding modules or be configured independently.

[0108] The value assessment model for each unit can be implemented using a neural network. Its nonlinear mapping method and candidate selection strategy can be determined based on the training data scale and computational resource allocation. The overall training process can be found in [reference needed]. Figure 2 As shown, the specific training process is based on the training sample order dataset as input and constructs training rounds based on the preset daily rolling total. After that, through the stages of environment and model parameter initialization, scheduling decision process, and model tuning, the trained hot rolling scheduling hierarchical reinforcement learning model is obtained.

[0109] Specifically, the environment and model parameter initialization phase mainly includes the following processes: (1) Parameter initialization of the main rolling mill decision unit and the transition material coordination unit: Value assessment models are configured for the main rolling mill decision unit and the transition material coordination unit, respectively, to evaluate the value of candidate selection results and output selection tendency. The value assessment model can be implemented by a neural network, and the input interface of the value assessment model is consistent with the fields of the state information set corresponding to the main rolling mill decision unit and the transition material coordination unit in the model construction stage. The output interface of the value assessment model is consistent with the output sequence definition of the main rolling mill decision unit and the transition material coordination unit in the model construction stage.

[0110] In other words, the input dimension of this value assessment model is determined by the number of fields in the state information set, and the output dimension is determined by the number of options in the action candidate set. The intermediate structure (such as the number of hidden layers, nodes, activation methods, etc.) can be determined according to the order size and computing resource configuration. This invention is not limited to specific structural parameters. In addition, the model parameters of the above two units can be initialized in a random manner or using historical model parameters / pre-trained parameters. There are no restrictions here.

[0111] (2) Initialization of experience cache unit: Experience cache units are set up for the main rolling mill decision unit and the transition material coordination unit respectively to store interactive sample data and support batch updates. Each interactive sample data includes at least: current status information, selection result, evaluation value, next status information, and a completion flag indicating whether the current rolling / connection subprocess has ended; the specific data structure of the interactive sample data can be implemented using an array, queue, or table structure; the cache capacity can be configured according to the training sample size, and this invention is not limited to a specific value or a fixed format.

[0112] (3) Initialization of training control parameters: Training control parameters include, but are not limited to, discount weights for value evaluation, parameter update step size, batch size, sample activation threshold, model stable update cycle, and the strength of random selection used to enhance candidate coverage. These parameters can be configured based on order size, constraint strength, and computing resources; this invention is not limited to specific parameter names or values.

[0113] (4) Parameter reset during each iteration: At the start of each training round, the scheduling environment is reset to its initial state, and the state variables and flags related to the round are reset. The reset parameters include at least: the number of scheduled rolling rounds, the count of reverse width between the main rolling material order groups in the current rolling round, the count of reverse width related to transition connection, the on / off flag for whether reverse width is allowed, the consumption of transition resources, and the completion flags of the two units, etc. At the same time, the order pool is restored to the initial selectable state of this round, and the starting order of the current rolling round is selected from the starting order dataset of the starting rolling order according to the first rolling determination rule.

[0114] Furthermore, the scheduling decision-making stage mainly includes the following processes: (1) Scheduling decision-making of the main rolling mill: During the rolling process, the rolling progress-related fields in the main rolling stock status information set are first updated, and key parameters such as the order information for the first rolling mill determined in this round and the order information for the previous row are written into the status information set at the current moment, thereby constructing the decision state of the main rolling stock decision unit at time t. Subsequently, a candidate set of main rolling stock actions is generated according to the action selection rules, and the main rolling stock decision unit outputs the selection result of the next main rolling stock order group based on the action candidate set.

[0115] After selecting the main rolling mill order group, it is determined whether a transition connection is needed: that is, whether the previous row of orders corresponding to the current state (initialized as open billet orders) can be directly connected with the selected main rolling mill order group without violating the transition rules; if they cannot be directly connected, a transition connection is required. If no transition connection is needed, or the transition material coordination unit has completed the transition connection, the scheduling environment is updated, the order group corresponding to the target selection result is added to the current rolling mill, and it is determined whether to end the current rolling mill and whether to end the current training round; at the same time, the evaluation value is calculated according to the main rolling mill evaluation rules, and the interaction information (including status information, selection result, evaluation value, next status information, and completion marker, etc.) is written into the experience cache unit of the main rolling mill decision unit.

[0116] If the current rolling process has not ended, the next moment's main rolling material status information set is constructed based on the updated order pool and count status, and the main rolling material order group selection continues.

[0117] If a transition is required, the next scheduling target information determined by the main rolled material order group selection result (such as the width, thickness, hardness information of the first order in the target order group, and status fields related to the transition) is transmitted to the transition material coordination unit to initiate the transition process.

[0118] (2) Scheduling decision for transitional material coordination unit: After receiving the connection target information from the main rolling mill decision unit, the transition material coordination unit initializes or updates the transition material status information set and generates a candidate set of transition material actions based on the actionable action filtering rules. The transition material coordination unit outputs the transition material selection result or a transition completion flag based on this candidate set, and updates the scheduling environment and connection-related status fields accordingly.

[0119] If the transition has not yet been completed, the information of this interaction (including status information, selection result, evaluation value, next status information and completion mark, etc.) will be written into the experience cache unit of the transition material coordination unit, and the next transition material selection result will continue to be output until the transition completion condition is met. If the transition is completed, the completion marker is recorded and the status field after the transition is updated. The transition completion information is then sent back to the main rolling mill decision unit to proceed to the next main rolling mill order group selection process.

[0120] The selection method within the action candidate set can be deterministic selection, probabilistic selection, or selection method that introduces random perturbation; the proportion of random perturbation can be preset or dynamically adjusted as the training process progresses, and this invention is not limited to specific algorithm names or update formulas.

[0121] (3) Parameter updates during each iteration of optimization: When the accumulated interactive sample data in the experience cache unit of the main rolling mill decision unit reaches the preset activation condition, a batch of samples is extracted from its experience cache unit to update the parameters of the value assessment model of the main rolling mill decision unit. Optionally, to improve training stability, a parameter synchronization or stable update mechanism can be set and executed according to a preset period or trigger condition. The intensity of random perturbation (if any) in the action candidate set can be adjusted according to the training process.

[0122] Furthermore, the model optimization phase mainly includes: after completing several rounds of training, the training control parameters and related thresholds can be configured and adjusted to ensure that the hot rolling scheduling hierarchical reinforcement learning model stably outputs rolling order scheduling sequences that meet the preset scheduling requirements in the hot rolling scheduling environment. This significantly improves training efficiency and stability, and enhances the process feasibility and smooth connection capability of the scheduling sequence.

[0123] Step S4: Validate the trained hot rolling scheduling hierarchical reinforcement learning model using the validation sample order dataset to obtain the final hot rolling scheduling hierarchical reinforcement learning model.

[0124] During implementation, the verification process is conducted without updating the model parameters. Specifically, it uses the model parameters of the main rolling mill decision unit and the transition material coordination unit obtained during the training phase, and utilizes the rolling sequence generated under the constraints of the action selection rules from the construction phase. Based on this, it determines whether the model passes verification. The specific process includes the following steps: (1) Preparation of validation dataset: The validation sample order dataset is divided into a first validation sample dataset and a second validation sample dataset; the order feature distribution of the first validation sample dataset is basically the same as that of the aforementioned training sample order dataset, while the order feature distribution of the second validation sample dataset is different from that of the aforementioned training sample order dataset.

[0125] (2) Model invocation and scheduling sequence generation: The trained hot rolling scheduling hierarchical reinforcement learning model is invoked on the first verification sample dataset and / or the second verification sample dataset to generate the overall rolling order scheduling sequence. During the generation process, the main rolling material decision unit and the transition material coordination unit input state information according to the aforementioned state information set construction method, and output the selection result after feasibility screening of the corresponding action candidate set according to the feasible action screening rules, until a complete rolling order scheduling sequence is formed.

[0126] (3) Validation and judgment: The generated rolling order scheduling sequence is judged according to the preset scheduling requirements; the preset scheduling requirements include at least: process constraint satisfaction, rolling integrity and stability requirements. If the model can continuously output scheduling results that meet the preset scheduling requirements under the constraints of the action filtering rules on the validation dataset, the model is judged to have passed the validation, and a validated hot rolling scheduling hierarchical reinforcement learning model is obtained.

[0127] Furthermore, in some implementations, the application of the hierarchical reinforcement learning model for hot rolling scheduling mainly includes: classifying and splitting the newly collected hot rolling production daily planned order data using the model construction method of the present invention to construct a single product order dataset that conforms to the model input format; then, inputting the single product order dataset together with the preset daily rolling stroke count into the trained and validated hierarchical reinforcement learning model for hot rolling scheduling, and running the model to directly output the overall rolling stroke order schedule of the corresponding hot rolling order.

[0128] Based on the overall rolling order schedule, coordinated control instructions are issued to the hot rolling production line, sequentially driving key production equipment including but not limited to the roughing mill, finishing mill, and coiler to complete the continuous rolling and forming of each slab, ultimately producing hot-rolled steel coils, hot-rolled flats, and other hot-rolled strip steel products that meet the order requirements. The steel grade, specifications, and dimensions of the products are all consistent with the order definition.

[0129] Therefore, it can be seen that the embodiments of the present invention can achieve at least one of the following beneficial effects: First, the present invention is based on a hierarchical reinforcement learning model for hot rolling scheduling with a two-layer collaborative decision-making mechanism. It decomposes the complex scheduling problem into two levels: “selection of main rolling material orders” and “connection of transition materials”. Through state transmission and reward feedback, it conducts collaborative and Markov decision-making, which reduces the computational pressure caused by the excessively high dimensionality of the state space under single-layer decision-making and improves decision-making efficiency and training stability.

[0130] Second, the hot rolling scheduling hierarchical reinforcement learning model constructed in this invention can be trained offline based on historical sample data, and continuously adaptively learned and adjusted parameters online through real-time interactive feedback. This adaptive dynamic learning and adjustment mechanism enables the hot rolling scheduling hierarchical reinforcement learning model to actively adapt to order fluctuations and process changes in the production environment, significantly improving the adaptive capability and robustness of the scheduling system, thereby achieving better and more stable long-term production scheduling performance.

[0131] Third, the present invention employs an action space constraint mechanism based on action screening rules. During the decision-making process of the hot rolling scheduling hierarchical reinforcement learning model, it dynamically applies process constraint-driven action masks such as uniqueness constraints and inverse width constraints to filter candidate actions in real time. This strictly limits the decision search range to the action space that meets the process requirements. As a result, it not only ensures that the final generated rolling order scheduling sequence meets the requirements of process feasibility and production stability, but also significantly reduces invalid exploration, improves the model decision efficiency and overall computational performance, and reduces computational costs.

[0132] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0133] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a hierarchical reinforcement learning model for hot rolling scheduling, characterized in that, include: Based on the hot-rolled production daily planned order data, a single product order dataset is obtained; Based on the single product order dataset, construct a hot rolling production scheduling sample dataset that includes an open billet order dataset, a transition material order dataset, and a main rolled material order group dataset. According to a preset ratio, the hot rolling production scheduling sample dataset is divided into a training sample order dataset and a verification sample order dataset. The total number of daily rolling strokes is preset, and a hierarchical reinforcement learning model for hot rolling scheduling corresponding to all rolling strokes is constructed. The training sample order dataset is used to train the hierarchical reinforcement learning model for hot rolling scheduling, and an overall rolling stroke order schedule containing open billet orders, transition material orders and main rolling material orders is generated. The trained hot rolling scheduling hierarchical reinforcement learning model was validated using the aforementioned validation sample order dataset to obtain the final hot rolling scheduling hierarchical reinforcement learning model.

2. The construction method according to claim 1, characterized in that, The process of obtaining a single-product order dataset based on hot-rolled production daily planned order data includes: The hot-rolled production daily planned order data is categorized and its features are extracted to obtain a hot-rolled production daily planned order dataset; the hot-rolled production daily planned order dataset is then split into multiple single-product orders to form a single-product order dataset.

3. The construction method according to claim 1, characterized in that, The hot rolling scheduling hierarchical reinforcement learning model includes at least: The main rolling mill decision unit is used to select the next main rolling mill order group from the main rolling mill order group dataset of the training sample order dataset to form the main layout of the rolling mill. The transition material coordination unit is used to select a transition material order from the transition material order dataset of the training sample order dataset or output a transition completion flag after receiving the connection request from the main rolling material decision unit, so as to achieve smooth connection between main rolling material order groups. The billet determination unit is used to determine the first rolling order for each rolling pass based on preset non-learning rules and the billet order dataset in the training sample order dataset.

4. The construction method according to claim 3, characterized in that, The construction process of the main rolling mill decision unit includes: Based on the main rolling mill order group dataset in the training sample order dataset, candidate main rolling mill order group information is determined; a main rolling mill status information set is constructed based on the candidate main rolling mill order group information, the scheduling progress information during the rolling mill construction process, the reverse width control information, and the current sequence end status information. Based on the main rolling material status information set, the decision output of the main rolling material decision unit is defined as a main rolling material order group selection action, and a main rolling material action candidate set consisting of the selection actions of the current main rolling material order group is formed. Construct evaluation rules for main rolled materials; the evaluation rules for main rolled materials include: after each main rolled material order group is added to the current rolling process, calculating the evaluation signal within the rolling process based on the rolling process status after the addition, and calculating the global evaluation signal based on the overall scheduling results after all rolling processes planned for the day have been constructed; The main rolling mill decision unit is configured to select actions from the main rolling mill action candidate set by processing the main rolling mill status information set, and to receive feedback based on the main rolling mill evaluation rules to generate a rolling mill order schedule for the main rolling mill order group.

5. The construction method according to claim 4, characterized in that, The main rolled material status information set is represented in the following form: ； in, This represents the set of information on the status of the main rolled material; The attribute information representing the first state of the main rolled material state information set; The attribute information representing the second state of the main rolled material state information set; The first part of the main rolled material status information set The attribute information of each state; Indicates the first One main rolled product order group; The first part of the main rolled material status information set The attribute information of each state specifically includes the main rolled product order group. The first order information Last order information and the total number of orders for the main rolled products order group ; Used to uniformly characterize main rolled product order groups The first and last order information in the database. Indicates the order sequence number; 、 、 Used to represent respectively The width, thickness, and hardness information of the corresponding main rolled product order group; Indicates the number of rolling cycles already scheduled; This indicates the number of times reverse width occurs between the current rolling mill main rolling order groups; This is a switch indicating whether reverse width is allowed between main rolled material order groups. This indicates that inverse width is allowed. This indicates that reverse bandwidth is prohibited; This indicates the width, thickness, and hardness information of the previous scheduled main rolled material order group; The candidate set of main rolling actions is represented in the following form: ； in, This represents the candidate set of main rolling actions; , and These respectively represent the selection of main rolled product order groups. , , Actions for the selected master rolled product order group; The evaluation rules for the main rolled material are expressed in the following form: ； in, This indicates the evaluation rules for main rolled materials. j Indicates the rolling sequence number; S This indicates the preset total daily rolling volume; Indicates the first j Reward and penalty functions for each rolling process; This represents a function that penalizes global resource waste.

6. The construction method according to claim 5, characterized in that, The construction process of the transition material coordination unit includes: Based on the transition material order dataset in the training sample order dataset, candidate transition material information is determined; a transition material status information set is constructed based on the candidate transition material information, the candidate main rolling material order group information, the transition material resource consumption information during the rolling process construction, the inverse width related aggregation information, the inverse width allowable marker information, and the current sequence end status information. Based on the transition material status information set, the decision output of the transition material coordination unit is defined as a transition material order selection action, and a transition material action candidate set consisting of the selection actions of the current transition material order is formed. Construct transition material evaluation rules; the transition material evaluation rules include: generating connection evaluation signals between main rolling order groups based on the parameter differences between the selected transition material and the current sequence end order, the compliance with reverse width process constraints, and the consumed transition material resources; The transition material coordination unit is configured to respond to a connection request from the main rolling material decision unit by processing the transition material status information set to select an action from the transition material action candidate set and receiving feedback based on the transition material evaluation rules to generate a transition connection order sequence between main rolling material order groups.

7. The construction method according to claim 6, characterized in that, The transition material state information set is represented in the following form: ； in, This represents the set of transition material status information; The attribute information representing the first state of the transition material state information set; The attribute information representing the second state of the transition material state information set; The first part represents the state information set of the transition material. The attribute information of each state; The transition material state information set represents the information from the first state to the second state. The attribute information of each state; Indicates the location number of the transition material; , , They represent the first Information on the width, thickness, and hardness of each transition material order product; Indicates the quantity of transitional materials already used in the order; This indicates the number of times reverse width occurs between the current rolling mill main rolling order groups; This represents the sum of the number of times reverse width occurs between the current rolling mill transition material and the main rolling material order group, and between transition materials; This is a switch indicating whether reverse width is allowed between main rolled material order groups; This is a switch quantity indicating whether reverse width is allowed between the order groups of transition materials and main rolled materials, or between transition materials themselves. This indicates that inverse width is allowed. This indicates that reverse bandwidth is prohibited; This indicates the width, thickness, and hardness information of the previous order for transitional materials. This indicates the width, thickness, and hardness information of the first order in the next scheduling target order group determined by the current main rolling mill decision unit; e represents an integer. The candidate set of transition material actions is represented in the following form: ； in, This represents the candidate set of transitional material actions; , , These represent the selection of the first... , , The action of a transitional material order; The evaluation rules for the transition material are expressed in the following form: ； in, Indicates the evaluation rules for transition materials; This indicates the upper limit for the number of orders that continuously use transitional materials; , Represents the reward factor, and , ; This indicates the quantity of transitional materials currently in use.

8. The construction method according to claim 7, characterized in that, When making action selections, both the main rolling mill decision unit and the transition material coordination unit perform constraint screening on their respective action candidate sets according to the action screening rules, so as to obtain the rolling order scheduling sequence of the main rolling mill order group and the transition connection order sequence between the main rolling mill order groups that meet the process and scheduling constraints. The action filtering rules include at least the following: Uniqueness constraints are used to prevent main rolled material order groups or transition material orders from being repeatedly scheduled within the same scheduling cycle. Reverse width constraint is used to restrict or prohibit the arrangement in which the width of the slab for the next order is greater than the width of the slab for the previous order within the same rolling stroke.

9. The construction method according to claim 8, characterized in that, The method further includes: Based on the first rolling order of each rolling cycle, the rolling order schedule of the main rolling material order group obtained after processing by the action filtering rules, and the transitional order sequence between the main rolling material order groups, an overall rolling order schedule containing the initial rolling billet order, transitional material order, and main rolling material order group is generated.

10. The construction method according to claim 3, characterized in that, The training process of the hot rolling scheduling hierarchical reinforcement learning model includes: A training round is constructed based on the preset daily rolling volume; Value assessment models are configured for the main rolling material decision unit and the transition material coordination unit, respectively; in each training round, the scheduling environment and model parameters of each value assessment model are initialized, and the initial decision state is constructed using the training sample order dataset; Based on the alternating decision-making and interaction between the main rolling mill decision unit and the transition material coordination unit, interaction sample data is collected and stored in the experience cache unit corresponding to the main rolling mill decision unit and the transition material coordination unit. Based on the interaction sample data accumulated by each of the experience caching units, the training control parameters of the corresponding value assessment model are updated. Based on the scheduling decisions of the main rolling mill decision unit and the transition material coordination unit, the training control parameters of each value assessment model are configured and optimized to obtain the trained hot rolling scheduling hierarchical reinforcement learning model.

Images