Steel production operation plan generation method and device based on reinforcement learning

Abstract

The invention discloses a steel production operation plan generation method and device based on reinforcement learning. According to the method, a steel production flexible operation process is converted into a partially observable Markov decision process system; carrying out representation processing on the observation space through an attention mechanism to obtain an observation space representation; performing representation processing on the action space through a heuristic decision rule to obtain action space representation; constructing a reward function according to the total electricity utilization cost data; constructing a penalty function according to a task execution constraint of steel production and an intermediate product transportation time constraint; constructing a plurality of deep neural network models with random parameters, performing iterative updating until a set number of iterations is reached, and selecting the model with the highest total return as a target deep neural network model; and generating a steel production operation plan through the target deep neural network model. According to the method, the system observation space dimension and the system action space dimension can be reduced, the search space is effectively expanded, and the model training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of steel production operation optimization, and in particular to a method and device for generating a steel production operation plan based on reinforcement learning. Background Art

[0002] With the advancement of low-carbon transformation, the power supply environment faced by the steel industry is becoming increasingly complex: local wind and solar and other renewable energy sources are highly intermittent, random and volatile, while the price and carbon emission factors of grid electricity also change with time and are highly uncertain. In this context, the traditional steel industry's continuous and stable production method with the goal of minimizing completion time has been difficult to adapt to actual needs. Therefore, it is necessary to develop flexible operation technology for steel production, adjust production plans in time according to changes in the power supply side, match the availability of local renewable energy and the low-price and low-emission periods of grid electricity, and reduce electricity energy consumption costs, improve the utilization rate of renewable energy, and reduce carbon emissions on the premise of completing production tasks on time.

[0003] At present, the methods and technologies used in the industry to achieve flexible operation of steel production mainly include two categories: methods and technologies based on mathematical programming and methods and technologies based on reinforcement learning.

[0004] Existing methods and technologies based on mathematical programming have two main disadvantages: First, it is necessary to represent the strongly nonlinear steel production process with a simplified mathematical model, such as modeling it as a mixed integer linear programming problem, which introduces model errors and causes difficult-to-quantify errors between the calculated optimal operation plan and the actual optimal operation plan; second, the mathematical programming model for steel production is large in scale, contains a large number of integer variables, and its solution time increases exponentially with the expansion of the problem scale, making it difficult to adapt to the need for steel production to be flexibly adjusted in real time according to external conditions. Reinforcement learning can largely overcome the above problems of methods and technologies based on mathematical programming.

[0005] However, reinforcement learning also faces three problems in the training of flexible operation models for steel production: first, the system observation space is too large; second, the system action space is too large; and third, the gradient-based search method is prone to fall into local optimality. These three problems limit the application of existing reinforcement learning methods and technologies in the flexible operation of steel production.

[0006] Therefore, how to invent a method for generating steel production operation plans based on reinforcement learning, reduce the dimensions of the system observation space and the system action space, effectively expand the search space, and improve the efficiency of model training has become an urgent problem to be solved. Summary of the invention

[0007] To this end, the present invention provides a method and device for generating a steel production operation plan based on reinforcement learning, which can reduce the dimensions of the system observation space and the system action space, effectively expand the search space, reduce the impact of falling into local optimality, and improve model training efficiency.

[0008] In order to achieve the above object, the present invention provides the following technical solution: a method for generating a steel production operation plan based on reinforcement learning, comprising:

[0009] Transform the steel production flexible operation process into a partially observable Markov decision process system;

[0010] The observation space in the partially observable Markov decision process system is represented and processed by an attention mechanism to obtain an observation space representation;

[0011] The action space in the partially observable Markov decision process system is characterized by using a heuristic decision rule to obtain an action space representation;

[0012] constructing a reward function for the partially observable Markov decision process system based on total electricity cost data during the flexible operation of steel production;

[0013] According to the task execution constraints of steel production and the transportation time constraints of intermediate products, a penalty function of the partially observable Markov decision process system is constructed;

[0014] Constructing several deep neural network models with random parameters, inputting the observation space representation into several of the deep neural network models, processing through the deep neural network models, and outputting the action space representation; calculating the action space representation through the reward function and the penalty function to obtain the total return of the partially observable Markov decision process system; updating the parameters of several of the deep neural network models according to the total return; iteratively updating the several deep neural network models after parameter update until the set number of iterations is reached, and selecting the deep neural network model with the highest total return as the target deep neural network model;

[0015] A steel production operation plan is generated through the target deep neural network model.

[0016] As a preferred solution of a steel production operation plan generation method based on reinforcement learning, the expression of the partially observable Markov decision process system is:

[0017]

[0018] In the formula, is the system state set; is the system action set; Ω is the system observation space; T is the system transfer function; O is the system observation function, and R is the system reward set.

[0019] As a preferred solution for a steel production operation plan generation method based on reinforcement learning, in the process of characterizing the observation space in the partially observable Markov decision process system through the attention mechanism: the observation space is processed in sequence by the QKV layer, the attention layer Softmax function, the first linear layer, the RELU activation layer and the second linear layer to obtain the observation space representation.

[0020] As a preferred solution of a method for generating a steel production operation plan based on reinforcement learning, the expression of the reward function is:

[0021] r t =((price) t *max[0,(π EL,t -E wind,t )]

[0022] In the formula, r t is the electricity cost of the system at time t; (price) t is the external electricity price at time t; π EL，t is the total power consumption at time t; E wind,t is the total amount of wind power generated in the steel plant at time t; the max function indicates that the additional benefits can be brought by not considering the wind power generation feeding back to the grid.

[0023] As a preferred solution of a method for generating a steel production operation plan based on reinforcement learning, the expression of the penalty function is:

[0024]

[0025] In the formula, c is the number of the considered constraint; C is the set of all considered constraints; rewrite all constraints into inequalities less than or equal to, the left side of the inequality includes all variable related terms, and the right side of the inequality is all constants, then Φ c is the value on the left side of the inequality after substituting the values ​​of the relevant variables in the action space, is the sum of all constant terms on the right side of the inequality.

[0026] The present invention also provides a device for generating a steel production operation plan based on reinforcement learning, based on the above method for generating a steel production operation plan based on reinforcement learning, comprising:

[0027] System transformation module, used to transform the steel production flexible operation process into a partially observable Markov decision process system;

[0028] An observation space representation processing module is used to represent the observation space in the partially observable Markov decision process system through an attention mechanism to obtain an observation space representation;

[0029] An action space representation processing module is used to characterize the action space in the partially observable Markov decision process system through a heuristic decision rule to obtain an action space representation;

[0030] A reward function building module, used for building a reward function of the partially observable Markov decision process system according to the total electricity cost data in the flexible operation process of steel production;

[0031] A penalty function construction module is used to construct a penalty function of the partially observable Markov decision process system according to the task execution constraints of steel production and the intermediate product transportation time constraints;

[0032] A target deep neural network model acquisition module is used to construct several deep neural network models with random parameters, input the observation space representation into several deep neural network models, process them through the deep neural network models, and output the action space representation; calculate the action space representation through the reward function and the penalty function to obtain the total reward of the partially observable Markov decision process system; update the parameters of several deep neural network models according to the total reward; iteratively update the several deep neural network models after parameter update until the set number of iterations is reached, and select the deep neural network model with the highest total reward as the target deep neural network model;

[0033] A steel production operation plan generation module is used to generate a steel production operation plan through the target deep neural network model.

[0034] As a preferred solution of a steel production operation plan generation device based on reinforcement learning, in the system conversion module, the expression of the partially observable Markov decision process system is:

[0035]

[0036] In the formula, is the system state set; is the system action set; Ω is the system observation space; T is the system transfer function; O is the system observation function, and R is the system reward set.

[0037] As a preferred solution for a steel production operation plan generation device based on reinforcement learning, in the observation space representation processing module, in the process of characterizing the observation space in the partially observable Markov decision process system through the attention mechanism: the observation space is processed in sequence by the QKV layer, the attention layer Softmax function, the first linear layer, the RELU activation layer and the second linear layer to obtain the observation space representation.

[0038] As a preferred solution of a steel production operation plan generation device based on reinforcement learning, in the reward function construction module, the expression of the reward function is:

[0039] r t =((price) t *max[0,(π EL,t -E wind,t )]

[0040] In the formula, r t is the electricity cost of the system at time t; (price) t is the external electricity price at time t; π EL，t is the total power consumption at time t; E wind,t is the total amount of wind power generated in the steel plant at time t; the max function indicates that the additional benefits can be brought by not considering the wind power generation feeding back to the grid.

[0041] As a preferred solution of a steel production operation plan generation device based on reinforcement learning, in the penalty function construction module, the expression of the penalty function is:

[0042]

[0043] In the formula, c is the number of the considered constraint; C is the set of all considered constraints; rewrite all constraints into inequalities less than or equal to, the left side of the inequality includes all variable related terms, and the right side of the inequality is all constants, then Φ c is the value on the left side of the inequality after substituting the values ​​of the relevant variables in the action space, is the sum of all constant terms on the right side of the inequality.

[0044] The present invention has the following advantages: converting the flexible operation process of steel production into a partially observable Markov decision process system; characterizing the observation space in the partially observable Markov decision process system through an attention mechanism to obtain an observation space characterization; characterizing the action space in the partially observable Markov decision process system through a heuristic decision rule to obtain an action space characterization; constructing a reward function of the partially observable Markov decision process system based on total electricity cost data in the flexible operation process of steel production; constructing a penalty function of the partially observable Markov decision process system based on task execution constraints of steel production and transportation time constraints of intermediate products; constructing several A deep neural network model with random parameters, inputting the observation space representation into several of the deep neural network models, processing through the deep neural network models, and outputting the action space representation; calculating the action space representation through the reward function and the penalty function to obtain the total return of the partially observable Markov decision process system; updating the parameters of several of the deep neural network models according to the total return; iteratively updating the several deep neural network models after parameter update until the set number of iterations is reached, selecting the deep neural network model with the highest total return as the target deep neural network model; and generating a steel production operation plan through the target deep neural network model. The present invention adopts a system observation space representation technology based on an attention mechanism, which significantly reduces the dimension of the system observation space, effectively extracts key observation information, and improves the model training efficiency. The present invention adopts a system action space representation technology based on heuristic decision rules, which significantly reduces the dimension of the system action space, effectively extracts reasonable action rules, and improves the model training efficiency. The present invention adopts a model search technology of a separable natural evolution strategy, which effectively expands the search space and reduces the impact of falling into a local optimum. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0046] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0047] Figure 1 This is a schematic flow chart of a method for generating a steel production operation plan based on reinforcement learning provided in Example 1 of the present invention;

[0048] Figure 2 This is a schematic diagram of the attention mechanism process in a method for generating a steel production operation plan based on reinforcement learning provided in Example 1 of the present invention;

[0049] Figure 3 A schematic diagram showing a comparison of the effects of the present invention and other technologies in a possible embodiment provided in Embodiment 1 of the present invention;

[0050] Figure 4 This is a schematic diagram of the architecture of a steel production operation plan generation device based on reinforcement learning provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0051] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Example 1

[0053] See also Figure 1 Embodiment 1 of the present invention provides a method for generating a steel production operation plan based on reinforcement learning, comprising the following steps:

[0054] S1. Convert the flexible operation process of steel production into a partially observable Markov decision process system;

[0055] S2. Characterizing the observation space in the partially observable Markov decision process system through an attention mechanism to obtain an observation space representation;

[0056] S3, characterizing the action space in the partially observable Markov decision process system by using a heuristic decision rule to obtain an action space representation;

[0057] S4. constructing a reward function of the partially observable Markov decision process system based on the total electricity cost data during the flexible operation of steel production;

[0058] S5. constructing a penalty function of the partially observable Markov decision process system according to the task execution constraints of steel production and the transportation time constraints of intermediate products;

[0059] S6, constructing several deep neural network models with random parameters, inputting the observation space representation into several of the deep neural network models, processing through the deep neural network models, and outputting the action space representation; calculating the action space representation through the reward function and the penalty function to obtain the total reward of the partially observable Markov decision process system; updating the parameters of several of the deep neural network models according to the total reward; iteratively updating the several deep neural network models after parameter update until the set number of iterations is reached, and selecting the deep neural network model with the highest total reward as the target deep neural network model;

[0060] S7. Generate a steel production operation plan through the target deep neural network model.

[0061] In this embodiment, in step S1, the steel production flexible operation process is converted into a partially observable Markov decision process system;

[0062] Specifically, the expression of the partially observable Markov decision process system is:

[0063]

[0064] In the formula, is the system state set; is the system action set; Ω is the system observation space; T is the system transfer function; O is the system observation function, and R is the system reward set.

[0065] In this embodiment, in step S2, the observation space in the partially observable Markov decision process system is characterized by an attention mechanism to obtain an observation space representation;

[0066] Specifically, Figure 2 As shown, the observation space is processed in sequence by the QKV layer, the attention layer Softmax function, the first linear layer, the RELU activation layer and the second linear layer to obtain the representation of the observation space.

[0067] In this embodiment, in step S3, the action space in the partially observable Markov decision process system is characterized by using a heuristic decision rule to obtain an action space representation;

[0068] Specifically, the heuristic decision rule is shown in Table 1:

[0069] Heuristic rule number Heuristic rule description 1 Prioritize batches that require the shortest processing time 2 Prioritize batches that require the longest processing time 3 Prioritize batches with the shortest remaining processing time 4 Prioritize batches with the longest remaining processing time 5 Prioritize batches with the longest waiting time 6 Prioritize batches with the shortest waiting time 7 Byes (no batches processed)

[0070] Table 1 Heuristic decision rules for characterizing the system action space

[0071] In this embodiment, in step S4, a reward function of the partially observable Markov decision process system is constructed based on the total electricity cost data during the flexible operation of steel production;

[0072] Specifically, the expression of the reward function is:

[0073] r t =((price) t *max[0,(π EL,t -E wind,t )]

[0074] In the formula, r t is the electricity cost of the system at time t; (price) t is the external electricity price at time t; π EL，t is the total power consumption at time t; E wind,t is the total amount of wind power generated in the steel plant at time t; the max function indicates that the additional benefits can be brought by not considering the wind power generation feeding back to the grid.

[0075] In this embodiment, in step S5, a penalty function of the partially observable Markov decision process system is constructed according to the task execution constraints of steel production and the intermediate product transportation time constraints;

[0076] Specifically, considering the task execution constraints of steel production and the transportation time constraints of intermediate products, the penalty function of the system is calculated, added to the reward function, and then the inverse is taken to obtain the total reward of the system.

[0077] Wherein, the expression of the penalty function is:

[0078]

[0079] In the formula, c is the number of the considered constraint; C is the set of all considered constraints; rewrite all constraints into inequalities less than or equal to, the left side of the inequality includes all variable related terms, and the right side of the inequality is all constants, then Φ c is the value on the left side of the inequality after substituting the values ​​of the relevant variables in the action space, is the sum of all constant terms on the right side of the inequality.

[0080] In this embodiment, in step S6, several deep neural network models with random parameters are constructed, the observation space representation is input into several deep neural network models, and the action space representation is output after being processed by the deep neural network models; the action space representation is calculated by the reward function and the penalty function to obtain the total reward of the partially observable Markov decision process system; the parameters of several deep neural network models are updated according to the total reward; the several deep neural network models after parameter update are iteratively updated until the set number of iterations is reached, and the deep neural network model with the highest total reward is selected as the target deep neural network model;

[0081] Specifically, a large number of deep neural network models (1500) with random parameters are generated. For each deep neural network model: first input the observation space of the system obtained by step S2, and output the action space represented by step S3; then use the obtained action space representation as input, and calculate the total return of the system through the reward function and penalty function established by steps S4 and S5; update the parameters of the deep neural network model according to the total return, and repeat this process 100 times. Finally, screen all the deep neural network models, and use the deep neural network with the highest total return as the target deep neural network model.

[0082] In this embodiment, in step S7, a steel production operation plan is generated by the target deep neural network model.

[0083] Specifically, according to the production operation plan objectives, a steel production operation plan is generated through a target deep neural network model.

[0084] In a possible embodiment, a comparison example of the effects of the present invention and other technologies is provided as follows:

[0085] like Figure 3 As shown, the figure shows the cumulative electricity costs of the steel plant in various periods of the day. It can be seen that compared with the traditional gradient descent search technology, since the present invention adopts a separable natural evolution strategy for search, the final cumulative electricity cost is greatly reduced (a decrease of about 45%). Compared with the direct matching technology, the final cumulative electricity cost of this technology using heuristic decision rules is comparable, but the training time is about 33% faster. Compared with the results of the mathematical programming method, the final cumulative electricity cost of the present invention is slightly higher (less than 5% higher), but the mathematical programming method has a long calculation time and linearly simplifies the actual system, making it difficult to apply to the real-time operation control of the actual system.

[0086] In summary, the present invention transforms the flexible operation process of steel production into a partially observable Markov decision process system; characterizes the observation space in the partially observable Markov decision process system through an attention mechanism to obtain an observation space characterization; characterizes the action space in the partially observable Markov decision process system through a heuristic decision rule to obtain an action space characterization; constructs a reward function of the partially observable Markov decision process system based on the total electricity cost data in the flexible operation process of steel production; constructs a penalty function of the partially observable Markov decision process system based on the task execution constraints of steel production and the transportation time constraints of intermediate products; constructs several random Markov decision process systems. The deep neural network model of machine parameters, inputs the observation space representation into several deep neural network models, processes through the deep neural network models, and outputs the action space representation; calculates the action space representation through the reward function and the penalty function to obtain the total return of the partially observable Markov decision process system; updates the parameters of several deep neural network models according to the total return; iteratively updates the several deep neural network models after parameter update until the set number of iterations is reached, and selects the deep neural network model with the highest total return as the target deep neural network model; generates a steel production operation plan through the target deep neural network model. The present invention adopts a system observation space representation technology based on the attention mechanism, which significantly reduces the dimension of the system observation space, effectively extracts key observation information, and improves the model training efficiency. The present invention adopts a system action space representation technology based on heuristic decision rules, which significantly reduces the dimension of the system action space, effectively extracts reasonable action rules, and improves the model training efficiency. The present invention adopts a model search technology of separable natural evolution strategy, which effectively expands the search space and reduces the impact of falling into local optimality.

[0087] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0088] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] Example 2

[0090] See also Figure 4 Embodiment 2 of the present invention further provides a device for generating a steel production operation plan based on reinforcement learning, comprising:

[0091] System transformation module 001, used to transform the steel production flexible operation process into a partially observable Markov decision process system;

[0092] An observation space representation processing module 002 is used to represent the observation space in the partially observable Markov decision process system through an attention mechanism to obtain an observation space representation;

[0093] An action space representation processing module 003 is used to represent the action space in the partially observable Markov decision process system through a heuristic decision rule to obtain an action space representation;

[0094] A reward function construction module 004 is used to construct a reward function of the partially observable Markov decision process system according to the total electricity cost data in the flexible operation process of steel production;

[0095] A penalty function construction module 005 is used to construct a penalty function of the partially observable Markov decision process system according to the task execution constraints of steel production and the intermediate product transportation time constraints;

[0096] The target deep neural network model acquisition module 006 is used to construct several deep neural network models with random parameters, input the observation space representation into several deep neural network models, process them through the deep neural network models, and output the action space representation; calculate the action space representation through the reward function and the penalty function to obtain the total reward of the partially observable Markov decision process system; update the parameters of several deep neural network models according to the total reward; iteratively update the several deep neural network models after parameter update until the set number of iterations is reached, and select the deep neural network model with the highest total reward as the target deep neural network model;

[0097] The steel production operation plan generation module 007 is used to generate a steel production operation plan through the target deep neural network model.

[0098] In this embodiment, in the system conversion module 001, the expression of the partially observable Markov decision process system is:

[0099]

[0100] In the formula, is the system state set; is the system action set; Ω is the observation space of the system; T is the transfer function of the system; O is the observation function of the system, and R is the reward set of the system.

[0101] In this embodiment, in the observation space representation processing module 002, in the process of characterizing the observation space in the partially observable Markov decision process system through the attention mechanism: the observation space is processed in sequence by the QKV layer, the attention layer Softmax function, the first linear layer, the RELU activation layer and the second linear layer to obtain the observation space representation.

[0102] In this embodiment, in the reward function construction module 004, the expression of the reward function is:

[0103] r t =((price) t *max[0,(π EL,t -E wind,t )]

[0104] In the formula, r t is the electricity cost of the system at time t; (price) t is the external electricity price at time t; π EL，t is the total power consumption at time t; E wind,t is the total amount of wind power generated in the steel plant at time t; the max function indicates that the additional benefits can be brought by not considering the wind power generation feeding back to the grid.

[0105] In this embodiment, in the penalty function construction module 005, the expression of the penalty function is:

[0106]

[0107] In the formula, c is the number of the considered constraint; C is the set of all considered constraints; rewrite all constraints into inequalities less than or equal to, the left side of the inequality includes all variable related terms, and the right side of the inequality is all constants, then Φ c is the value on the left side of the inequality after substituting the values ​​of the relevant variables in the action space, is the sum of all constant terms on the right side of the inequality.

[0108] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0109] Example 3

[0110] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code for a method for generating a steel production operation plan based on reinforcement learning is stored. The program code includes instructions for executing embodiment 1 or any possible implementation method of a method for generating a steel production operation plan based on reinforcement learning.

[0111] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0112] Example 4

[0113] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0114] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute a method for generating a steel production operation plan based on reinforcement learning in Example 1 or any possible implementation thereof.

[0115] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0116] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0117] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing system, they can be concentrated on a single computing system, or distributed on a network composed of multiple computing systems, and optionally, they can be implemented by a program code executable by a computing system, so that they can be stored in a storage system and executed by the computing system, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0118] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for generating a steel production operation plan based on reinforcement learning, characterized in that: include: Transform the steel production flexible operation process into a partially observable Markov decision process system; The observation space in the partially observable Markov decision process system is represented and processed by an attention mechanism to obtain an observation space representation; The action space in the partially observable Markov decision process system is characterized by using a heuristic decision rule to obtain an action space representation; constructing a reward function for the partially observable Markov decision process system based on total electricity cost data during the flexible operation of steel production; According to the task execution constraints of steel production and the transportation time constraints of intermediate products, a penalty function of the partially observable Markov decision process system is constructed; Constructing several deep neural network models with random parameters, inputting the observation space representation into the several deep neural network models, processing through the deep neural network models, and outputting the action space representation; calculating the action space representation through the reward function and the penalty function to obtain the total reward of the partially observable Markov decision process system; and updating the parameters of the several deep neural network models according to the total reward; Iteratively updating the several deep neural network models after parameter update until a set number of iterations is reached, and selecting the deep neural network model with the highest total return as the target deep neural network model; A steel production operation plan is generated through the target deep neural network model.

2. The method for generating a steel production operation plan based on reinforcement learning according to claim 1, characterized in that: The expression of the partially observable Markov decision process system is: In the formula, is the system state set; is the system action set; Ω is the system observation space; T is the system transfer function; O is the system observation function, and R is the system reward set.

3. The method for generating a steel production operation plan based on reinforcement learning according to claim 2, characterized in that: In the process of characterizing the observation space in the partially observable Markov decision process system through the attention mechanism: the observation space is processed in sequence by the QKV layer, the attention layer Softmax function, the first linear layer, the RELU activation layer and the second linear layer to obtain the observation space representation.

4. The method for generating a steel production operation plan based on reinforcement learning according to claim 3, characterized in that: The expression of the reward function is: r t =((price) t *max[0,(π EL,t -E wind,t )] In the formula, r t is the electricity cost of the system at time t; (price) t is the external electricity price at time t; π EL，t is the total power consumption at time t; E wind,t is the total amount of wind power generated in the steel plant at time t; the max function indicates that the additional benefits can be brought by not considering the wind power generation feeding back to the grid.

5. The method for generating a steel production operation plan based on reinforcement learning according to claim 4, characterized in that: The penalty function is expressed as: In the formula, c is the number of the considered constraint; C is the set of all considered constraints; rewrite all constraints into inequalities less than or equal to, the left side of the inequality includes all variable related terms, and the right side of the inequality is all constants, then Φ c is the value on the left side of the inequality after substituting the values ​​of the relevant variables in the action space, is the sum of all constant terms on the right side of the inequality.

6. A steel production operation plan generation device based on reinforcement learning, using a steel production operation plan generation method based on reinforcement learning according to any one of claims 1 to 5, characterized in that: include: System transformation module, used to transform the steel production flexible operation process into a partially observable Markov decision process system; An observation space representation processing module is used to represent the observation space in the partially observable Markov decision process system through an attention mechanism to obtain an observation space representation; An action space representation processing module is used to characterize the action space in the partially observable Markov decision process system through a heuristic decision rule to obtain an action space representation; A reward function building module, used for building a reward function of the partially observable Markov decision process system according to the total electricity cost data in the flexible operation process of steel production; A penalty function construction module is used to construct a penalty function of the partially observable Markov decision process system according to the task execution constraints of steel production and the intermediate product transportation time constraints; A target deep neural network model acquisition module is used to construct several deep neural network models with random parameters, input the observation space representation into several of the deep neural network models, process them through the deep neural network models, and output the action space representation; calculate the action space representation through the reward function and the penalty function to obtain the total reward of the partially observable Markov decision process system; and update the parameters of several of the deep neural network models according to the total reward; Iteratively updating the several deep neural network models after parameter update until a set number of iterations is reached, and selecting the deep neural network model with the highest total return as the target deep neural network model; A steel production operation plan generation module is used to generate a steel production operation plan through the target deep neural network model.

7. The device for generating a steel production operation plan based on reinforcement learning according to claim 6, characterized in that: In the system conversion module, the expression of the partially observable Markov decision process system is: In the formula, is the system state set; is the system action set; Ω is the system observation space; T is the system transfer function; O is the system observation function, and R is the system reward set.

8. The device for generating a steel production operation plan based on reinforcement learning according to claim 7, characterized in that: In the observation space representation processing module, in the process of characterizing the observation space in the partially observable Markov decision process system through the attention mechanism: the observation space is processed in sequence by the QKV layer, the attention layer Softmax function, the first linear layer, the RELU activation layer and the second linear layer to obtain the observation space representation.

9. The device for generating a steel production operation plan based on reinforcement learning according to claim 8, characterized in that: In the reward function construction module, the expression of the reward function is: r t =((price) t *max[0,(π EL,t -E wind,t )] In the formula, r t is the electricity cost of the system at time t; (price) t is the external electricity price at time t; π EL，t is the total power consumption at time t; E wind,t is the total amount of wind power generated in the steel plant at time t; the max function indicates that the additional benefits can be brought by not considering the wind power generation feeding back to the grid.

10. The device for generating a steel production operation plan based on reinforcement learning according to claim 9, characterized in that: In the penalty function construction module, the expression of the penalty function is: In the formula, c is the number of the considered constraint; C is the set of all considered constraints; rewrite all constraints into inequalities less than or equal to, the left side of the inequality includes all variable related terms, and the right side of the inequality is all constants, then Φ c is the value on the left side of the inequality after substituting the values ​​of the relevant variables in the action space, is the sum of all constant terms on the right side of the inequality.

Images