Continuous casting process parameter optimization method and device based on deep reinforcement learning

Abstract

The invention relates to the technical field of fine wire production, in particular to a continuous casting process parameter optimization method and device based on deep reinforcement learning, and the method comprises the steps: obtaining an initial experiment data set of continuous casting; the initial experiment data set comprises process parameters and performance parameters of continuous casting; optimizing the initial experiment data set by using a Bayesian performance function; and inputting the optimized initial experimental data set into the deep reinforcement learning network, obtaining an optimized Pareto leading edge, and selecting an optimal strategy from the optimized Pareto leading edge so as to adopt an optimal process parameter according to a target value of the performance parameter. The continuous casting process parameters are optimized through the deep reinforcement learning network based on the Pareto frontier, it is ensured that the continuous casting process can reach the ideal level on different performance indexes, and the comprehensive quality of products is improved; and the initial experimental data set is optimized through multiple iterations in advance, so that a foundation can be laid for rapid convergence of a subsequent deep reinforcement learning network, and the performance is improved.

Description

Technical Field

[0001] The present application relates to the field of micro-filament production technology, and in particular to a method and device for optimizing continuous casting process parameters based on deep reinforcement learning. Background Art

[0002] As the core technology for producing metal materials in the modern metallurgical industry, the continuous casting process has a direct impact on the quality of the ingot, production efficiency and energy consumption. With the development of metallurgical technology, the market demand for high-quality ingots and efficient production is getting higher and higher. The continuous casting process not only involves complex physical and chemical processes, but is also dynamically affected by the coupling of multiple fields such as temperature field, flow field, and stress field. Traditional continuous casting process optimization methods mostly rely on the experience of process engineers and step-by-step optimization strategies based on experiments, which are difficult to cope with the increasingly complex process requirements in modern production.

[0003] In current industrial production, the optimization of continuous casting process is usually guided by a single goal, lacking systematic optimization under multiple goals and multiple constraints. These goals often include the surface quality, composition uniformity, and organizational morphology of the ingot, which are mutually constrained and constitute a complex multi-objective optimization problem. In addition, the continuous casting process involves highly nonlinear dynamic processes, which are difficult to effectively model and control in real time through traditional optimization methods. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present application provides a method and device for optimizing continuous casting process parameters based on deep reinforcement learning, which can ensure that the continuous casting process can reach an ideal level in different performance indicators and improve the overall quality of the product.

[0005] In a first aspect, the present application provides a method for optimizing continuous casting process parameters based on deep reinforcement learning, the method comprising the following steps:

[0006] Acquire an initial experimental data set for continuous casting; the initial experimental data set includes process parameters and performance parameters for continuous casting; the process parameters include at least one of feed weight, superheat, stirring time, holding time, mold cooling water temperature difference, and casting speed; the performance parameters include at least one of center segregation, grain angle, grain size, and surface quality grade;

[0007] The initial experimental data set is optimized using the Bayesian efficiency function;

[0008] The optimized initial experimental data set is input into the deep reinforcement learning network to obtain the optimized Pareto frontier, and the optimal strategy is selected from the optimized Pareto frontier to adopt the optimal process parameters according to the target values ​​of the performance parameters.

[0009] In a possible implementation manner, after the initial experimental data set of continuous casting is obtained, the initial experimental data set is also preprocessed; the preprocessing includes data filling and data normalization.

[0010] In a possible implementation, the optimization of the initial experimental data set using the Bayesian performance function includes the following steps:

[0011] Based on the preprocessed initial experimental data set, machine learning prediction models corresponding to different performance parameters are established respectively, and each performance parameter is predicted using the established machine learning prediction model, and the Bayesian effectiveness function EI value of each performance parameter is calculated based on the prediction results;

[0012] The Bayesian effectiveness function EI value of each performance parameter is used as the fitness function in the NSGA-II multi-objective algorithm, the Pareto frontier of the Bayesian effectiveness function EI value within the process parameters is determined, and the target process parameters are selected from the Pareto frontier;

[0013] A continuous casting experiment is carried out according to the target process parameters, and the obtained experimental data is used to update the initial experimental data set, and the updated initial experimental data set is used to optimize the machine learning prediction model, and the optimized initial experimental data is obtained by iteration.

[0014] In a possible implementation, the method of using the established machine learning prediction model to predict each performance parameter and calculating the Bayesian effectiveness function EI value of each performance parameter based on the prediction result includes the following steps:

[0015] The bootstrap method was used to sample multiple data subsets with the same size as the initial experimental data set;

[0016] The established machine learning prediction model is trained using the data subset to obtain prediction results for each performance parameter, and the Bayesian effectiveness function EI value of each performance parameter is calculated in the following manner:

[0017]

[0018] Where y represents the mean of the performance parameter prediction, μ * Represents the maximum value of the performance parameter in the data subset, y-μ * Indicates the possible degree of improvement of performance parameters; represents the standard normal distribution of the prediction performance parameters, and φ(z) is the cumulative distribution function of the prediction performance parameters.

[0019] In a possible implementation, a K-means clustering algorithm is used to screen out target process parameters from the Pareto front.

[0020] In a possible implementation, inputting the optimized initial experimental data set into the deep reinforcement learning network and obtaining the optimized Pareto frontier includes the following steps:

[0021] The deep reinforcement learning network is optimized according to the defined state, action, reward and new state of the intelligent agent; the process parameters and performance parameters of continuous casting are defined as the state of the intelligent agent, the change of the process parameters of continuous casting is defined as the action of the intelligent agent, and the epsilon-greedy strategy is used to take action;

[0022] If the multi-objective value of the new state obtained by taking an action under the current state is better than a solution in the current Pareto frontier, a positive reward is given to the action and the Pareto frontier is updated until the Pareto frontier no longer changes; the initial Pareto frontier is determined from the optimized initial experimental data.

[0023] In a possible implementation, the number of genetic iterations, crossover rate, and mutation rate are set in the NSGA-II multi-objective algorithm; and the learning rate, discount factor, and activation function are set in the deep reinforcement learning network.

[0024] In a second aspect, the present application provides a continuous casting process parameter optimization device based on deep reinforcement learning, the device comprising:

[0025] An acquisition module is used to acquire an initial experimental data set for continuous casting; the initial experimental data set includes process parameters and performance parameters of continuous casting; the process parameters include at least one of feed weight, superheat, stirring time, holding time, mold cooling water temperature difference, and casting speed; the performance parameters include at least one of center segregation, grain angle, grain size, and surface quality grade;

[0026] A data optimization module, used to optimize the initial experimental data set using the Bayesian effectiveness function;

[0027] The strategy selection module is used to input the optimized initial experimental data set into the deep reinforcement learning network and obtain the optimized Pareto frontier, and select the optimal strategy from the optimized Pareto frontier to adopt the optimal process parameters according to the target values ​​of the performance parameters.

[0028] In a third aspect, the present application provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the continuous casting process parameter optimization method based on deep reinforcement learning as described in any one of the first aspects are performed.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the steps of the method for optimizing continuous casting process parameters based on deep reinforcement learning as described in any one of the first aspects.

[0030] The present embodiment provides a method and device for optimizing continuous casting process parameters based on deep reinforcement learning, which obtains an initial experimental data set for continuous casting; the initial experimental data set includes process parameters and performance parameters for continuous casting; the process parameters include at least one of feed weight, superheat, stirring time, holding time, crystallizer cooling water temperature difference, and casting speed; the performance parameters include at least one of center segregation, grain angle, grain size, and surface quality grade; the initial experimental data set is optimized using a Bayesian efficiency function; the optimized initial experimental data set is input into a deep reinforcement learning network, and an optimized Pareto front is obtained, and the optimal strategy is selected from the optimized Pareto front to adopt the optimal process parameters according to the target value of the performance parameter. Thus, the process parameters of continuous casting are optimized by a deep reinforcement learning network based on the Pareto front, ensuring that the continuous casting process can reach an ideal level in different performance indicators and improve the overall quality of the product; and the initial experimental data set is optimized in advance through multiple iterations, which can lay a foundation for the rapid convergence of the subsequent deep reinforcement learning network and improve performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A flow chart of a method for optimizing continuous casting process parameters based on deep reinforcement learning according to an embodiment of the present application is shown;

[0033] Figure 2 A flow chart of optimizing an initial experimental data set using a Bayesian performance function according to an embodiment of the present application is shown;

[0034] Figure 3 A flowchart of inputting an optimized initial experimental data set into a deep reinforcement learning network and obtaining an optimized Pareto frontier according to an embodiment of the present application is shown;

[0035] Figure 4 A flowchart of repeated execution of actions, state transition, reward evaluation, and Pareto frontier update according to an embodiment of the present application is shown;

[0036] Figure 5 A schematic diagram of the structure of a continuous casting process parameter optimization device based on deep reinforcement learning according to an embodiment of the present application is shown;

[0037] Figure 6 A structural block diagram of an electronic device described in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0039] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0040] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0041] In the existing continuous casting process optimization, a single goal is usually taken as the guide, but in fact, multiple performance indicators in the continuous casting process often restrict each other. When optimizing these goals at the same time, it is difficult to find a balance point, resulting in poor overall process optimization effect, and poor adaptability to process parameters, unable to respond to changes in the production environment in real time, such as temperature fluctuations, raw material changes, etc., which makes the effect of process optimization unable to maintain consistency in actual production. In addition, in the process of continuous casting process optimization, a large amount of experimental data support is often required, and a large number of experiments in actual production are not only time-consuming and labor-intensive, but may also affect normal production, and it is difficult to quickly converge to the optimal solution without increasing production costs. Based on this, the present application provides a method and device for optimizing continuous casting process parameters based on deep reinforcement learning, which can optimize the process parameters of continuous casting through a deep reinforcement learning network based on the Pareto frontier, ensure that the continuous casting process can reach an ideal level in different performance indicators, and improve the overall quality of the product; and the initial experimental data set is optimized in advance through multiple iterations, which can lay the foundation for the rapid convergence of the subsequent deep reinforcement learning network and improve performance.

[0042] In one embodiment, see the attached specification Figure 1 The present application provides a method for optimizing continuous casting process parameters based on deep reinforcement learning, the method comprising the following steps:

[0043] S1. Acquire an initial experimental data set for continuous casting; the initial experimental data set includes process parameters and performance parameters for continuous casting; the process parameters include at least one of feed weight, superheat, stirring time, holding time, mold cooling water temperature difference, and casting speed; the performance parameters include at least one of center segregation, grain angle, grain size, and surface quality grade;

[0044] S2, using the Bayesian efficiency function to optimize the initial experimental data set;

[0045] S3. Input the optimized initial experimental data set into the deep reinforcement learning network to obtain the optimized Pareto frontier, and select the optimal strategy from the optimized Pareto frontier to adopt the optimal process parameters according to the target values ​​of the performance parameters.

[0046] In step S1, the parameter space is first defined. In the continuous casting process, the feed weight affects the solidification and cooling process of the metal. The appropriate feed weight can ensure uniform solidification of the metal and obtain better grain structure and mechanical properties. Superheat mainly affects the fluidity and uniformity of the metal. Appropriate superheat helps to improve the fluidity of molten steel and promote the floating and removal of inclusions; but too high superheat may lead to the formation of new inclusions, and may cause coarse grains, aggravate macrosegregation, and affect the uniformity of chemical composition. Stirring time affects the mixing degree and uniformity of the metal. Sufficient stirring time can ensure that the metal is completely mixed to obtain uniform chemical composition and temperature distribution. Appropriate holding time can fully release stress during the solidification process of the ingot and reduce the generation of cracks. The temperature difference of the cooling water of the crystallizer mainly affects the solidification rate and grain structure of the metal. The appropriate cooling water temperature difference can ensure uniform solidification of the metal and obtain good grain structure and mechanical properties. The casting speed mainly affects the flow rate of the metal and the stability of continuous casting. The appropriate casting speed can ensure stable metal flow and obtain uniform product quality. Center segregation is one of the common defects in the continuous casting process, which has a significant impact on the performance of the continuous casting product; reasonable grain angle can improve the mechanical properties of steel; smaller grain size usually means higher strength and hardness; good surface quality can improve subsequent processing performance and final product performance, and increase the market competitiveness of products. Therefore, in this application, feed weight, superheat, stirring time, holding time, crystallizer cooling water temperature difference, and casting speed are selected as the process parameters of continuous casting, and center segregation, grain angle, grain size, and surface quality grade are selected as the target performance parameters of continuous casting.

[0047] In one embodiment, an initial experimental data set containing 30 sets of data (each set of data contains the above-mentioned continuous casting process parameters and corresponding continuous casting performance parameters) is obtained through historical data or a small number of initial experiments. Compared with the prior art, the number of initial experimental data sets obtained is small because the present application also needs to perform the optimization task of the initial experimental data set in the subsequent step S2. By optimizing the initial experimental data set, the most valuable samples for the deep reinforcement learning network can be screened out.

[0048] In other embodiments, after obtaining the initial experimental data set of continuous casting, the initial experimental data set is also preprocessed to improve the quality and availability of the data, so as to provide a more reliable basis for subsequent analysis and decision-making. For example, the missing data is filled with the mean to ensure data integrity, so as to avoid the problem of inaccurate model parameter estimation and prediction accuracy caused by missing data when establishing the prediction model of the continuous casting process; the continuous casting process parameters are normalized to convert different continuous casting process parameters into dimensionless values, so as to facilitate faster convergence of the prediction model established subsequently and reduce the risk of overfitting, wherein the values ​​of different continuous casting process parameters are converted into the range of [0, 1] by the following formula:

[0049]

[0050] Among them, y represents the normalized data, x represents the input continuous casting process parameters, and x max and x min They represent the maximum and minimum values ​​of the input continuous casting process parameters respectively.

[0051] See the instruction manual Figure 2 In step S2, the initial experimental data set is optimized using the Bayesian performance function, including the following steps:

[0052] S201, establishing machine learning prediction models corresponding to different performance parameters based on the preprocessed initial experimental data set, and using the established machine learning prediction model to predict each performance parameter, and calculating the Bayesian effectiveness function EI value of each performance parameter based on the prediction result;

[0053] S202, using the Bayesian effectiveness function EI value of each performance parameter as the fitness function in the NSGA-II multi-objective algorithm, determining the Pareto frontier of the Bayesian effectiveness function EI value within the process parameter, and selecting the target process parameter from the Pareto frontier;

[0054] S203, performing a continuous casting experiment according to the target process parameters, and using the obtained experimental data to update the initial experimental data set, and using the updated initial experimental data set to optimize the machine learning prediction model, and iteratively obtain the optimized initial experimental data.

[0055] Specifically, in step S201, a Gaussian kernel support vector machine (SVM) can be used to establish a machine learning prediction model for each performance parameter. The Gaussian kernel support vector machine (SVM) is a very powerful machine learning algorithm with the advantages of strong nonlinear classification ability, good generalization ability, and effectiveness for high-dimensional data. In this embodiment, a central segregation prediction model, a grain angle prediction model, a grain size prediction model, and a surface quality grade prediction model are established respectively, and a 10-fold cross validation is used for hyperparameter tuning, and the error evaluation adopts the root mean square error Among them, n is the number of samples, y exp and pre represent the true value and predicted value of the performance parameter respectively.

[0056] Among them, the bootstrap method can be used to sample and obtain the Bayesian effectiveness function EI value of the machine learning prediction model for each performance parameter. In one embodiment, the bootstrap method is used to sample the initial experimental data set, and the number of samples is consistent with the size of the initial experimental data set. The sampling is repeated 1000 times to obtain 1000 data subsets of the same size but different samples. Different machine learning models are trained using the data subsets to predict all performance parameters in the process space, thereby obtaining 1000 results for the prediction of each performance parameter. The Bayesian effectiveness function EI value of each performance parameter is then calculated by the following formula (used to express the expectation of potential improvement under the current known information):

[0057]

[0058] Where y represents the mean of the performance parameter prediction, μ * Represents the maximum value of the performance parameter in the training data, y-μ * Indicates the possible degree of improvement of performance parameters; represents the standard normal distribution of the prediction performance parameters, and φ(z) is the cumulative distribution function of the prediction performance parameters.

[0059] In step S202, the obtained Bayesian effectiveness function EI value of each performance parameter is used as the fitness function in the NSGA-II multi-objective algorithm to determine the Pareto frontier of the Bayesian effectiveness function EI value in the process space. The Pareto frontier represents the optimal trade-off between multiple continuous casting performance parameters, that is, it is impossible to further improve any continuous casting performance parameter without sacrificing at least one continuous casting performance parameter. Combining the Bayesian effectiveness function EI value with the NSGA-II multi-objective algorithm can consider the uncertainty of performance parameters while exploring new solutions, which helps to find a solution with a better balance in multiple performance parameters. Among them, the K-means clustering algorithm can be used to screen out process parameter combinations from the Pareto frontier. The K-means clustering algorithm (K-Means Clustering Algorithm) is a widely used unsupervised learning algorithm, which should be a technical means well known to those skilled in the art and will not be elaborated here. In step S203, the target process parameters are screened out from the Pareto front, and the target process parameter experiments are performed. The data of the performance parameters obtained from the experiments and the corresponding process parameter data are added to update the initial experimental data set. The machine learning prediction model is further optimized based on the updated initial experimental data set. The optimization is iterated five times in a cycle to obtain continuous casting experimental data under different production conditions, thereby improving the diversity and representativeness of the data, reducing the training time of the subsequent deep reinforcement learning network, accelerating convergence, and reducing the need for large-scale experiments compared to the existing technology.

[0060] See the instruction manual Figure 3 In step S3, the optimized initial experimental data set is input into the deep reinforcement learning network to obtain the optimized Pareto frontier, including the following steps:

[0061] S301, optimizing the deep reinforcement learning network according to the defined state, action, reward and new state of the intelligent agent; wherein the process parameters and performance parameters of continuous casting are defined as the state of the intelligent agent, the change of the process parameters of continuous casting is defined as the action of the intelligent agent, and the epsilon-greedy strategy is used to take action;

[0062] S302. If the multi-objective value of the new state obtained by taking an action under the current state is better than a solution in the current Pareto frontier, a positive reward is given to the action, and the Pareto frontier is updated until the Pareto frontier no longer changes; wherein the initial Pareto frontier is determined from the optimized initial experimental data.

[0063] Specifically, in step S301-step S302, the constructed deep reinforcement learning network includes an input layer, an output layer and two hidden layers. The input layer contains six neurons, the hidden layer contains four neurons, the output layer is one neuron, and the learning rate is set to 0.0001, the discount factor is 0.98, the activation function uses the ReLU function, and the minimum exploration rate of the epsilon-greedy strategy is 0.01. The deep reinforcement learning network is a method that combines deep learning and reinforcement learning. It allows the agent to learn the optimal strategy through interaction with the environment. In the present application, the state space of the environment in which the agent is located is the continuous casting process space, that is, the performance parameters reflected under different process parameters; the action space is the change of the process parameters of continuous casting, and the epsilon-greedy strategy is used to take actions; the reward function is a reward design based on Pareto frontier optimization. Specifically, after each step of state transfer, a reward is given for the action taken in the current state. t The new state s obtained after t+1 , define the reward r t , if the new state s t+1 The multi-objective value f i (s t+1 ) is better than a solution in the current Pareto front, then the action is considered beneficial and given a positive reward, defined as follows:

[0064]

[0065] Among them, w i is the performance parameter f i The weight of i norm is the improvement of the new state over the nearest solution in the Pareto front in terms of the ith performance parameter, if s t+1 can dominate some solutions in the Pareto front, then Δf i norm is positive, and the action will update the Pareto frontier and obtain a higher reward. t+1 Cannot improve any solution in the current Pareto front, then r t Set to zero or negative.

[0066] That is, the present application combines the Pareto front with the deep reinforcement learning network, and repeats the process of executing actions, state transfer, reward evaluation, and Pareto front update until the deep reinforcement learning network converges and the Pareto front is stable. The process of repeating actions, state transfer, reward evaluation, and Pareto front update can be found in the attached manual. Figure 4 Then, the optimal strategy is extracted from the final deep reinforcement learning network, which can maximize the long-term reward under different continuous casting process parameters and performs best in the optimization of multiple performance parameters of continuous casting.

[0067] It can be seen that the present application provides a method for optimizing continuous casting process parameters based on deep reinforcement learning, which optimizes the process parameters of continuous casting through a deep reinforcement learning network based on the Pareto frontier, ensuring that the continuous casting process can reach an ideal level in multiple different performance indicators such as center segregation, grain angle, grain size and surface quality, thereby improving the overall quality of the product; and optimizing the initial experimental data set through multiple iterations in advance, which can lay the foundation for the rapid convergence of the subsequent deep reinforcement learning network and improve performance.

[0068] Based on the same inventive concept, an embodiment of the present application also provides a continuous casting process parameter optimization device based on deep reinforcement learning. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned continuous casting process parameter optimization method based on deep reinforcement learning in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0069] As the instruction manual Figure 5 As shown, an embodiment of the present application provides a continuous casting process parameter optimization device based on deep reinforcement learning, and the device includes:

[0070] The acquisition module 501 is used to acquire an initial experimental data set for continuous casting; the initial experimental data set includes process parameters and performance parameters of continuous casting; the process parameters include at least one of feed weight, superheat, stirring time, holding time, mold cooling water temperature difference, and casting speed; the performance parameters include at least one of center segregation, grain angle, grain size, and surface quality grade;

[0071] A data optimization module 502, for optimizing the initial experimental data set using a Bayesian performance function;

[0072] The strategy selection module 503 is used to input the optimized initial experimental data set into the deep reinforcement learning network, and obtain the optimized Pareto frontier, and select the optimal strategy from the optimized Pareto frontier to adopt the optimal process parameters according to the target values ​​of the performance parameters.

[0073] In one embodiment, after the acquisition module 501 acquires the initial experimental data set of continuous casting, it also preprocesses the initial experimental data set; the preprocessing includes data filling and data normalization.

[0074] In one embodiment, the data optimization module 502 optimizes the initial experimental data set using the Bayesian effectiveness function, including: establishing machine learning prediction models corresponding to different performance parameters based on the preprocessed initial experimental data set, and predicting each performance parameter using the established machine learning prediction model, and calculating the Bayesian effectiveness function EI value of each performance parameter based on the prediction result; using the Bayesian effectiveness function EI value of each performance parameter as the fitness function in the NSGA-II multi-objective algorithm, determining the Pareto frontier of the Bayesian effectiveness function EI value within the process parameters, and screening out target process parameters from the Pareto frontier; conducting continuous casting experiments according to the target process parameters, and updating the initial experimental data set using the obtained experimental data, and optimizing the machine learning prediction model using the updated initial experimental data set, and iterating to obtain optimized initial experimental data; wherein, the K-means clustering algorithm is used to screen out the target process parameters from the Pareto frontier, and the genetic iteration number, crossover rate and mutation rate are set in the NSGA-II multi-objective algorithm.

[0075] In one embodiment, the data optimization module 502 predicts each performance parameter using the established machine learning prediction model, and calculates the Bayesian effectiveness function EI value of each performance parameter based on the prediction result, including: using the bootstrap method to sample multiple data subsets with the same size as the initial experimental data set; using the data subsets to train the established machine learning prediction model, and obtain the prediction results of each performance parameter, and calculate the Bayesian effectiveness function EI value of each performance parameter in the following manner:

[0076]

[0077] Where y represents the mean of the performance parameter prediction, μ * Represents the maximum value of the performance parameter in the data subset, y-μ * Indicates the possible degree of improvement of performance parameters; represents the standard normal distribution of the prediction performance parameters, and φ(z) is the cumulative distribution function of the prediction performance parameters.

[0078] In one embodiment, the strategy selection module 503 inputs the optimized initial experimental data set into the deep reinforcement learning network and obtains the optimized Pareto frontier, including: optimizing the deep reinforcement learning network according to the defined state, action, reward and new state of the intelligent agent; wherein, the process parameters and performance parameters of continuous casting are defined as the state of the intelligent agent, and the change of the process parameters of continuous casting is defined as the action of the intelligent agent, and the epsilon-greedy strategy is adopted to take action; if the multi-objective value of the new state obtained by taking action under the current state is better than a solution in the current Pareto frontier, a positive reward is given to the action, and the Pareto frontier is updated until the Pareto frontier no longer changes; wherein, the initial Pareto frontier is determined from the optimized initial experimental data; wherein, the learning rate, discount factor, and activation function are set in the deep reinforcement learning network.

[0079] The present application provides a continuous casting process parameter optimization device based on deep reinforcement learning, which obtains the initial experimental data set of continuous casting through an acquisition module; the initial experimental data set includes the process parameters and performance parameters of continuous casting; the process parameters include at least one of the feed weight, superheat, stirring time, holding time, crystallizer cooling water temperature difference, and casting speed; the performance parameters include at least one of center segregation, grain angle, grain size, and surface quality grade; the initial experimental data set is optimized by using the Bayesian efficiency function through the data optimization module; the optimized initial experimental data set is input into the deep reinforcement learning network through the strategy selection module, and the optimized Pareto front is obtained, and the optimal strategy is selected from the optimized Pareto front to take the optimal process parameters according to the target value of the performance parameter. Thus, the process parameters of continuous casting are optimized through the deep reinforcement learning network based on the Pareto front, ensuring that the continuous casting process can reach the ideal level in different performance indicators and improve the comprehensive quality of the product; and the initial experimental data set is optimized in advance through multiple iterations, which can lay the foundation for the rapid convergence of the subsequent deep reinforcement learning network and improve the performance.

[0080] Based on the same concept of the present invention, the specification is attached Figure 6 As shown, the structure of an electronic device 600 provided in an embodiment of the present application includes: at least one processor 601, at least one network interface 604 or other user interface 603, a memory 605, and at least one communication bus 602. The communication bus 602 is used to realize the connection and communication between these components. The electronic device 600 optionally includes a user interface 603, including a display (for example, a touch screen, LCD, CRT, holographic imaging (Holographic) or projection (Projector), etc.), a keyboard or a pointing device (for example, a mouse, a trackball (trackball), a touch pad or a touch screen, etc.).

[0081] The memory 605 may include a read-only memory and a random access memory, and provides instructions and data to the processor 601. A portion of the memory 605 may also include a non-volatile random access memory (NVRAM).

[0082] In some implementations, the memory 605 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof:

[0083] Operating system 6051, including various system programs for implementing various basic services and processing hardware-based tasks;

[0084] The application module 6052 includes various application programs, such as a launcher, a media player, a browser, etc., which are used to implement various application services.

[0085] In an embodiment of the present application, by calling the program or instructions stored in the memory 605, the processor 601 is used to execute steps in a continuous casting process parameter optimization method based on deep reinforcement learning, which can ensure that the continuous casting process can reach an ideal level in different performance indicators and improve the overall quality of the product.

[0086] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes steps in a method for optimizing continuous casting process parameters based on deep reinforcement learning.

[0087] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned continuous casting process parameter optimization method based on deep reinforcement learning can be executed.

[0088] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0090] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0091] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0092] Finally, it should be noted that the above embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for optimizing continuous casting process parameters based on deep reinforcement learning, characterized in that: The method comprises the following steps: Acquire an initial experimental data set for continuous casting; the initial experimental data set includes process parameters and performance parameters for continuous casting; the process parameters include at least one of feed weight, superheat, stirring time, holding time, mold cooling water temperature difference, and casting speed; the performance parameters include at least one of center segregation, grain angle, grain size, and surface quality grade; The initial experimental data set is optimized using the Bayesian efficiency function; The optimized initial experimental data set is input into the deep reinforcement learning network to obtain the optimized Pareto frontier, and the optimal strategy is selected from the optimized Pareto frontier to adopt the optimal process parameters according to the target values ​​of the performance parameters.

2. According to the method for optimizing continuous casting process parameters based on deep reinforcement learning according to claim 1, it is characterized in that: in, After the initial experimental data set of continuous casting is obtained, the initial experimental data set is also preprocessed; the preprocessing includes data filling and data normalization.

3. According to the method for optimizing continuous casting process parameters based on deep reinforcement learning according to claim 2, it is characterized in that: The method of optimizing the initial experimental data set by using the Bayesian efficiency function includes the following steps: Based on the preprocessed initial experimental data set, machine learning prediction models corresponding to different performance parameters are established respectively, and each performance parameter is predicted using the established machine learning prediction model, and the Bayesian effectiveness function EI value of each performance parameter is calculated based on the prediction results; The Bayesian effectiveness function EI value of each performance parameter is used as the fitness function in the NSGA-II multi-objective algorithm, the Pareto frontier of the Bayesian effectiveness function EI value within the process parameters is determined, and the target process parameters are selected from the Pareto frontier; A continuous casting experiment is carried out according to the target process parameters, and the obtained experimental data is used to update the initial experimental data set, and the updated initial experimental data set is used to optimize the machine learning prediction model, and the optimized initial experimental data is obtained by iteration.

4. According to claim 3, a method for optimizing continuous casting process parameters based on deep reinforcement learning is characterized in that: The method of using the established machine learning prediction model to predict each performance parameter and calculating the Bayesian effectiveness function EI value of each performance parameter based on the prediction result includes the following steps: The bootstrap method was used to sample multiple data subsets with the same size as the initial experimental data set; The established machine learning prediction model is trained using the data subset to obtain prediction results for each performance parameter, and the Bayesian effectiveness function EI value of each performance parameter is calculated in the following manner: Where y represents the mean of the performance parameter prediction, μ * Represents the maximum value of the performance parameter in the data subset, y-μ * Indicates the possible degree of improvement of performance parameters; represents the standard normal distribution of the prediction performance parameters, and φ(z) is the cumulative distribution function of the prediction performance parameters.

5. A method for optimizing continuous casting process parameters based on deep reinforcement learning according to claim 4, characterized in that: in, The K-means clustering algorithm is used to screen the target process parameters from the Pareto front.

6. A method for optimizing continuous casting process parameters based on deep reinforcement learning according to claim 5, characterized in that: The step of inputting the optimized initial experimental data set into the deep reinforcement learning network and obtaining the optimized Pareto frontier includes the following steps: The deep reinforcement learning network is optimized according to the defined state, action, reward and new state of the intelligent agent; the process parameters and performance parameters of continuous casting are defined as the state of the intelligent agent, the change of the process parameters of continuous casting is defined as the action of the intelligent agent, and the epsilon-greedy strategy is used to take action; If the multi-objective value of the new state obtained by taking an action under the current state is better than a solution in the current Pareto frontier, a positive reward is given to the action and the Pareto frontier is updated until the Pareto frontier no longer changes; the initial Pareto frontier is determined from the optimized initial experimental data.

7. A method for optimizing continuous casting process parameters based on deep reinforcement learning according to claim 6, characterized in that: in, Set the number of genetic iterations, crossover rate, and mutation rate in the NSGA-II multi-objective algorithm; set the learning rate, discount factor, and activation function in the deep reinforcement learning network.

8. A continuous casting process parameter optimization device based on deep reinforcement learning, characterized in that: The device comprises: An acquisition module is used to acquire an initial experimental data set for continuous casting; the initial experimental data set includes process parameters and performance parameters of continuous casting; the process parameters include at least one of feed weight, superheat, stirring time, holding time, mold cooling water temperature difference, and casting speed; the performance parameters include at least one of center segregation, grain angle, grain size, and surface quality grade; A data optimization module, used to optimize the initial experimental data set using the Bayesian effectiveness function; The strategy selection module is used to input the optimized initial experimental data set into the deep reinforcement learning network and obtain the optimized Pareto frontier, and select the optimal strategy from the optimized Pareto frontier to adopt the optimal process parameters according to the target values ​​of the performance parameters.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for optimizing continuous casting process parameters based on deep reinforcement learning are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for optimizing continuous casting process parameters based on deep reinforcement learning as described in any one of claims 1 to 7.

Images