A method and device for optimizing process parameters of a tin smelting process based on reinforcement learning

Abstract

The application relates to the technical field of tin smelting parameter optimization, and provides a tin smelting process parameter optimization method and equipment based on reinforcement learning, which comprises the following steps: acquiring smelting data of each tin smelting process at multiple time points; for each tin smelting process, generating a predicted action of the tin smelting process by using a parameter prediction model based on all smelting data of the tin smelting process, and calculating a reward function value corresponding to the predicted action; performing online training and offline training on the parameter prediction model according to all predicted actions and reward function values, to obtain a trained parameter prediction model; performing parameter prediction on a tin smelting process to be optimized by using the trained parameter prediction model, obtaining a final predicted action of the tin smelting process to be optimized, and optimizing process parameters of the tin smelting process to be optimized based on the final predicted action. The method can improve the accuracy of tin smelting process parameter optimization.

Description

Technical Field

[0001] This application relates to the field of tin smelting parameter optimization technology, and in particular to a method and equipment for optimizing process parameters in tin smelting based on reinforcement learning. Background Technology

[0002] Currently, in tin smelting production, the adjustment of key process parameters mainly relies on the operator's experience. Common control methods involve manually adjusting parameters such as oxygen flow rate, coal flow rate, lance position, furnace pressure, and smelting temperature to ensure stable operation of the smelting process. However, this method has the following main shortcomings: First, manual adjustment based on experience is inherently subjective and uncertain. Different operators, due to differences in knowledge and experience, often make different decisions regarding the same process parameter, leading to poor stability and consistency in the production process. When raw material composition, furnace conditions, or the external environment fluctuate, manual adjustments often fail to respond promptly and accurately, easily causing the smelting process to deviate from optimal conditions, resulting in reduced tin yield and increased metal loss.

[0003] Secondly, traditional process optimization methods rely on establishing precise mathematical models. However, tin smelting is characterized by long processes, complex procedures, large fluctuations in raw material composition, and unclear chemical reaction mechanisms, making it difficult to establish accurate mechanistic models of mass transfer, heat transfer, and chemical reaction processes within the smelting furnace. Even simplified mathematical models often require numerous assumptions and approximations, failing to accurately reflect the dynamic characteristics of the production process and leading to significant deviations between optimization results and actual operation. Furthermore, optimization calculations based on mechanistic models are typically complex and lack real-time capability, making it difficult to meet the rapid control requirements of actual production.

[0004] Furthermore, process optimization relying on manual labor and traditional models struggles to achieve a balance between product quality, energy consumption, and environmental protection. Under current production methods, energy parameters such as oxygen and coal are often poorly controlled, easily leading to over-combustion or under-combustion, resulting in high energy consumption and large furnace temperature fluctuations. Simultaneously, incomplete combustion also increases flue gas volume and dust concentration, exacerbating environmental pollution. This not only increases smelting costs but also contradicts the current development requirements for green and low-carbon smelting.

[0005] Furthermore, as tin smelting enterprises transform towards intelligent and digital development, traditional methods relying on manual experience and fixed models are insufficient to meet the demands of intelligent manufacturing. Existing methods lack adaptive learning capabilities, cannot effectively utilize historical production data to summarize patterns and optimize strategies, and struggle to achieve dynamic optimization and globally optimal control under complex operating conditions. This results in poor accuracy in optimizing process parameters during tin smelting. Summary of the Invention

[0006] This application provides a method and equipment for optimizing process parameters in tin smelting based on reinforcement learning, which can solve the problem of poor accuracy in optimizing process parameters in tin smelting.

[0007] Firstly, this application provides a method for optimizing process parameters in a tin smelting process based on reinforcement learning. This method includes: Acquire melting data at multiple points in each tin melting process; For each tin smelting process, based on all smelting data for that process, a parametric prediction model is used to generate predicted actions for the tin smelting process, and the reward function value corresponding to the predicted actions is calculated. The predicted actions include the predicted process parameters for the tin smelting process. The parametric prediction model is trained online and offline based on all predicted actions and reward function values ​​to obtain the trained parametric prediction model. Using the trained parameter prediction model, the parameters of the tin smelting process to be optimized are predicted to obtain the final predicted action of the tin smelting process to be optimized, and the process parameters of the tin smelting process to be optimized are optimized based on the final predicted action.

[0008] Optionally, based on all smelting data of the tin smelting process, a parametric prediction model is used to generate predicted actions for the tin smelting process, including: For each time point, a state vector is constructed based on the melting data of the tin melting process at that time point; Based on parameter prediction of all state vectors, the predicted actions of the tin smelting process are obtained.

[0009] Optionally, calculate the reward function value corresponding to the predicted action, including: Through the formula:

[0010] Calculate the reward function value ; in, and As a weighting factor, Represents the normalization function. To predict the direct recovery rate of tin corresponding to the action, This is to predict the unit energy consumption corresponding to the action.

[0011] Optionally, the parametric prediction model can be trained online and offline based on all predicted actions and reward function values ​​to obtain a trained parametric prediction model, including: For each prediction action, the parameter prediction model is trained offline based on the prediction action and the reward function value to obtain the offline trained parameter prediction model. Store all offline-trained parameter prediction models into the offline buffer pool and initialize the online buffer pool; The parameter prediction model is trained online using both offline and online buffer pools to obtain the trained parameter prediction model.

[0012] Optionally, the parameter prediction model can be trained offline based on the predicted action and reward function value to obtain the offline trained parameter prediction model, including: Determine whether the predicted action meets the convergence condition; If so, the parameter prediction model will be used as the parameter prediction model after offline training. Otherwise, the model is updated using the reward function value, the updated parameter prediction model is used as the parameter prediction model, and the steps are returned for each tin smelting process, based on all smelting data of the tin smelting process, using the parameter prediction model to generate the predicted action of the tin smelting process, and calculating the reward function value corresponding to the predicted action.

[0013] Optionally, the parameter prediction models include Q-networks, value V-networks, policy π-networks, and target Q-networks; Updating the parameter prediction model using reward function values ​​includes: The Q-network, value V-network, policy π-network, and target Q-network in the parameter prediction model are updated based on the reward function values ​​to obtain the updated parameter prediction model.

[0014] Optionally, the parameter prediction model can be trained online using both an offline buffer pool and an online buffer pool to obtain the trained parameter prediction model, including: Training data is sampled from both the offline and online buffer pools; The parameter prediction model is trained using the training data to obtain the trained parameter prediction model.

[0015] Secondly, this application provides a reinforcement learning-based device for optimizing process parameters in tin smelting, comprising: The acquisition module is used to acquire smelting data at multiple moments in each tin smelting process; The calculation module is used to generate predicted actions for each tin smelting process based on all smelting data of the tin smelting process using a parametric prediction model, and to calculate the reward function value corresponding to the predicted actions; the predicted actions include the predicted process parameters of the tin smelting process. The training module is used to train the parametric prediction model online and offline based on all predicted actions and reward function values, so as to obtain the trained parametric prediction model. The optimization module is used to predict the parameters of the tin smelting process to be optimized using the trained parameter prediction model, obtain the final predicted action of the tin smelting process to be optimized, and optimize the process parameters of the tin smelting process to be optimized based on the final predicted action.

[0016] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for optimizing process parameters of the tin smelting process based on reinforcement learning.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned reinforcement learning-based method for optimizing process parameters in tin smelting.

[0018] The above-mentioned solution in this application has the following beneficial effects: In the embodiments of this application, smelting data for each tin smelting process at multiple time points is acquired. Then, for each tin smelting process, based on all smelting data, a parameter prediction model is used to generate predicted actions for the tin smelting process, and the reward function value corresponding to the predicted action is calculated. The parameter prediction model is then trained online and offline based on all predicted actions and reward function values ​​to obtain a trained parameter prediction model. Finally, the trained parameter prediction model is used to predict the parameters of the tin smelting process to be optimized, obtaining the final predicted action for the tin smelting process to be optimized. Based on the final predicted action, the process parameters of the tin smelting process to be optimized are optimized. Specifically, predicting the predicted actions of the tin smelting process based on smelting data at multiple time points considers the historical state of the tin smelting process, effectively learning the production patterns and changes of the tin smelting process, improving the accuracy of process parameter prediction. Online and offline training of the parameter prediction model enables adaptive model enhancement, improving the performance of the parameter prediction model. Using a high-performance model for process parameter optimization effectively improves the accuracy of process parameter optimization in the tin smelting process.

[0019] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a reinforcement learning-based method for optimizing process parameters in tin smelting, provided as an embodiment of this application; Figure 2 A schematic diagram illustrating the predicted tin recovery rate according to an embodiment of this application; Figure 3 A schematic diagram illustrating the predicted unit energy consumption of an embodiment of this application; Figure 4 This is a schematic diagram of model comparison and evaluation provided in an embodiment of this application; Figure 5 A schematic diagram of a reinforcement learning-based tin smelting process parameter optimization device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] To address the issue of poor accuracy in existing tin smelting process parameter optimization methods, this application provides a reinforcement learning-based method for optimizing tin smelting process parameters. This method acquires smelting data at multiple time points for each tin smelting process. Then, for each tin smelting process, based on all the smelting data, it uses a parameter prediction model to generate predicted actions for the tin smelting process and calculates the reward function value corresponding to the predicted actions. The parameter prediction model is then trained online and offline based on all predicted actions and reward function values ​​to obtain a trained parameter prediction model. Finally, the trained parameter prediction model is used to predict the parameters of the tin smelting process to be optimized, obtaining the final predicted actions for the tin smelting process to be optimized. Based on these final predicted actions, the process parameters of the tin smelting process to be optimized are then optimized. Among them, the prediction action of the tin smelting process is predicted based on smelting data at multiple times. It takes into account the historical state of the tin smelting process, effectively learns the production rules and changes of the tin smelting process, improves the accuracy of the prediction of process parameters of the tin smelting process, and enables adaptive enhancement of the model by online and offline training of the parameter prediction model, thereby improving the performance of the parameter prediction model. The high-performance model is used to optimize the process parameters, which effectively improves the accuracy of the optimization of process parameters of the tin smelting process.

[0029] The following is an illustrative example of the reinforcement learning-based method for optimizing process parameters in tin smelting provided in this application.

[0030] like Figure 1 As shown, the reinforcement learning-based method for optimizing process parameters in tin smelting provided in this application includes the following steps: Step 11: Obtain melting data for each tin melting process at multiple times.

[0031] The tin smelting process described above is a sample tin smelting process. The smelting data includes state parameters (such as furnace temperature, furnace atmosphere, coal flow rate, lance air pressure, oxygen flow rate, etc.) and output indicators (such as tin direct recovery rate, unit energy consumption, etc.). Among the state parameters, some are controllable process parameters (such as furnace temperature, coal flow rate, oxygen flow rate, etc.), and all the times mentioned above are historical times.

[0032] In some embodiments of this application, smelting data can be acquired using devices such as sensors.

[0033] For example, after obtaining the smelting data, the smelting data can be preprocessed, including: Outlier handling: When fluctuations occur in the raw material composition, furnace condition, or external environment of the top-blown furnace, the collected data may deviate from the normal range. In addition, some data that needs to be manually entered into the system may also be abnormal due to human error. Box plots are used to remove outliers.

[0034] Missing value handling: Outliers, human error, and equipment failure can all cause data loss. Directly deleting missing values ​​may result in the loss of some key information, so it is necessary to fill in the missing values ​​accordingly.

[0035] Mean normalization: The sampling frequency of the state variables and action variables of the top blown furnace in the historical database is on the order of seconds. However, the corresponding key target information (such as the weight of tin A, tin B, and coal, as well as the tin content of tin A and tin B) is only detected offline when the material is discharged. In order to match the state variables, action variables, and key target information on the timestamp, these three files need to be averaged first and then matched according to the timestamp.

[0036] Standardization process: The data collected on-site have significantly different dimensions. To eliminate the impact of these large dimensions on the model, data standardization is necessary. The Z-Score method is used for data standardization. It is the input observable parameter data. These are the given modeling output parameter data. It is a model of the overall input and output dataset. The dataset S is then standardized.

[0037]

[0038] in , and Let N be a column vector with a length equal to the number of samples in the dataset. The mean of the column vectors is calculated and then expanded to the original column vector size, denoted as N. , σ The standard deviation is denoted as .

[0039] It should be noted that the random uncertainties and dynamic laws exhibited in the tin smelting process can be simulated based on smelting data. For example, a probabilistic neural network (BPNN, ensemble-bootstraped probabilistic neural network) structure can be used for simulation. The BPNN network assumes that the state changes of tin recovery rate, unit energy consumption, etc., in the tin smelting production process follow a Gaussian distribution. Multiple PNN networks are fitted and trained simultaneously. Each PNN network has the same structure, consisting of deep fully connected neural networks (FC), using the Swish activation function. The state transition model based on the BPNN network can be expressed as:

[0040] In the formula, the network input represents the action at the current moment. ,state and historical status L represents the historical time step length. The output layer has a two-head structure and parameterizes the state difference between adjacent time steps. and current step reward The probability distribution mean and diagonal covariance θ represents the BPNN network parameters, meaning that different networks differ only in their weights. This allows for the use of the Boostrap method to shuffle the data order, learn the potential state transition patterns in tin smelting production data, and capture any uncertainties in the real tin smelting system. Therefore, for the next state that needs to be predicted... The value sampled from the state difference distribution of the network's final output using the reparameter re-parameter technique is compared with the current step state of the current input. sum:

[0041] Define the loss function of the BPNN network as the negative log-likelihood function, and minimize this loss objective during network training:

[0042] Simultaneously, L2 regularization loss is introduced to mitigate the overfitting behavior of the trained state transition model. Furthermore, after training the BPNN network, the top k PNNs with the smallest validation loss are selected as the optimal sub-model library, and the corresponding network parameters are stored. Finally, the parameter solution for the time-interval-affected variables involves adding the parameter values ​​at the current time interval to the calculated difference, inversely standardizing the solution using the same method as standardizing the collected process parameters, and then using the training set to validate the performance of the constructed state transition model.

[0043] The BPNN network described above will be illustrated with a specific example below.

[0044] The tin recovery rate was predicted using the BPNN network described above, and the prediction results are as follows: Figure 2 As shown, Figure 2 Figure 'a' shows the prediction results from the integrated BPNN. The horizontal axis represents the test sample number, and the vertical axis represents the tin recovery rate in percentage (%). The curves are the actual recovery rate curve and the predicted recovery rate curve, respectively. The figure also shows the 95% confidence interval for the prediction results. Figure 2 b is the residual analysis chart, with the horizontal axis representing the predicted value in percentage (%) and the vertical axis representing the residual. Figure 2 c represents the comparison between predicted and actual values. The horizontal axis represents the actual value, and the vertical axis represents the predicted value. All units are percentages (%). Figure 2 d represents the prediction error distribution, with the vertical axis representing the prediction error as a percentage (%) and the vertical axis representing the frequency. Figure 2 The prediction results show that the coefficient of determination (R²) is 0.9531, the mean squared error (MSE) is 0.887, the mean absolute error (MAE) is 0.667%, the 95% interval coverage is 0.986, the average interval width is 3.930%, the number of test samples is 1000, and the number of ensemble models is 8.

[0045] The above BPNN network is used to predict unit energy consumption. The prediction results are as follows: Figure 3 As shown, Figure 3 Figure 'a' shows the energy consumption prediction results from the integrated BPNN. The horizontal axis represents the test sample number, and the vertical axis represents the energy consumption per ton of tin, in kilowatt-hours per ton (kWh / ton). The curves represent the actual energy consumption, predicted energy consumption, and average energy consumption (1228 kWh / ton), respectively. The figure also illustrates the 95% confidence interval for the prediction results. Figure 3 b is the residual analysis chart. The horizontal axis represents the predicted energy consumption in kWh / ton, and the vertical axis represents the residual in kWh / ton. Figure 3 c represents the comparison between predicted and actual values. The horizontal axis represents actual energy consumption, and the vertical axis represents predicted energy consumption. The unit for both values ​​is kilowatt-hours per ton (kWh / ton). Figure 3 d represents the prediction error distribution fitted to a normal distribution. The horizontal axis represents the prediction error, and the two vertical axes represent the frequency and cumulative proportion, respectively. The bar chart shows the error distribution, with the dashed line representing the zero error line and the curve representing the cumulative distribution. Figure 3 It can be seen that the R-squared of the prediction result is... 2=0.8914, RMSE=28.24kWh / ton, MAE=21.50kWh / ton, 95% interval coverage=0.945, average interval k-width=109.27 kWh / ton, average energy consumption baseline is 1227.5 kWh / ton, and predicted maximum energy consumption is 924.2 kWh / ton. Experimental results show that the top-blown furnace simulation model constructed using probabilistic neural networks performs well.

[0046] Step 12: For each tin smelting process, based on all smelting data of the tin smelting process, use the parametric prediction model to generate the predicted action of the tin smelting process, and calculate the reward function value corresponding to the predicted action.

[0047] The aforementioned prediction actions include the prediction of process parameters for the tin smelting process, that is, the process parameters for the next moment after the last moment.

[0048] In some embodiments of this application, the steps of generating predicted actions for the tin smelting process using a parametric prediction model based on all smelting data of the tin smelting process, and calculating the reward function value corresponding to the predicted actions, include: The first step is to construct a state vector for each time step based on the smelting data of the tin smelting process at that time step.

[0049] For example, a state space S is defined, which represents a vector consisting of state parameters, process parameters, and process performance indicators (tin yield V and unit energy consumption E), and can be expressed as: .

[0050] Define an action space A, which represents the various process parameter control variables that need to be manually adjusted during the tin smelting process. Only continuous optimization tasks are considered, meaning optimization is performed in a continuous domain rather than discretely. The value range of each action is constrained to [-1, 1], limiting the changes in the action vector to fluctuate within a normal range, based on the current state. Choose an action , can be represented as: .

[0051] The second step is to predict the parameters based on all state vectors to obtain the predicted actions for the tin smelting process.

[0052] For example, algorithms such as Markov Decision Process (MDP) can be used to predict parameters based on all state vectors, thereby obtaining the predicted actions for the tin smelting process.

[0053] The third step is to calculate the reward function value corresponding to the predicted action.

[0054] Specifically, through the formula:

[0055] Calculate the reward function value .

[0056] in, and As a weighting factor, Represents the normalization function. To predict the direct recovery rate of tin corresponding to the action, This is to predict the unit energy consumption corresponding to the action.

[0057] For example, the tin direct recovery rate and unit energy consumption corresponding to the predicted action can be calculated by the BPNN network that simulates the random uncertainty and dynamic laws reflected in the tin smelting process in step 11. For example, the predicted action is input into the BPNN network for calculation, and the BPNN network outputs the corresponding tin direct recovery rate and unit energy consumption.

[0058] It is understandable that the above calculation process is the same as the calculation process of the parameter prediction model.

[0059] Step 13: Train the parameter prediction model online and offline based on all predicted actions and reward function values ​​to obtain the trained parameter prediction model.

[0060] In some embodiments of this application, the steps of training the parameter prediction model online and offline based on all predicted actions and reward function values ​​to obtain the trained parameter prediction model include: The first step is to train the parameter prediction model offline for each prediction action based on the prediction action and the reward function value, so as to obtain the offline trained parameter prediction model.

[0061] Specifically, determine whether the predicted action meets the convergence condition.

[0062] If so, the parameter prediction model will be used as the parameter prediction model after offline training.

[0063] Otherwise, the model is updated using the reward function value, the updated parameter prediction model is used as the parameter prediction model, and the steps are returned for each tin smelting process, based on all smelting data of the tin smelting process, using the parameter prediction model to generate the predicted action of the tin smelting process, and calculating the reward function value corresponding to the predicted action.

[0064] For example, the above convergence condition can be that the reward function value or state value value corresponding to the predicted action is less than a preset threshold.

[0065] In some embodiments of this application, the parameter prediction model includes a Q network, a value V network, a policy π network, and a target Q network. The Q network, value V network, policy π network, and target Q network in the parameter prediction model are updated according to the reward function value to obtain the updated parameter prediction model.

[0066] The specific update process is as follows: Initialize the current Q network, value V network, policy π network, and target Q network.

[0067] Update the V network and calculate the Q-value of the current state-action pair. :

[0068] Fit the τ-quantile of Q using Huber quantile regression loss. :

[0069] Where τ is V quantile threshold of the network I It is an indicator function. Huber The loss is a smoothed L1 loss.

[0070] Update the Q network, with the TD target represented by V(s'):

[0071] in, Indicates TD target, Indicates the discount factor. Indicates the end of the process. This represents the valuation of the target value network.

[0072] Update strategy π, advantage-weighted behavior cloning:

[0073]

[0074] in β It is the temperature coefficient in strategy updates, which controls the trade-off between strategy utilization and exploration.

[0075] The Gaussian log probability of the policy network output is calculated using the following formula:

[0076] Where μ is the mean and σ is the standard deviation.

[0077] Soft update target Q-value network weight parameters =θ:

[0078] in The target update rate ranges from 0.9 to 0.999. Indicates the target network parameters. This indicates the current network parameters.

[0079] The second step is to store all the offline trained parameter prediction models into the offline buffer pool and initialize the online buffer pool.

[0080] For example, the offline strategy used to predict the output of the offline-trained parameter prediction model initializes the network parameters and the target network parameters. A fine-tuning strategy is constructed based on the offline and online strategies, and the strategy interacts with the environment to obtain and store data as a trajectory δ. The trajectory is stored in an offline buffer pool. ← ∪{δ}.

[0081] The third step is to use offline and online buffer pools to train the parameter prediction model online, thereby obtaining the trained parameter prediction model.

[0082] Specifically, training data is sampled from both the offline and online buffer pools; The parameter prediction model is trained using the training data to obtain the trained parameter prediction model.

[0083] For example, N / 2 batches of data are sampled from both the offline buffer pool and the online buffer pool.

[0084] Calculate the target Q value of the online network: ; Calculate the target Q-value for the offline network: ; Calculate the target Q value: ; The network is updated using mean squared variance. ; Update strategy parameter Φ via ; Update target network parameters , .

[0085] It should be noted that before each execution of the above training process, it is possible to first determine whether the predicted action output by the parameter prediction model meets the convergence condition. If so, the training is considered complete and there is no need to execute the training process again. Otherwise, the model is trained through the above training process.

[0086] Step 14: Using the trained parameter prediction model, perform parameter prediction on the tin smelting process to be optimized, obtain the final predicted action of the tin smelting process to be optimized, and optimize the process parameters based on the final predicted action.

[0087] The aforementioned tin smelting process to be optimized refers to a tin smelting process that requires optimization of process parameters. The aforementioned final prediction action includes the predicted values ​​of multiple process parameters of the tin smelting process to be optimized at future times.

[0088] Specifically, the process acquires smelting data for the tin smelting process to be optimized at multiple historical moments (the last historical moment can be the current moment), then constructs a state vector based on all smelting data, uses a trained parameter prediction model to predict parameters based on the state vector, obtains the final predicted action, and then adjusts the process parameters to the predicted values ​​corresponding to the final predicted action at future moments through the control system of the tin smelting process to be optimized, thereby achieving parameter optimization.

[0089] It is worth mentioning that the prediction action of the tin smelting process is based on smelting data at multiple times. It takes into account the historical state of the tin smelting process, effectively learns the production rules and changes of the tin smelting process, improves the accuracy of the prediction of process parameters of the tin smelting process, and enables adaptive enhancement of the model by online and offline training of the parameter prediction model, thereby improving the performance of the parameter prediction model. Using the high-performance model for process parameter optimization, the accuracy of process parameter optimization of the tin smelting process is effectively improved.

[0090] Furthermore, this application has the following advantages: 1. The method proposed in this application can fully utilize historical production data and optimize process parameters under real-time monitoring, while making corresponding adjustments according to constantly changing operating conditions and requirements. Compared with traditional methods of constructing parameter optimization models, this method has stronger resistance to disturbances and migration capabilities, and is more in line with real-world application scenarios.

[0091] 2. By weighted fusion of offline and online strategies, a fine-tuning strategy is reconstructed for action selection. This reduces over-reliance on short-term dynamics, enhances exploration capabilities, and ensures that the stability of offline strategies and the adaptability of online strategies are balanced during interaction.

[0092] 3. It possesses excellent generalization ability, not only adapting to the optimization of complex process parameters in tin reduction smelting, but also extending to complex scenarios in multiple production processes across the entire tin industry chain, meeting the optimization needs of complex process parameters in different processes. This ensures continuous and efficient operation of the production process, reducing the uncertainty and additional costs associated with relying on empirical parameter tuning.

[0093] The method of this application will be illustrated below with a specific example.

[0094] The method described in this application is compared with other offline-online reinforcement learning algorithms, such as the Advantage Weighted Actor Critic (AWAC) algorithm, the Optimistic Actor-Critic for Beyond Visual Range (OAC-BVR) algorithm, and the Independent Q-Learning (IQL) algorithm. The comparison results are as follows: Figure 4 As shown, the two bar statistics are the direct tin recovery rate (100%) and the unit energy consumption (kWh / ton), respectively. It can be seen that the prediction results of the algorithm proposed in this application are better than those of the comparison algorithm.

[0095] The following is an exemplary description of the tin smelting parameter optimization device based on reinforcement learning provided in this application.

[0096] like Figure 5 As shown in the figure, this application provides a reinforcement learning-based device for optimizing process parameters in a tin smelting process. The tin smelting process parameter optimization device 500 includes: The acquisition module 501 is used to acquire smelting data at multiple moments in each tin smelting process; The calculation module 502 is used to generate predicted actions for each tin smelting process based on all smelting data of the tin smelting process using a parameter prediction model, and to calculate the reward function value corresponding to the predicted actions; the predicted actions include the predicted process parameters of the tin smelting process. Training module 503 is used to train the parameter prediction model online and offline based on all predicted actions and reward function values ​​to obtain the trained parameter prediction model. The optimization module 504 is used to use the trained parameter prediction model to predict the parameters of the tin smelting process to be optimized, obtain the final predicted action of the tin smelting process to be optimized, and optimize the process parameters of the tin smelting process to be optimized based on the final predicted action.

[0097] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0098] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0099] like Figure 6 As shown, an embodiment of this application provides a terminal device, wherein the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 6 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0100] Specifically, when the processor D100 executes the computer program D102, it acquires smelting data for each tin smelting process at multiple times. Then, for each tin smelting process, based on all the smelting data, it uses a parameter prediction model to generate predicted actions for the tin smelting process and calculates the reward function value corresponding to the predicted actions. The parameter prediction model is then trained online and offline based on all predicted actions and reward function values ​​to obtain a trained parameter prediction model. Finally, the trained parameter prediction model is used to predict the parameters of the tin smelting process to be optimized, obtaining the final predicted actions for the tin smelting process to be optimized. Based on the final predicted actions, the process parameters of the tin smelting process to be optimized are optimized. In this process, predicting the predicted actions of the tin smelting process based on smelting data at multiple times considers the historical state of the tin smelting process, effectively learning the production patterns and changes of the tin smelting process, improving the accuracy of process parameter prediction. Online and offline training of the parameter prediction model enables adaptive model enhancement, improving the performance of the parameter prediction model. Using a high-performance model for process parameter optimization effectively improves the accuracy of process parameter optimization in the tin smelting process.

[0101] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0102] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0103] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0104] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the apparatus / terminal device for the reinforcement learning-based tin smelting parameter optimization method; a recording medium; a computer memory; a read-only memory (ROM); a random access memory (RAM); an electrical carrier signal; a telecommunication signal; and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0106] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0107] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.

Claims

1. A method for optimizing process parameters in tin smelting based on reinforcement learning, characterized in that, include: Acquire melting data at multiple points in each tin melting process; For each of the tin smelting processes, based on all smelting data of the tin smelting process, a predictive action for the tin smelting process is generated using a parametric prediction model, and the reward function value corresponding to the predictive action is calculated; the predictive action includes the predicted process parameters of the tin smelting process. The parameter prediction model is trained online and offline based on all predicted actions and reward function values ​​to obtain the trained parameter prediction model. Using the trained parameter prediction model, the parameters of the tin smelting process to be optimized are predicted to obtain the final predicted action of the tin smelting process to be optimized, and the process parameters of the tin smelting process to be optimized are optimized based on the final predicted action.

2. The method for optimizing process parameters in tin smelting according to claim 1, characterized in that, The method of generating predicted actions for the tin smelting process using a parametric prediction model based on all smelting data from the tin smelting process includes: For each time point, a state vector is constructed based on the smelting data of the tin smelting process at that time point; Based on parameter prediction of all state vectors, the predicted actions of the tin smelting process are obtained.

3. The method for optimizing process parameters in tin smelting according to claim 1, characterized in that, The calculation of the reward function value corresponding to the predicted action includes: Through the formula: Calculate the reward function value ; in, and As a weighting factor, Represents the normalization function. To predict the direct recovery rate of tin corresponding to the action, This is to predict the unit energy consumption corresponding to the action.

4. The method for optimizing process parameters in tin smelting according to claim 3, characterized in that, The step of training the parameter prediction model online and offline based on all predicted actions and reward function values ​​to obtain the trained parameter prediction model includes: For each prediction action, the parameter prediction model is trained offline based on the prediction action and the reward function value to obtain the offline trained parameter prediction model. Store all offline-trained parameter prediction models into the offline buffer pool and initialize the online buffer pool; The parameter prediction model is trained online using the offline buffer pool and the online buffer pool to obtain the trained parameter prediction model.

5. The method for optimizing process parameters in tin smelting according to claim 4, characterized in that, The offline training of the parameter prediction model based on the predicted action and reward function value to obtain the offline trained parameter prediction model includes: Determine whether the predicted action meets the convergence condition; If so, the parameter prediction model will be used as the parameter prediction model after offline training. Otherwise, the parameter prediction model is updated using the reward function value, the updated parameter prediction model is used as the parameter prediction model, and the steps of generating the prediction action of the tin smelting process based on all smelting data of the tin smelting process using the parameter prediction model and calculating the reward function value corresponding to the prediction action are returned for each of the tin smelting processes.

6. The method for optimizing process parameters in tin smelting according to claim 5, characterized in that, The parameter prediction model includes a Q-network, a value V-network, a policy π-network, and a target Q-network; The step of updating the parameter prediction model using the reward function value includes: The Q-network, value V-network, policy π-network, and target Q-network in the parameter prediction model are updated based on the reward function value to obtain the updated parameter prediction model.

7. The method for optimizing process parameters in tin smelting according to claim 6, characterized in that, The step of using the offline buffer pool and the online buffer pool to train the parameter prediction model online, and obtaining the trained parameter prediction model, includes: Training data is sampled from the offline buffer pool and the online buffer pool; The parameter prediction model is trained using the training data to obtain the trained parameter prediction model.

8. A device for optimizing process parameters in tin smelting based on reinforcement learning, characterized in that, include: The acquisition module is used to acquire smelting data at multiple moments in each tin smelting process; The calculation module is used to generate a predicted action for each tin smelting process based on all smelting data of the tin smelting process using a parameter prediction model, and to calculate the reward function value corresponding to the predicted action; the predicted action includes the predicted process parameters of the tin smelting process. The training module is used to train the parameter prediction model online and offline based on all predicted actions and reward function values ​​to obtain the trained parameter prediction model. The optimization module is used to use the trained parameter prediction model to predict the parameters of the tin smelting process to be optimized, obtain the final predicted action of the tin smelting process to be optimized, and optimize the process parameters of the tin smelting process to be optimized based on the final predicted action.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the reinforcement learning-based process parameter optimization method for tin smelting as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the reinforcement learning-based process parameter optimization method for tin smelting as described in any one of claims 1 to 7.

Images