A reinforcement learning-based method for parameter optimization and control of the sulfidation process for the resource recovery of waste acid.

By optimizing the parameters of the sulfurization process for the resource recovery of waste acid using a reinforcement learning-based method, the problem of dynamic adjustment of the amount of sulfurizing agent added was solved, and stable control of the reactor outlet ORP and efficient recovery of valuable metals were achieved, thereby reducing the processing cost.

CN123113340APending Publication Date: 2026-11-17CENT SOUTH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611298135.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In the existing waste acid resource recovery sulfidation process, it is difficult to dynamically adjust the amount of sulfiding agent added according to the changes in waste acid flow rate and metal ion concentration, resulting in fluctuations in reactor outlet ORP, which affects the recovery effect of valuable metals and the treatment cost.

Method used

A reinforcement learning-based approach is adopted. By constructing a reinforcement learning model, combined with waste acid flow rate, metal ion concentration, and reactor operating status, continuous control actions are generated. Historical data is used to determine the control parameter variation boundary, optimize the amount of sulfurizing agent added, achieve ORP target control, and coordinate resource recovery effect and sulfurization cost.

Benefits of technology

This achieved stable convergence of the reactor outlet ORP under dynamic operating conditions, reduced the consumption of sulfiding agent, and improved the recovery efficiency of valuable metals and the stability of the production process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113340A_ABST
    Figure CN123113340A_ABST
Patent Text Reader

Abstract

This invention relates to the field of intelligent optimization control technology for non-ferrous metallurgical processes, specifically disclosing a parameter optimization control method for the sulfurization process of waste acid resource recovery based on reinforcement learning. This method constructs the current state by collecting operational data of the waste acid resource recovery sulfurization process and the ORP values ​​at the outlet of each reactor; determines the action change boundary based on the differential statistics of control parameters at adjacent sampling times in historical operational data; inputs the current state into the reinforcement learning model to generate the original control action; constrains the change in the original control action relative to the actual control action in the previous control cycle according to the action change boundary to obtain the actual control action and acquire the next state; calculates the comprehensive reward based on resource recovery effect, sulfurization cost, and operational trend, and updates the reinforcement learning model accordingly, thereby achieving dynamic optimization control of sulfurization process parameters.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of intelligent optimization control technology for non-ferrous metallurgical processes, and specifically discloses a parameter optimization control method for the resource recovery and sulfidation process of waste acid based on reinforcement learning. Background Technology

[0002] Waste acid is a type of industrial waste acid generated during non-ferrous metallurgical production, which typically contains metal ions with recycling value. In the resource recovery process of waste acid, a sulfiding agent can be added to the waste acid to cause the target metal ions to form corresponding sulfide precipitates, thereby achieving the separation and recovery of valuable metal resources. The reaction state of the sulfidation process is related to factors such as the waste acid flow rate, metal ion concentration, amount of sulfiding agent added, and reactor operating status. The oxidation-reduction potential (ORP) value at the outlet of each reactor can be used to characterize the reaction state of the corresponding reaction stage.

[0003] In actual production, the flow rate of waste acid and the concentration of metal ions vary with the upstream production conditions, resulting in differences in the required amount of sulfiding agent at different times. When the sulfiding agent is insufficient, the sulfidation reaction state in each reactor cannot meet the corresponding process requirements, affecting the sulfidation precipitation of valuable metals and subsequent resource recovery. When the sulfiding agent is excessive, it increases the consumption of sulfiding agents and causes unnecessary treatment costs. Therefore, the waste acid resource recovery sulfidation process is not simply a matter of ORP setpoint control, but requires coordination of sulfiding agent consumption while meeting the sulfidation reaction control requirements, taking into account the real-time raw material status and reaction status.

[0004] Current sulfurization process parameter control typically relies on historical operating experience, pre-set control rules, or mathematical models establishing the relationship between process parameters and control objectives to determine control parameters. However, the sulfurization process for waste acid resource recovery exhibits nonlinear, time-varying, and multivariate coupling characteristics. When the state of the waste acid feed material changes, fixed control rules or pre-established parameter mapping relationships are difficult to adapt to the new operating state in a timely manner. While data-driven optimization models can establish a correlation between process state and control results using historical operating data, they do not adequately consider the dynamic constraints between control actions before and after the continuous sulfurization process. When the control parameters output by the model change too much, it can easily cause fluctuations in the amount of sulfurizing agent added and the reactor outlet ORP. At the same time, a single control objective cannot simultaneously reflect the control requirements for resource recovery effect, sulfurizing agent consumption, and process operational stability.

[0005] It is evident that how to generate continuous control actions based on the real-time operating status of the waste acid resource recovery sulfidation process, and how to ensure that the ORP at the outlet of each reactor converges to the corresponding target state while meeting the dynamic adjustment range of the actual production process, and how to coordinate the resource recovery effect, sulfidation cost, and process operation stability, are problems that need to be solved in the parameter optimization control of the waste acid resource recovery sulfidation process. In view of this, the present invention provides a reinforcement learning-based parameter optimization control method for the waste acid resource recovery sulfidation process to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a parameter optimization control method for the waste acid resource recovery sulfidation process based on reinforcement learning. By incorporating the waste acid sulfidation process state information, the reasonable change law of control parameters during historical operation, as well as resource recovery, sulfidation cost and operation trend into the reinforcement learning strategy optimization process, the control action output by the reinforcement learning model meets the change constraints of the actual production process, and takes into account the resource recovery effect and sulfidation treatment cost while achieving the target control of reactor outlet ORP.

[0007] To achieve the above objectives, the basic solution of this invention provides a method for optimizing and controlling parameters in the sulfidation process of waste acid resource recovery based on reinforcement learning, comprising the following steps: Step S1: Collect the operating data of the waste acid resource utilization sulfidation process and the ORP value of each reactor outlet to construct the current status; Step S2: Based on the differential statistics of control parameters at adjacent sampling times in historical operation data, determine the action change boundary of the control parameters; Step S3: Input the current state into the reinforcement learning model to generate the original control action. Constrain the change of the original control action relative to the actual control action of the previous control cycle according to the action change boundary to obtain the actual control action of the current control cycle. Apply the actual control action to the waste acid resource utilization sulfidation process to obtain the next state. Step S4: Calculate the comprehensive reward based on the resource recovery effect, vulcanization cost and operating trend corresponding to the actual control action, and update the reinforcement learning model based on the current state, actual control action, comprehensive reward and next state; Steps S3 and S4 are repeated using the next state as the new current state.

[0008] Furthermore, the current status includes the waste acid flow rate, metal ion concentration, sulfurizing agent addition amount, reactor operating parameters, and ORP value at the outlet of each reactor.

[0009] Furthermore, the boundary of the action change is determined based on the differential sequence of historical control parameters under normal operating conditions.

[0010] Furthermore, the change in the original control action relative to the actual control action in the previous control cycle is limited according to the action change boundary, and the actual control action in the current control cycle is determined based on the limited change.

[0011] Furthermore, when there are multiple control parameters to be optimized, constraints are applied according to the action change boundaries corresponding to each control parameter to be optimized. The control parameters to be optimized include at least the vulcanizing agent addition parameters.

[0012] Furthermore, in step S3, the change in the original control action between the original control action and the actual control action of the previous control cycle is calculated: ; The original motion change amount is constrained based on the motion change boundary to obtain the constrained motion change amount. , so that: ; And when the original change in motion is within the boundary of the change in motion, ; The actual control action for the current control cycle is determined based on the change in the constraint action. ; in, Indicates the original control action. This indicates the actual control action taken in the previous control cycle. This indicates the actual control action taken during the current control cycle.

[0013] Furthermore, the current state, actual control action, comprehensive reward, and next state are used to form a state transition sample, and the policy network and value network are updated based on the state transition sample.

[0014] Furthermore, the comprehensive reward is obtained by weighting the reward items corresponding to resource recovery effect, sulfidation cost and operation trend.

[0015] Furthermore, the resource recovery performance index is determined based on the deviation of the ORP value at the outlet of each reactor from the corresponding target ORP value; The economic indicators of the vulcanization process are determined based on the vulcanizing agent consumption corresponding to the current control action. The operational trend index is determined based on the changes in the ORP values ​​at the reactor outlets and the changes in the control parameters at adjacent sampling times. The corresponding reward items are obtained based on the resource recovery effect indicators, sulfidation process economic indicators, and operation trend indicators, and the comprehensive reward is obtained by weighting each reward item.

[0016] Furthermore, after the reinforcement learning model is trained, the original control actions are generated based on the real-time collected data of the waste acid resource utilization sulfidation process. After the action change boundary constraints are applied, the actual control actions are obtained, and parameter optimization control is repeated according to the control cycle.

[0017] The principle and effect of this solution are as follows: Compared with existing technologies, this invention constructs a reinforcement learning state space by using the operating parameters of the waste acid resource recovery sulfidation process and the ORP of each reactor outlet. This enables the reinforcement learning strategy to determine the control action for the next control cycle based on the waste acid feed status, the current sulfiding agent addition status, and the reactor feedback status. Furthermore, it determines the range of control parameter variation based on the differential statistical results of adjacent time points of control parameters in historical stable operating data, and applies the range of variation to the action output of the strategy network. This ensures that the control parameters for the next control cycle are not only affected by the current process status but also constrained by the dynamic changes of the actual control parameters in the previous control cycle, thereby reducing the fluctuations in the sulfidation process caused by abrupt control actions output by the agent.

[0018] This invention further integrates resource recovery efficiency, economic indicators of the sulfidation process, and operational trend indicators into a multi-objective reward, and uses the multi-objective reward after the action is executed as the basis for updating the reinforcement learning strategy, thus forming a closed loop between control parameter differential constraints, ORP feedback at each reactor outlet, and multi-objective reward evaluation. Therefore, the reinforcement learning model can learn the temporal relationship between process state, control actions, and control results during continuous operating condition changes. This allows the ORP at each reactor outlet to converge towards the corresponding target value while suppressing unnecessary sulfiding agent consumption and control action fluctuations, achieving dynamic optimization of control parameters in the waste acid resource recovery sulfidation process. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of a reinforcement learning-based parameter optimization and control method for the resource recovery sulfidation process of waste acid is shown in an embodiment of this application. Figure 2 The flowchart of a parameter optimization control method for the resource recovery sulfidation process of waste acid based on reinforcement learning proposed in an embodiment of this application is shown. Figure 3 This illustration shows a schematic diagram of a reinforcement learning-based parameter optimization control method for the resource recovery and sulfidation process of waste acid, as proposed in an embodiment of this application. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0022] This embodiment provides a method for optimizing and controlling parameters in the sulfidation process of waste acid resource recovery based on reinforcement learning, such as... Figures 1 to 3 As shown, this method uses the waste acid resource recovery sulfidation process as the control object of reinforcement learning. The environmental state at the current moment is constructed based on the waste acid feed state, current control parameters, and ORP of each reactor outlet. The agent outputs the control action corresponding to the next control cycle based on the current environmental state. After the control action is executed, the updated sulfidation process state is obtained. The comprehensive reward is calculated based on the resource recovery effect, economic indicators, and operation trend indicators corresponding to the control action, so that the agent updates the control strategy based on continuous state, action, and reward feedback.

[0023] In this embodiment, the training process and parameter optimization control process of the reinforcement learning model include the following steps.

[0024] Step S1: Collect operational data of the waste acid resource utilization sulfidation process and construct the state space of the reinforcement learning environment.

[0025] Operational data for the waste acid resource utilization sulfidation process are collected according to a preset sampling period. The operational data includes waste acid flow rate, metal ion concentration, sulfiding agent addition amount, reactor operating parameters that reflect the current operating status of the reactor, and the ORP value at the outlet of each reactor.

[0026] Among them, the waste acid flow rate and metal ion concentration are used to characterize the raw material load entering the sulfidation process; the amount of sulfiding agent added is used to characterize the agent addition status in the current control cycle; and the ORP value at the outlet of each reactor is used to characterize the current reaction status of each stage of the sulfidation reactor. By simultaneously inputting raw material side information, control side information, and reaction feedback information, the same state space can reflect changes in waste acid feed, the current control status, and the actual sulfidation reaction results.

[0027] The operational data obtained from different data acquisition sources are time-aligned according to the sampling time. Data that significantly exceeds the effective range of the corresponding measuring point and missing data are processed to form a vulcanization process state sample arranged in chronological order.

[0028] At any moment The reinforcement learning environment state is represented as: ; in, express Constantly reinforce the learning environment. express The first moment A state variable, such as Let be the first state variable at time t. Indicates the number of state variables.

[0029] For a waste acid resource recovery sulfidation process involving multiple sulfidation reactors, the outlet ORP of each reactor is used as an independent state variable in the state space and corresponds to a pre-determined target ORP, so that the agent can distinguish the actual reaction state of different reaction stages.

[0030] Because the waste acid flow rate, metal ion concentration, sulfiding agent dosage, reactor operating parameters, and ORP have different dimensions and numerical ranges, in order to reduce the impact of state variables with different dimensions on the network training process, the state variables are normalized before being input into the reinforcement learning model. As one implementation method, for the first... Each state variable can be normalized based on the valid range of values ​​for the corresponding state variable in the training data.

[0031] A normalized state vector is formed by all normalized state variables, and this normalized state vector is used as the state input for the policy network and the value network.

[0032] It should be noted that the normalization method is not limited to the above methods; while maintaining the same parameter transformation rules in the training and control phases, other data standardization methods that can eliminate the differences in the dimensions of different process parameters can also be used.

[0033] Step S2: Establish control action constraints based on the differential statistical results of control parameters.

[0034] Historical operating data of the waste acid resource utilization sulfidation process is obtained, continuous time series data in normal operating condition are selected from the historical operating data, and the actual control parameters corresponding to each sampling time are extracted.

[0035] The following explanation uses a single control parameter as an example. Let the control parameter at time t be denoted as... The control parameters at the previous sampling time are denoted as The change in control parameters between two adjacent sampling times is: ; in, for Timing control parameters; These are the control parameters from the previous moment; This represents the change in control parameters between two adjacent control cycles.

[0036] The changes in control parameters between adjacent sampling times in historical operating data are calculated sequentially to obtain a control parameter difference sequence. Statistical analysis is then performed on this control parameter difference sequence to determine the permissible reduction boundary of control parameters under normal operating conditions. and allowable increase in the boundary of change Thus we get: ; in, The minimum allowable change in the control parameter; This represents the maximum permissible variation in the control parameter.

[0037] The purpose of differential statistics of control parameters is not to restrict control parameters to a fixed value, but to constrain the allowable range of control quantity changes between two adjacent control cycles by utilizing the continuous control laws that have been formed in the actual production process.

[0038] As one possible implementation method, after removing abnormal operation samples corresponding to equipment failure, sensor anomalies, and manual emergency intervention, the statistical distribution of the remaining effective difference samples can be used to determine... and For example, preset lower quantiles and preset upper quantiles of the effective difference sequence can be used as the change boundaries to reduce the impact of a small number of abnormal control actions on the constraint range.

[0039] When the agent outputs the original control action based on the current state First, calculate the change in the original control action relative to the previous actual control action: ; The change is then constrained: ; And receive the actual control actions issued: ; The actual control actions performed; This represents the change in the original control action relative to the previous actual action. This means that the input value is restricted to the corresponding upper and lower boundaries.

[0040] In this way, when the original control action output by the policy network changes too much compared to the actual control action of the previous control cycle, the original action is not directly sent to the vulcanization process. Instead, it is first corrected according to the allowable range of control parameter changes during normal operation in history, and then the actual control action is formed. Therefore, the reinforcement learning model can still change the control parameters according to the real-time state, while avoiding sudden changes in control quantity that exceed the actual production operation pattern between two consecutive control cycles.

[0041] When multiple control parameters to be optimized exist simultaneously in the process of resource recovery and sulfidation of waste acid, the control action is represented as an action vector. Each action component establishes a change boundary based on the historical difference statistics of the corresponding control parameter, and is constrained in the manner described above.

[0042] In this embodiment, the control parameters include at least the vulcanizing agent addition parameters for changing the vulcanization reaction state. During actual control, the vulcanizing agent addition state for the next control cycle is changed according to the constrained control actions output by the intelligent agent, allowing the vulcanization process to be dynamically adjusted based on the incoming material load and the current reaction state.

[0043] Step S3: Generate constrained control actions based on MOD-DDPG.

[0044] This embodiment employs the Improved Deep Deterministic Policy Gradient Algorithm (MOD-DDPG) to establish a reinforcement learning policy. MOD-DDPG uses an actor-critic structure and includes a policy network, a value network, a corresponding target network, and an experience replay mechanism.

[0045] Among them, the policy network is used to generate continuous control actions based on the current normalized state vector; the value network is used to evaluate the cumulative reward that can be obtained after taking the corresponding control action in the current state; the target network is used to calculate the target value in the network update process; and the experience replay mechanism is used to store the state transition samples formed by the interaction between the agent and the sewage sulfidation process, and to extract training samples from the historical interaction samples in the model update process.

[0046] The policy network generates the initial control action based on the current state: ; in, This represents the control action output by the intelligent agent. Represents the policy network, This represents the actor's network parameters.

[0047] The original control action is processed by the differential constraints of the control parameters established in step S2 to form the actual control action. Therefore, compared with directly using the strategy network output as production control instructions, this embodiment adds action change constraints based on historical normal operation data between the strategy network and the actual vulcanization process, so that the strategy search process is limited by the dynamic adjustment capability of the actual industrial process.

[0048] Actual control actions After being applied to the corresponding reinforcement learning environment, the environment state at the next time step is obtained. During the model training phase, the current state, the actual control action, the corresponding reward, and the next state are combined to form a state transition sample: ; in, This is the current state; The constrained control actions that are actually executed; The overall reward obtained after performing this control action; This indicates the state at the next moment.

[0049] State transition samples are stored in the experience replay pool.

[0050] During training, N state transition samples are drawn from the experience replay pool. For the ... For each sample, the target value is calculated using the target policy network and the target value network. The corresponding state-action value is calculated using the current value network. The loss function of the value network is expressed as: ; in, For value network loss; The number of samples in a training batch; Number the training samples; For the first The target value corresponding to each sample; Predicted state-action value for the preceding value network; These are the parameters of the value network.

[0051] The policy network is updated based on the evaluation of the state-action value by the value network, so that the policy network outputs control actions that increase cumulative rewards. The target policy network and the target value network are softly updated by the corresponding current network according to a preset update ratio to reduce training oscillations caused by rapid changes in the target value during network training.

[0052] The improvements of MOD-DDPG over the basic continuous action reinforcement learning control process in this embodiment are at least reflected in the following aspects: On the one hand, differential constraints of control parameters based on actual historical operating data are introduced at the output end of the policy network, so that the agent's action search meets the dynamic adjustment law of the waste acid resource recovery sulfidation process; on the other hand, the resource recovery effect, sulfidation cost and operating trend are all incorporated into the reinforcement learning reward, so that the agent does not determine the control strategy based solely on a single ORP deviation or a single reagent consumption.

[0053] Step S4: Calculate the multi-objective reward and update the MOD-DDPG strategy based on the reward feedback.

[0054] After the actual control action is applied to the waste acid resource utilization sulfidation process, the operation data of the waste acid resource utilization sulfidation process and the ORP value of each reactor outlet are collected at the next sampling time. Based on the data, the multi-objective reward corresponding to the current control action is calculated.

[0055] Multi-objective rewards are represented as follows: ; in, This represents the comprehensive reward corresponding to the actual control action at time t. This indicates the reward items corresponding to the resource recycling effectiveness indicators. This indicates the reward items corresponding to the economic indicators of the vulcanization process. This indicates the reward items corresponding to the process's operational trend; , and These represent the weights of the corresponding reward items.

[0056] To avoid the different dimensions and numerical ranges of different reward items causing an excessive influence of a particular reward item on the overall reward, all reward items are converted to a unified evaluation scale before weighting.

[0057] In one implementation, resource recovery effectiveness is evaluated online using the deviation of each reactor outlet ORP from the corresponding target ORP. One reactor in The actual exit ORP at time is denoted as , will the The target ORP corresponding to each reactor is denoted as . Then, a resource recovery effectiveness evaluation metric is constructed based on the ORP deviation of each reactor level: ; in, This indicates the number of reactors participating in the evaluation. Indicates the first The evaluation weights corresponding to each reactor.

[0058] When the outlet ORP of each reactor is closer to the corresponding target ORP Reduce, and make Increase the reward; decrease the corresponding reward when the ORP deviates from the target reaction state. This guides the agent to adjust the sulfurizing agent addition parameters based on changes in the incoming waste acid, causing each reaction stage to converge towards the target reaction state.

[0059] When the production site has data on the concentration of metal ions or the metal recovery rate before and after sulfidation, the actual resource recovery rate and the deviation from the ORP target can be used together to calculate the resource recovery effect reward, thereby using the actual resource recovery results to correct the ORP control target.

[0060] Economic indicators are used to evaluate the consumption of vulcanizing agents corresponding to the current control action. Within the same evaluation period, when the amount of vulcanizing agent added, or the agent cost calculated based on the amount of vulcanizing agent added, increases, the consumption rate decreases. While meeting the reaction state requirements of the vulcanization process, reducing unnecessary vulcanizing agent consumption can improve... This avoids the situation where the agent only achieves ORP tracking by continuously increasing the amount of vulcanizing agent added.

[0061] Operating trend indicators are used to evaluate the changing state of the vulcanization process between consecutive control cycles. As one implementation method, an operating trend evaluation metric is constructed based on the changes in ORP at the reactor outlet and the changes in control parameters at adjacent sampling times. When ORP experiences continuous large fluctuations or control parameters change frequently between adjacent control cycles, the indicator is reduced. When the ORP change tends to stabilize and the control parameters remain continuously changing, increase .

[0062] Therefore, the multi-objective reward is not three independent outcome indicators, but rather evaluates the same control action from three aspects: the sulfurization reaction target, reagent consumption, and continuous operation status. If the agent simply increases the amount of sulfurizing agent added, although it can change the corresponding reactor outlet ORP, it will also be limited by the economic indicator reward; if the agent causes the ORP to deviate from the corresponding target state in order to reduce reagent consumption, it will be limited by the resource recovery effect reward; if the agent frequently changes the control action, it will also be limited by the operation trend reward and the control action differential constraint in step S2.

[0063] After the comprehensive reward calculation is completed, Write the experience into the experience replay pool and update the policy network and value network based on the experience samples.

[0064] Subsequently ordered That is, the next state generated after the current control cycle is completed. In the next control cycle, it will serve as the new current state. .

[0065] The processes of state construction, action generation, action differential constraints, environmental interaction, and multi-objective reward calculation are re-executed. Thus, steps S3 and S4 are cyclically executed during the reinforcement learning training phase, enabling the agent to gradually establish the policy mapping relationship between the sulfurization process state, control parameters, and control effects.

[0066] After the model training meets the preset stopping conditions, the parameters of the trained strategy network are saved. In the online control stage of the actual waste acid resource utilization sulfidation process, the latest waste acid flow rate, metal ion concentration, current sulfiding agent addition amount, reactor operating parameters, and ORP of each reactor outlet are collected according to a fixed control cycle. The current state is formed according to the same data processing method as in the training stage. The current state is input into the trained strategy network to obtain the original control action. Then, the actual control action is determined according to the differential constraints in step S2, and the actual control action is sent to the corresponding sulfidation process control equipment.

[0067] The process status is re-acquired in the next control cycle, and the above process is repeated to continuously adjust the sulfidation process control parameters based on changes in the waste acid feed and actual feedback from the reactor.

[0068] During the model training phase, initial empirical data can be constructed using historical continuous operation data of the waste acid resource recovery sulfidation process. Following the sampling time sequence, the states, actual control parameters, and the state at the next sampling time are formed into time-series samples. Rewards are calculated based on the resource recovery effect, sulfidation agent consumption, and operational trends corresponding to each sample. The resulting state transition samples are then written into the experience playback pool.

[0069] When a vulcanization process simulation environment or a data-driven process environment that can generate the next process state based on the control action is adopted, the agent can also interact in the training environment according to the control action after differential constraint to obtain new state transition samples, and use them together with historical running samples for MOD-DDPG training.

[0070] When actual production equipment participates in model interactive training, any control action output by the policy network is first constrained by the change of control parameters in step S2 before being issued to the actual vulcanization process, in order to prevent control actions that significantly exceed the historical normal adjustment range from being generated during the training phase.

[0071] like Figure 3 As shown, in any control cycle The reinforcement learning environment provides the agent with the current state. The agent outputs an action based on the current state. After the environment performs the action, it transitions to the next state. Simultaneously, rewards are generated based on a multi-objective reward function. Next state The current state is input into the agent again as the next control cycle, thus forming a continuous closed loop.

[0072] Among them, actions Before the actual output, it is subject to the boundary constraints of the differential statistics of historical control parameters, and the reward... It also includes evaluations of resource recovery effectiveness, economic indicators, and operational trends. Therefore, the reinforcement learning interaction process in this embodiment simultaneously features process state feedback, dynamic constraints on control actions, and multi-objective result feedback.

[0073] To verify the above-mentioned reinforcement learning-based parameter optimization and control method for the waste acid resource recovery sulfidation process, the MOD-DDPG of this embodiment was tested with four comparative schemes: SVR, BC, IDPG, and EAG-AC, under the same historical operating condition dataset of the waste acid sulfidation process and the same evaluation criteria.

[0074] The comprehensive optimization benefit value is used as the evaluation index. The comprehensive optimization benefit value is obtained based on three evaluation directions: metal resource recovery effect, sulfidation agent cost, and process operation stability, and is consistent with the evaluation direction of the multi-objective reward function in this embodiment. Under the same dataset and evaluation rules, the larger the comprehensive optimization benefit value, the better the overall coordination result of the corresponding scheme for the three evaluation objectives.

[0075] Table 1. Comparison of the comprehensive optimization benefits of the five optimization methods

[0076] As shown in Table 1, under the same test data and evaluation rules, the comprehensive optimization benefit value of MOD-DDPG in this embodiment is higher than that of the other four comparative schemes. This indicates that by introducing the state feedback of the waste acid resource recovery sulfidation process, the differential constraint of control parameters, and the multi-objective reward into the reinforcement learning control process, the agent can simultaneously consider the ORP target state, the consumption of sulfidation reagents, and the process change trend when adjusting the control parameters of the sulfidation process, thereby improving the coordination ability among multiple control objectives.

[0077] It should be noted that, under different sources of waste acid, reactor stages, metal ion composition, and production loads, the effective range of each state variable, the target ORP of each reactor, the differential constraint range of control parameters, and the multi-objective reward weights can be determined based on the historical operating data and process control requirements of the corresponding production process. After the model is deployed, it will still be controlled according to the same state construction rules, action constraint rules, and reward evaluation logic, thereby adapting to the operating conditions of the corresponding waste acid resource recovery sulfidation production line.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any indirect modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for optimizing control of process parameters of a resource utilization of waste acid sulfidation based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Collect the operating data of the waste acid resource utilization sulfidation process and the ORP value of each reactor outlet to construct the current status; Step S2: Based on the differential statistics of control parameters at adjacent sampling times in historical operation data, determine the action change boundary of the control parameters; Step S3: Input the current state into the reinforcement learning model to generate the original control action. Constrain the change of the original control action relative to the actual control action of the previous control cycle according to the action change boundary to obtain the actual control action of the current control cycle. Apply the actual control action to the waste acid resource utilization sulfidation process to obtain the next state. Step S4: Calculate the comprehensive reward based on the resource recovery effect, vulcanization cost and operating trend corresponding to the actual control action, and update the reinforcement learning model based on the current state, actual control action, comprehensive reward and next state; Steps S3 and S4 are repeated using the next state as the new current state.

2. The method according to claim 1, wherein, The current status includes the waste acid flow rate, metal ion concentration, sulfurizing agent addition amount, reactor operating parameters, and ORP value at the outlet of each reactor.

3. The method according to claim 1, wherein, The boundary of the action change is determined based on the differential sequence of historical control parameters under normal operating conditions.

4. The method according to claim 3, wherein, The change in the original control action relative to the actual control action in the previous control cycle is limited according to the action change boundary, and the actual control action in the current control cycle is determined based on the limited change.

5. The method according to claim 4, wherein, When there are multiple control parameters to be optimized, constraints are applied according to the action change boundaries corresponding to each control parameter to be optimized. The control parameters to be optimized include at least the vulcanizing agent addition parameters.

6. The method for optimizing and controlling parameters of the sulfidation process for the resource recovery of waste acid based on reinforcement learning according to claim 4, characterized in that, In step S3, the change in the original control action between the original control action and the actual control action in the previous control cycle is calculated: ; The original motion change amount is constrained based on the motion change boundary to obtain the constrained motion change amount. , so that: ; And when the original change in motion is within the boundary of the change in motion, ; The actual control action for the current control cycle is determined based on the change in the constraint action. ; in, Indicates the original control action. This indicates the actual control action taken in the previous control cycle. This indicates the actual control action taken during the current control cycle.

7. The method for optimizing and controlling parameters of the sulfidation process for the resource recovery of waste acid based on reinforcement learning as described in claim 6, characterized in that, The current state, actual control action, comprehensive reward, and next state constitute a state transition sample, and the policy network and value network are updated based on the state transition sample.

8. The method for optimizing and controlling parameters of the sulfidation process for the resource recovery of waste acid based on reinforcement learning according to claim 1, characterized in that, The comprehensive reward is obtained by weighting the reward items corresponding to resource recovery effect, vulcanization cost and operation trend.

9. The method for optimizing and controlling parameters of the sulfidation process for the resource recovery of waste acid based on reinforcement learning according to claim 1, characterized in that, The resource recovery performance index is determined based on the deviation of the ORP value at the outlet of each reactor from the corresponding target ORP value. The economic indicators of the vulcanization process are determined based on the vulcanizing agent consumption corresponding to the current control action. The operational trend index is determined based on the changes in the ORP values ​​at the outlet of each reactor and the changes in the control parameters at adjacent sampling times. The corresponding reward items are obtained based on the resource recovery effect indicators, sulfidation process economic indicators, and operation trend indicators, and the comprehensive reward is obtained by weighting each reward item.

10. The method for optimizing and controlling parameters of the sulfidation process for the resource recovery of waste acid based on reinforcement learning according to claim 1, characterized in that, After the reinforcement learning model is trained, the original control actions are generated based on the real-time collected data of the sulfurization process of waste acid resource utilization. The actual control actions are obtained after the action change boundary constraints are applied, and parameter optimization control is repeated according to the control cycle.