Reinforcement learning-based anaerobic digestion process methane yield extremum search method and system

Abstract

The application discloses a kind of based on reinforcement learning's anaerobic digestion process methane production rate extreme value search method and system, it is related to anaerobic digestion process intelligent control technical field.For the existing extreme value search scheme cannot simultaneously consider optimization steady-state accuracy and convergence speed, classical algorithm exists fixed amplitude disturbance and causes steady-state oscillation, improved no steady-state oscillation scheme exists integrator gain fixed, slow convergence problem, the application constructs extreme value search control framework, realizes sine disturbance signal amplitude adaptive update by exponential decay module, introduces DDPG deep reinforcement learning algorithm, defines adaptive state, action and multi-objective reward function, realizes the online adaptive optimization of integrator gain, completes methane production rate extreme value search closed-loop control.The application can eliminate steady-state oscillation, improve optimization convergence speed simultaneously, significantly improve anaerobic digestion process running efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for anaerobic digestion processes, and in particular to a method and system for searching the extreme value of methane yield in anaerobic digestion processes based on reinforcement learning. Background Technology

[0002] Anaerobic digestion technology enables the harmless disposal of organic waste and the efficient recovery of biomass energy, serving as a core technological pathway for the resource utilization of organic solid waste. Methane yield is a key indicator for evaluating the operational efficiency and economic benefits of anaerobic digestion processes. For the complex biochemical process of anaerobic digestion, characterized by strong nonlinearity, large time lag, and multivariate coupling, the extreme value search control algorithm has become the mainstream control method for tracking the optimal operating point of methane yield because it can achieve output extreme value optimization without relying on precise mechanistic models.

[0003] Existing classical extremum search algorithms extract gradient information and find extrema by continuously applying a sinusoidal perturbation signal of fixed amplitude to the control input dilution rate. However, this approach causes the dilution rate to fluctuate slightly, leading to synchronous steady-state oscillations in the output methane yield. This severely reduces the steady-state accuracy of the extremum search and results in continuous methane yield loss. To address the steady-state oscillation problem, an extremum search scheme without steady-state oscillations based on exponentially decaying sinusoidal perturbations has been proposed. This scheme can eliminate steady-state oscillations by adaptively decaying the perturbation amplitude after the system reaches steady state. However, this introduces a new problem: a significant reduction in the convergence speed of the extremum search.

[0004] The integrator gains of the above schemes are all set with fixed values, which cannot be adaptively adjusted according to the operating status of the anaerobic digestion process. It is difficult to simultaneously take into account the convergence speed and steady-state accuracy of the optimization process, and cannot meet the dynamic response requirements of substrate fluctuations and operating condition changes in the anaerobic digestion process. This restricts the further improvement of the operating efficiency of the anaerobic digestion process and its large-scale engineering application. Summary of the Invention

[0005] To address this, this invention provides a method and system for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning. This method solves the problems in existing technologies where the extreme value search scheme for methane yield in anaerobic digestion cannot simultaneously consider both optimization convergence speed and steady-state accuracy. Classical extreme value search algorithms suffer from steady-state oscillations caused by fixed-amplitude sinusoidal perturbations and insufficient steady-state accuracy. In contrast, the improved exponentially decaying steady-state oscillation-free scheme suffers from fixed integrator gain that cannot be adaptively adjusted and a significant reduction in convergence speed.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a method for searching the extreme value of methane yield in an anaerobic digestion process based on reinforcement learning. This method includes the following steps:

[0007] Step S1: Determine the control input variables and controlled output variables of the anaerobic digestion process, and construct a search control framework for the extreme value of methane yield;

[0008] Step S2: Based on the control framework, the gradient information of the output methane yield of the anaerobic digestion process is extracted through the filter bank, and the gradient information is input into the pre-established exponential decay module to realize the adaptive update of the amplitude of the sinusoidal perturbation signal for extreme value search.

[0009] Step S3: Construct an adaptive optimization module for the parameters of the extreme value search integrator using the DDPG algorithm in deep reinforcement learning. The DDPG algorithm adopts an Actor-Critic dual network architecture and stabilizes the network training process through an experience replay mechanism and a soft update mechanism.

[0010] Step S4: Define the state space, action space, and reward function required for training the DDPG algorithm, and use the anaerobic digestion process as the environment object for real-time interaction of the DDPG algorithm;

[0011] Step S5: Based on the real-time operational data of the anaerobic digestion process, iteratively update the network parameters of the Actor network and Critic network in the DDPG algorithm;

[0012] Step S6: Based on the adaptive integrator gain, the sinusoidal perturbation signal with adaptive amplitude output by the updated DDPG algorithm, and the extracted gradient information, perform extreme value search and closed-loop optimization control on the methane yield of the anaerobic digestion process.

[0013] Preferably, in step S1, the control input variable for the anaerobic digestion process is the dilution rate. The bottom stream acceleration rate affects the matrix concentration in the reactor through the dilution rate; the controlled output variable is the methane yield. During the extreme value search control process, when the dilution rate is input... The superimposed amplitude is , frequency is For a sinusoidal disturbance signal, the mapping relationship between the input and output variables is as follows:

[0014] ;

[0015] In the formula, This is a real-time estimate of the unknown optimal input dilution rate. Let be the nonlinear mapping function from dilution rate to methane yield in the anaerobic digestion process. It is a continuous-time variable.

[0016] Preferably, in step S2, the pre-established sinusoidal disturbance signal amplitude structure of the exponential decay module is as follows:

[0017] ;

[0018] In the formula, The amplitude of the sinusoidal disturbance signal. The initial amplitude of the sinusoidal disturbance signal is given. This is a custom adjustment coefficient based on the characteristics of the anaerobic digestion process, used to control the decay rate of the disturbance signal amplitude. To extract the methane yield gradient information; the output value of the exponential decay module ranges from 0 to 1, and is related to the gradient information. The absolute value is positively correlated with the gradient information in the initial stage of optimization. When the disturbance signal is large, it maintains a high amplitude; when approaching the optimal value, the gradient information... Reduced, disturbance amplitude decays synchronously, gradient information in steady state Approaching 0, the disturbance amplitude decays to a negligible level.

[0019] Preferably, in step S3, the DDPG algorithm includes an Actor policy network, a Critic value network, and an experience replay pool;

[0020] The Actor policy network is used to adjust according to the real-time state of the environment. Output the action that maximizes value Q in this state. ;

[0021] The Critic value network is used to analyze the current state. With action Perform a value assessment and output the corresponding value function. ;

[0022] The experience replay pool is used to store interaction data... The data is stored in a specific format, and samples are randomly selected for network training. This is the reward value at the current moment. This represents the environmental state at the next moment.

[0023] Preferably, in step S4, the state space required for training the DDPG algorithm is the real-time input dilution rate of the anaerobic digestion process. With real-time output methane yield The action space required for training the DDPG algorithm is the integrator gain of the extreme value search controller. The integrator gain is updated online adaptively using the DDPG algorithm.

[0024] Preferably, in step S4, the reward function consists of a high-output reward. Production growth rate bonus Input fluctuation penalty The weighted sum of the three parts is obtained, and the specific expression is as follows:

[0025] ;

[0026] in:

[0027] ;

[0028] ;

[0029] ;

[0030] In the formula, , , They are respectively time, time, methane yield at time t, for dilution rate at time, The sampling interval; , They are respectively time, Integrator gain at time step The penalty coefficient is... These are the weighting coefficients.

[0031] Preferably, in step S5, the network parameters of the Actor network are... The gradient ascent algorithm is used for iterative updates. By adjusting the parameters, the long-term cumulative reward of the anaerobic digestion process is continuously increased. The update formula is as follows:

[0032] ;

[0033] in, For the first In the next iteration, the updated Actor network policy parameters For the first In the next iteration, the policy parameters of the Actor network, The learning rate of the Actor network. For the objective function For strategy parameters The gradient of is expressed as:

[0034] ;

[0035] In the formula, For expectation operations, represent the state. Obedience strategy State distribution under , Output action pair parameters for the Actor network gradient, In the strategy parameters Under, state The corresponding action output; For the output of the Critic network Value-pair action gradient, In the state Next action And follow the strategy The expected cumulative reward that can be obtained This indicates the action during gradient calculation. The value is the action output by the current policy network.

[0036] Preferably, in step S5, the network parameters of the Critic network are... By iteratively updating by minimizing the mean square Bellman error, the update formula is:

[0037] ;

[0038] in, For the first In the next iteration, the updated Critic network parameters For the first In the next iteration, the parameters of the Critic network... The learning rate of the Critic network. The mean square Bellman error loss function Critic network parameters gradient, Let Bellman's mean square error loss function be expressed as follows:

[0039] ;

[0040] In the formula, The expectation operator indicates that the average loss is calculated over a batch of sampled data. For parameters Under these conditions, the Critic network considers the state. and actions Output value, This is the Bellman target value.

[0041] Preferably, in step S6, the method for extreme value search and closed-loop optimization control of methane yield in the anaerobic digestion process based on the adaptive integrator gain, the sinusoidal perturbation signal with adaptive amplitude, and the extracted gradient information output by the updated DDPG algorithm specifically includes:

[0042] Step S61: Initialize the input dilution rate for the anaerobic digestion process;

[0043] Step S62: Superimpose a sinusoidal perturbation signal with adaptive amplitude onto the input dilution rate;

[0044] Step S63: Measure the methane yield from the anaerobic digestion process in real time, using a cutoff frequency of... The high-pass filter and the cutoff frequency are A filter bank composed of low-pass filters is used for filtering to remove the DC component, retain the AC response, and extract the gradient information of methane yield. ;

[0045] Step S64: The DDPG algorithm observes the input dilution rate in real time at the current moment. With output methane yield The output strategy action is the integrator gain. This enables online adaptive updating of the integrator gain.

[0046] Step S65: Transfer gradient information With adaptive integrator gain Multiplication drives the dilution rate to converge toward the optimal value;

[0047] Step S66: Determine gradient information If the value approaches 0, then proceed to step S62 to continue optimizing for the extreme value; if it does, then continue to output the current optimal dilution rate to maintain the steady-state maximum value of the methane yield.

[0048] This invention also provides a reinforcement learning-based methane yield extremum search system for anaerobic digestion processes. This system is used to implement the reinforcement learning-based methane yield extremum search method described above, specifically including:

[0049] The module for defining variables and constructing an extreme value search control framework for the anaerobic digestion process is used to determine the control input variables and controlled output variables of the anaerobic digestion process and to construct an extreme value search control framework for methane yield.

[0050] The gradient information extraction and sinusoidal perturbation signal amplitude adaptive adjustment module is used to extract the gradient information of the output methane yield of the anaerobic digestion process through a filter bank based on the control framework, and input the gradient information into a pre-established exponential decay module to realize the adaptive update of the amplitude of the sinusoidal perturbation signal for extreme value search.

[0051] The DDPG algorithm integrator parameter adaptive optimization architecture building module is used to construct the extreme value search integrator parameter adaptive optimization module using the DDPG algorithm in deep reinforcement learning. The DDPG algorithm adopts the Actor-Critic dual network architecture and stabilizes the network training process through the experience replay mechanism and the soft update mechanism.

[0052] The DDPG algorithm training elements and interaction environment definition module is used to define the state space, action space and reward function required for DDPG algorithm training, and to use the anaerobic digestion process as the environment object for real-time interaction of the DDPG algorithm.

[0053] The DDPG algorithm dual-network parameter iterative update module is used to iteratively update the network parameters of the Actor network and Critic network in the DDPG algorithm based on real-time running data of the anaerobic digestion process.

[0054] The methane yield extremum search closed-loop optimization control module is used to perform extremum search and closed-loop optimization control on the methane yield of the anaerobic digestion process based on the adaptive integrator gain, the adaptive amplitude sinusoidal disturbance signal, and the extracted gradient information output by the updated DDPG algorithm.

[0055] As can be seen from the above technical solutions, this invention application has the following beneficial effects:

[0056] (1) The present invention designs an exponentially decaying sinusoidal perturbation amplitude structure that is linked with gradient information. When the gradient information is large in the early stage of optimization, the perturbation amplitude is maintained at a high level to ensure effective extraction of gradient information and optimization driving force. As the dilution rate approaches the optimal value, the gradient information decreases and the perturbation amplitude decays synchronously and adaptively. After the system enters steady state, the perturbation amplitude decays to a negligible level, which completely solves the industry pain point of continuous steady-state oscillation of dilution rate and methane yield caused by traditional fixed amplitude perturbation, avoids continuous loss of methane yield caused by steady-state oscillation, and ensures that the anaerobic digestion process operates stably at the maximum methane yield for a long time.

[0057] (2) This invention introduces the DDPG deep reinforcement learning algorithm to optimize the fixed-value integrator gain into an adjustable amount that can be updated online adaptively. By designing a state space, action space, and multi-objective reward function adapted to the anaerobic digestion process, the integrator gain is intelligently controlled in stages: in the early stage of optimization, a large gain is output to amplify gradient information and quickly drive the system to converge toward the optimal operating point; when approaching the optimal value, the gain is adaptively reduced to suppress overshoot; after steady state, a stable gain is maintained to ensure the ability to track the optimal operating point. Compared with the traditional unsteady-state oscillation extreme value search scheme, this invention shortens the convergence time by nearly 80 hours and can quickly respond to the optimal operating point shift caused by changes in operating conditions such as substrate concentration fluctuations and inhibitor accumulation, significantly improving the dynamic response speed and adaptability of the system.

[0058] (3) This invention combines adaptive perturbation amplitude adjustment with deep reinforcement learning adaptive parameter optimization. Without relying on a precise mechanistic model of the anaerobic digestion process, it simultaneously solves the core contradiction of steady-state oscillation and slow convergence in traditional extreme value search algorithms, achieving a simultaneous improvement in optimization convergence speed and steady-state control accuracy. Through the fluctuation penalty term in the reward function, it effectively suppresses drastic jumps in control parameters, avoiding the impact of large fluctuations in feed rate on the metabolic balance of anaerobic microbial communities, and ensuring the operational stability of the anaerobic digestion process. This solution can be directly integrated into the control system of existing anaerobic digestion reactors, adapting to various organic waste anaerobic digestion engineering scenarios, effectively improving the biomass energy conversion efficiency and operational economic benefits of the anaerobic digestion process. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Referring to the drawings will make the features and advantages of the present invention clearer. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0060] Figure 1 This is an overall flowchart of the methane yield extremum search method for anaerobic digestion based on reinforcement learning described in this invention;

[0061] Figure 2 This is a block diagram of the methane yield extremum search control method for the anaerobic digestion process based on reinforcement learning as described in this invention.

[0062] Figure 3 This is a flowchart illustrating the execution of the extreme value search algorithm in this invention.

[0063] Figure 4 This is a comparison chart of the dilution rate changes in the anaerobic digestion process of the present invention and existing technologies;

[0064] Figure 5 This is a comparison chart of methane yield changes in the anaerobic digestion process of the present invention and existing technologies;

[0065] Figure 6 This is an adaptive variation curve of the gain of the extreme value search integrator in this invention;

[0066] Figure 7 This is a structural block diagram of the methane yield extremum search system for anaerobic digestion based on reinforcement learning described in this invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] This invention discloses a method and system for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning. The core of this invention addresses the shortcomings of existing extreme value search techniques for methane yield in anaerobic digestion. The classic Extreme Value Search (ESC) algorithm suffers from steady-state oscillations caused by fixed-amplitude sinusoidal perturbations and insufficient steady-state accuracy. While the Unsteady Oscillation-Free Extreme Value Search (UESC) algorithm solves the steady-state oscillation problem, its fixed integrator gain cannot be adaptively adjusted, leading to a significant reduction in optimization convergence speed. This invention combines an exponentially decaying sinusoidal perturbation amplitude structure with DDPG deep reinforcement learning for adaptive optimization of the integrator gain. This approach simultaneously achieves high steady-state accuracy and fast convergence speed in searching the extreme value of methane yield in anaerobic digestion, comprehensively improving the operational efficiency and economic benefits of the anaerobic digestion process.

[0069] Example 1:

[0070] This embodiment provides a method for searching the extreme value of methane yield in an anaerobic digestion process based on reinforcement learning, such as... Figure 1 As shown, the method specifically includes the following steps:

[0071] In step S1, the control input variables and controlled output variables of the anaerobic digestion process are determined, and a search control framework for the extreme value of methane yield is constructed.

[0072] First, we need to define the core variables in the anaerobic digestion process:

[0073] Control input variable: dilution rate The bottom flow acceleration rate directly affects the substrate concentration in the reactor through the dilution rate, and is a core operational variable that can be controlled in real time in the anaerobic digestion process;

[0074] Controlled output variable: methane yield It is a core indicator for measuring the biomass energy conversion efficiency and economic benefits of anaerobic digestion.

[0075] In the extreme value search control process, in order to extract the gradient information of methane yield relative to dilution rate, it is necessary to input the dilution rate. The superimposed amplitude is , frequency is For a sinusoidal disturbance signal, the mapping relationship between the input and output variables is as follows:

[0076] ;

[0077] In the formula, This is a real-time estimate of the unknown optimal input dilution rate. Let be the nonlinear mapping function from dilution rate to methane yield in the anaerobic digestion process. Since it is a continuous-time variable, there is no need to build an accurate mechanistic model; the optimization of extreme values ​​can be achieved simply by using the input and output data.

[0078] Based on the above variable definitions, combined with Figure 2 The closed-loop control block diagram shown illustrates a control framework for methane yield extremum search. This framework uses the anaerobic digestion process as the controlled object, forming a complete extremum optimization control chain. The core of the framework includes five functional units, each connected to… Figure 2 The hardware modules and signal flows correspond one-to-one, as detailed below:

[0079] 1. Sine perturbation superposition unit: corresponding to Figure 2 sinusoidal signal ( Generation module, initial amplitude adjustment module The signal is combined with an end adder to generate a sinusoidal excitation signal of a set frequency. After adaptive amplitude adjustment, the signal is superimposed on the real-time estimate of the optimal dilution rate to provide a stable signal excitation for gradient information extraction.

[0080] 2. Filter bank gradient extraction unit: corresponding to Figure 2 High-pass filter, first multiplier, and low-pass filter; methane yield output from anaerobic digestion process. After the transfer function is After the high-pass filter removes the DC component, it is multiplied with the original sinusoidal disturbance signal, and then passed through a transfer function of... A low-pass filter is used for denoising, and finally the gradient information of methane yield relative to dilution rate is extracted. This provides core data for subsequent disturbance amplitude adjustment and optimal convergence control;

[0081] 3. Adaptive disturbance amplitude adjustment unit: corresponding to Figure 2 Medium exponential decay module ( The second multiplier is used to extract the resulting gradient information. With the input as the sole input, the output amplitude of the sinusoidal disturbance signal is adjusted in real time through a preset exponential decay structure, realizing the adaptive update of the disturbance amplitude throughout the optimization process, and fundamentally solving the problem of steady-state oscillation caused by fixed amplitude disturbances.

[0082] 4. Integrator gain adaptive optimization unit: corresponding to Figure 2 The DDPG algorithm network module uses the real-time input dilution rate of the anaerobic digestion process. methane yield To observe the state, a policy network trained offline and optimized online outputs an adaptively adjusted integrator gain in real time. This solves the core problems of slow convergence speed and poor adaptability to operating conditions caused by traditional fixed integrator gain;

[0083] 5. Closed-loop control execution unit: corresponding to Figure 2 The third multiplier, integrator, and final adder in the middle process transfer gradient information. With adaptive integrator gain After multiplication, the integrator calculates the real-time estimate of the optimal dilution rate. Then, an adaptive amplitude sinusoidal disturbance signal is superimposed to generate the final dilution rate control command. The output is sent to the anaerobic digester to achieve fully closed-loop optimization control for finding the extreme value of methane yield.

[0084] In step S2, based on the control framework, the gradient information of the output methane yield of the anaerobic digestion process is extracted through a filter bank, and the gradient information is input into a pre-established exponential decay module to achieve adaptive update of the amplitude of the sinusoidal perturbation signal for extreme value search.

[0085] The core of this step is to achieve adaptive adjustment of the amplitude of the sinusoidal disturbance signal, thereby solving the steady-state oscillation problem caused by fixed amplitude disturbances. The specific implementation process is as follows:

[0086] 1. Gradient Information Extraction: Real-time measurement of methane yield from the anaerobic digestion process, using a cutoff frequency of... The high-pass filter and the cutoff frequency are A filter bank composed of low-pass filters is used for filtering to remove the DC component from the methane yield signal, retaining the AC response with the same frequency as the sinusoidal perturbation signal, and finally extracting the gradient information of methane yield relative to dilution rate. The cutoff frequencies of the high-pass and low-pass filters are determined based on the frequency of the sinusoidal perturbation signal. Configure settings to ensure effective filtering of interference signals and accurate extraction of gradient information.

[0087] 2. Adaptive Update of Sinusoidal Disturbance Signal Amplitude: An exponential decay module is pre-established to construct an exponentially decaying sinusoidal disturbance signal amplitude structure. The specific expression is as follows:

[0088] ;

[0089] In the formula, The amplitude of the sinusoidal disturbance signal. The initial amplitude of the sinusoidal disturbance signal is given. This is a custom adjustment coefficient based on the characteristics of the anaerobic digestion process, used to control the decay rate of the disturbance signal amplitude. The methane yield gradient information extracted in the above steps.

[0090] The output value of the exponential decay module ranges from 0 to 1, and is related to the gradient information. The absolute values ​​are positively correlated, and its working characteristics are:

[0091] In the initial optimization phase, the dilution rate is far from the optimal operating point, and gradient information... With a large absolute value, the output of the attenuation stage approaches 1, the disturbance signal maintains a high amplitude, ensuring the effective extraction of gradient information and accelerating the optimization speed of the anaerobic digestion process.

[0092] As the dilution rate gradually approaches the optimal value, gradient information... The absolute value decreases synchronously, the output of the attenuation stage decreases synchronously, and the amplitude of the disturbance signal attenuates adaptively.

[0093] When the anaerobic digestion process reaches a steady state, and the dilution rate reaches near the optimal operating point, gradient information... When the value approaches 0, the output of the attenuation circuit approaches 0, and the amplitude of the disturbance signal is attenuated to a negligible level, thus eliminating the steady-state oscillation problem caused by fixed amplitude disturbances from the root and ensuring the steady-state operating accuracy of methane yield.

[0094] In step S3, the DDPG algorithm in deep reinforcement learning is used to construct an adaptive optimization module for the parameters of the extreme value search integrator. The DDPG algorithm adopts an Actor-Critic dual network architecture and stabilizes the network training process through an experience replay mechanism and a soft update mechanism.

[0095] This step addresses the core issue of slow convergence speed in exponentially decaying extremum search algorithms by introducing the Deep Deterministic Policy Gradient (DDPG) algorithm to construct an adaptive optimization module for integrator parameters. The specific architecture and mechanism are as follows:

[0096] The core of the DDPG algorithm includes an Actor policy network, a Critic value network, and an experience replay pool. It also includes a target Actor network and a target Critic network. The target network parameters are iteratively updated through a soft update mechanism to avoid oscillations and divergence during the training process.

[0097] Actor strategy network: used to adjust the real-time status of the anaerobic digestion environment. Output the value in this state. Biggest movement That is, to output the optimal integrator gain and achieve real-time optimization of the strategy;

[0098] Critic value network: used to analyze the state at the current moment. With action Perform a value assessment and output the corresponding value function. That is, to evaluate the merits of the current integrator gain selection, and to provide direction for the policy optimization of the Actor network;

[0099] Experience replay pool: Used to process real-time data generated by the interaction between the DDPG algorithm and the anaerobic digestion process. Stored in the format of, where This is the reward value at the current moment. This represents the environmental state for the next moment. During network training, samples are randomly selected from the experience replay pool for network training, breaking the temporal correlation between training samples and effectively improving the stability of network training.

[0100] Meanwhile, the DDPG algorithm updates the parameters of the target Actor network and the target Critic network through a soft update mechanism. Compared with the hard update mechanism, this can effectively avoid drastic changes in target values, further stabilize the network training process, and adapt to the large lag and strong nonlinearity of the anaerobic digestion process.

[0101] In step S4, the state space, action space, and reward function required for training the DDPG algorithm are defined, and the anaerobic digestion process is used as the environment object for real-time interaction of the DDPG algorithm.

[0102] This step defines the core interactive elements for the training and real-time operation of the DDPG algorithm, as follows:

[0103] 1. State Space Definition: The state space of the DDPG algorithm is the real-time input dilution rate of the anaerobic digestion process. With real-time output methane yield State vector ,in for dilution rate at time, for The methane yield at time t. This state space can comprehensively characterize the real-time operating conditions of the anaerobic digestion process, and both variables can be measured in real time by sensors, making it engineering feasible.

[0104] 2. Action Space Definition: The action space of the DDPG algorithm is the integrator gain of the extreme value search controller. Action vector ,in for The integrator gain at any given time. By using the real-time decision-making of the DDPG algorithm, the integrator gain is changed from a traditional fixed value to an adaptive adjustment, realizing online adaptive updating of the integrator gain and solving the problem that a fixed integrator gain cannot simultaneously achieve optimization speed and steady-state accuracy.

[0105] 3. Reward Function Definition: The reward function is the core guiding principle for DDPG algorithm strategy optimization. This invention divides the reward function into high-yield rewards. Production growth rate bonus Input fluctuation penalty The weighted sum of the three parts is obtained, and the specific expression is as follows:

[0106] ;

[0107] In the formula, , , These are the weighting coefficients for the three sub-rewards, which can be adjusted according to the actual operational needs of the anaerobic digestion process.

[0108] The specific expressions and physical meanings of each sub-reward item are as follows:

[0109] High production bonus :

[0110] ;

[0111] In the formula, , They are respectively time, methane yield at time t, for The dilution rate at any given time. This represents the increase in methane yield per unit dilution rate. When the increase in methane yield per unit dilution rate is higher, more rewards are given to guide the algorithm to quickly increase methane yield and accelerate strategy optimization.

[0112] Production growth rate bonus :

[0113] ;

[0114] In the formula, for methane yield at time t, The sampling interval is denoted as . The second derivative of methane yield is used to measure the trend of methane yield change. It can reflect changes in methane yield earlier and more sensitively, guide the algorithm to optimize in the direction of continuous increase in methane yield, and further accelerate the convergence speed of optimization.

[0115] Input fluctuation penalty :

[0116] ;

[0117] In the formula, , They are respectively time, Integrator gain at time step This is the penalty coefficient. This is a negative penalty term that penalizes drastic changes in the integrator gain, preventing large fluctuations in the control input, improving the overall stability of the control system, and ensuring the stable growth of the microbial community during the anaerobic digestion process.

[0118] 4. Interactive Environment Definition: The complete anaerobic digestion process is used as the environment object for real-time interaction of the DDPG algorithm. The integrator gain output by the DDPG algorithm acts on the extreme value search controller, the dilution rate output by the controller acts on the anaerobic digestion reactor, and the methane yield and dilution rate output by the reactor are fed back to the DDPG algorithm in real time, forming a complete closed-loop interaction of "environmental perception - action output - environmental feedback - reward calculation - strategy optimization".

[0119] In step S5, the network parameters of the Actor network and Critic network in the DDPG algorithm are iteratively updated based on the real-time running data of the anaerobic digestion process.

[0120] This step utilizes real-time interactive data from the anaerobic digestion process to iteratively update the parameters of the Actor and Critic networks, enabling continuous optimization of the strategy. The specific update rules are as follows:

[0121] 1. Actor network parameter update:

[0122] Network parameters of the Actor network The gradient ascent algorithm is used for iterative updates. By adjusting the parameters, the long-term cumulative reward of the anaerobic digestion process is continuously increased. The update formula is as follows:

[0123] ;

[0124] in, For the first In the next iteration, the updated Actor network policy parameters For the first In the next iteration, the policy parameters of the Actor network, The learning rate of the Actor network. For the objective function For strategy parameters The gradient of is expressed as:

[0125] ;

[0126] In the formula, For expectation operations, represent the state. Obedience strategy State distribution under , Output action pair parameters for the Actor network gradient, In the strategy parameters Under, state The corresponding action output; For the output of the Critic network Value-pair action The gradient indicates the ability to increase The direction of action, that is, the direction of action that can increase the methane yield in the anaerobic digestion process. In the state Next action And follow the strategy The expected cumulative reward that can be obtained This indicates the action during gradient calculation. The value is the action output by the current policy network.

[0127] 2. Critic network parameter update:

[0128] Critic network parameters By iteratively updating by minimizing the mean square Bellman error, the update formula is:

[0129] ;

[0130] in, For the first In the next iteration, the updated Critic network parameters For the first In the next iteration, the parameters of the Critic network... The learning rate of the Critic network. The mean square Bellman error loss function Critic network parameters gradient, Let Bellman's mean square error loss function be expressed as follows:

[0131] ;

[0132] In the formula, The expectation operator indicates that the average loss is calculated over a batch of sampled data. For parameters Under these conditions, the Critic network considers the state. and actions Output value, The target value is the Bellman objective, calculated using the target Critic network.

[0133] In step S6, based on the adaptive integrator gain, the sinusoidal perturbation signal with adaptive amplitude, and the extracted gradient information output by the updated DDPG algorithm, extreme value search and closed-loop optimization control are performed on the methane yield of the anaerobic digestion process.

[0134] This step is the final execution stage of the method. Combining the outputs of the aforementioned modules, it completes the optimization of the methane yield and closed-loop control, as follows: Figure 3 As shown, the specific steps include:

[0135] Step S61: Initialize the input dilution rate for the anaerobic digestion process and set the initial amplitude of the sinusoidal perturbation signal. ,frequency The initial values ​​of the filter bank cutoff frequency, network hyperparameters of the DDPG algorithm, and weight coefficients of the reward function;

[0136] Step S62: Superimpose an adaptive amplitude sinusoidal disturbance signal onto the current input dilution rate, and output the superimposed dilution rate as a control command to the feed metering pump of the anaerobic digester;

[0137] Step S63: Measure the methane yield from the anaerobic digestion process in real time, using a cutoff frequency of... The high-pass filter and the cutoff frequency are A filter bank composed of low-pass filters is used for filtering to remove the DC component, retain the AC response, and extract the gradient information of methane yield. ;

[0138] Step S64: The DDPG algorithm observes the input dilution rate in real time at the current moment. With output methane yield Based on the optimized network model, the output policy action is the integrator gain. This enables online adaptive updating of the integrator gain.

[0139] Step S65: Transfer gradient information With adaptive integrator gain After multiplication and integrator operation, the estimated value of dilution rate is updated, driving the dilution rate to converge to the optimal dilution rate corresponding to the optimal value of methane yield.

[0140] Step S66: Determine gradient information If the value approaches 0, proceed to step S62 to continue the search for the extreme value; if it does, it means that the dilution rate has reached the optimal operating point, and the current optimal dilution rate is continuously output to maintain the steady-state maximum value of the methane yield.

[0141] In the performance verification experiment of this embodiment, parallel comparative experiments were conducted using the reinforcement learning-based methane yield extremum search method for anaerobic digestion (DDPGUESC), the non-steady-state oscillatory anaerobic digestion extremum search algorithm (UESC), and the classical anaerobic digestion extremum search algorithm (ESC) proposed in this invention. The performance comparison results of the three algorithms are as follows:

[0142] Figure 4 The curves show a comparison of the dilution rate changes during anaerobic digestion under the three algorithms. The curves show that all three algorithms eventually converge the dilution rate to the optimal operating point, but the convergence speeds differ significantly. Among them, the DDPGUESC algorithm of this invention exhibits a significantly faster dilution rate optimization speed than the UESC and ESC algorithms; the dilution rate converges to the optimal value after 410 hours of system operation. The UESC algorithm has the slowest convergence speed; the dilution rate only converges to the same optimal value after running for 490 hours. The convergence time of the DDPGUESC algorithm of this invention is nearly 80 hours shorter than that of the UESC algorithm; the convergence speed of the ESC algorithm is between the two, but after the dilution rate converges to the optimal value, there is still obvious continuous small fluctuation, that is, steady-state oscillation phenomenon, which cannot achieve stable steady-state operation.

[0143] The above experimental results fully verify that the present invention achieves adaptive updating of the integrator gain for extreme value search through the DDPG algorithm, effectively solving the problem of slow convergence caused by the fixed integrator gain in the traditional extreme value search algorithm, and significantly improving the dynamic response speed and adaptability of the feeding optimization in the anaerobic digestion process.

[0144] Figure 5 The curves show a comparison of methane yield changes during anaerobic digestion under the three algorithms. As can be seen from the curves, during the optimization process, the peak methane yield under the DDPGUESC algorithm is... The peak methane yield under the UESC algorithm is The peak methane yield under the ESC algorithm is The steady-state values ​​of methane yield under the three algorithms are all It can be seen that while the DDPGUESC algorithm of this invention significantly improves the convergence speed, it also has a larger methane yield overshoot compared to the UESC algorithm due to the use of a larger integrator gain in the early stage of optimization. However, from the perspective of practical engineering applications, the overshoot is within an acceptable range after a comprehensive trade-off between the overshoot and the convergence speed, and the algorithm of this invention can fully meet the actual working conditions of the anaerobic digestion process for optimization convergence speed.

[0145] Figure 6 The gain of the extremum search integrator under the action of the DDPGUESC algorithm of this invention. The adaptive change curve shows that the integrator gain adaptive adjustment strategy designed in this invention exhibits staged control characteristics:

[0146] During the initial 0-100 hours of operation, the input dilution rate of the anaerobic digestion process is far from the optimal operating point, and the amplitude of the methane yield gradient is large. At this time, the DDPG network outputs a large integrator gain. The value is used to amplify gradient information, drive the dilution rate to quickly approach the optimal value, and achieve rapid optimization.

[0147] During the middle stage of operation (100-300 hours), as the input dilution rate gradually approaches the optimal dilution rate, methane yield gradient information is obtained. As the gain of the DDPG network gradually decreases, the integrator gain of the output is adaptively reduced. The value is adjusted to avoid excessive overshoot in methane yield due to excessive gain, while balancing optimization speed and operational stability.

[0148] After 300 hours of operation, the anaerobic digestion process gradually entered a steady-state operation phase, and the integrator gain... Ultimately, it stabilizes at around 0.53, which not only ensures the ability to track the optimal operating point of the dilution rate in real time and cope with the deviation of the optimal operating point caused by fluctuations in operating conditions, but also maintains the stable operation of the anaerobic digestion process under the condition of maximum methane yield.

[0149] Traditional extreme value search algorithms for finding the maximum methane yield in anaerobic digestion processes typically use a fixed integrator gain setting. This results in slow optimization speed and an inability to quickly respond to changes in operating conditions, such as substrate concentration fluctuations and inhibitor accumulation, which can cause deviations from the optimal operating point for methane yield. This invention optimizes and improves traditional extreme value search algorithms using the DDPG algorithm. By designing a multi-objective reward function to guide network training, the DDPG algorithm can output the integrator gain that maximizes the system's long-term cumulative reward in real time. This significantly improves the optimization speed for the maximum methane yield in anaerobic digestion processes while ensuring the system's steady-state operating accuracy.

[0150] Example 2:

[0151] This embodiment provides a reinforcement learning-based methane yield extremum search system for anaerobic digestion processes. The system is used to implement the reinforcement learning-based methane yield extremum search method for anaerobic digestion processes described in Embodiment 1. Figure 7 As shown, the system specifically includes the following six modules:

[0152] The anaerobic digestion process variable definition and extreme value search control framework construction module 100 is used to determine the control input variables and controlled output variables of the anaerobic digestion process and construct the methane yield extreme value search control framework.

[0153] The gradient information extraction and sinusoidal perturbation signal amplitude adaptive adjustment module 200 is used to extract the gradient information of the output methane yield of the anaerobic digestion process through a filter bank based on the control framework, and input the gradient information into a pre-established exponential decay module to realize the adaptive update of the amplitude of the sinusoidal perturbation signal for extreme value search.

[0154] The DDPG algorithm integrator parameter adaptive optimization architecture building module 300 is used to construct the extreme value search integrator parameter adaptive optimization module using the DDPG algorithm in deep reinforcement learning. The DDPG algorithm adopts the Actor-Critic dual network architecture and stabilizes the network training process through the experience replay mechanism and the soft update mechanism.

[0155] The DDPG algorithm training elements and interaction environment definition module 400 is used to define the state space, action space and reward function required for DDPG algorithm training, and to take the anaerobic digestion process as the environment object for real-time interaction of the DDPG algorithm.

[0156] The DDPG algorithm dual-network parameter iterative update module 500 is used to iteratively update the network parameters of the Actor network and Critic network in the DDPG algorithm based on real-time running data of the anaerobic digestion process.

[0157] The methane yield extremum search closed-loop optimization control module 600 is used to perform extremum search and closed-loop optimization control on the methane yield of the anaerobic digestion process based on the adaptive integrator gain, the adaptive amplitude sinusoidal disturbance signal, and the extracted gradient information output by the updated DDPG algorithm.

[0158] In this embodiment, the specific implementation methods of each module are consistent with the corresponding steps in Embodiment 1, and will not be repeated here. This system can be integrated into the programmable logic controller (PLC) or host computer control system of the anaerobic digester to realize fully automatic extreme value search and optimization control of methane yield in the anaerobic digestion process without manual intervention, and has strong engineering application value.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for searching for a maximum of a methane production rate of an anaerobic digestion process based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Determine the control input variables and controlled output variables of the anaerobic digestion process, and construct a search control framework for the extreme value of methane yield; Step S2: Based on the control framework, the gradient information of the output methane yield of the anaerobic digestion process is extracted through the filter bank, and the gradient information is input into the pre-established exponential decay module to realize the adaptive update of the amplitude of the sinusoidal perturbation signal for extreme value search. Step S3: Construct an adaptive optimization module for the parameters of the extreme value search integrator using the DDPG algorithm in deep reinforcement learning. The DDPG algorithm adopts an Actor-Critic dual network architecture and stabilizes the network training process through an experience replay mechanism and a soft update mechanism. Step S4: Define the state space, action space, and reward function required for training the DDPG algorithm, and use the anaerobic digestion process as the environment object for real-time interaction of the DDPG algorithm; Step S5: Based on the real-time operational data of the anaerobic digestion process, iteratively update the network parameters of the Actor network and Critic network in the DDPG algorithm; Step S6: Based on the adaptive integrator gain, the sinusoidal perturbation signal with adaptive amplitude output by the updated DDPG algorithm, and the extracted gradient information, perform extreme value search and closed-loop optimization control on the methane yield of the anaerobic digestion process.

2. The method for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning according to claim 1, characterized in that, In step S1, the control input variable for the anaerobic digestion process is the dilution rate. The bottom stream acceleration rate affects the matrix concentration in the reactor through the dilution rate; the controlled output variable is the methane yield. During the extreme value search control process, when the dilution rate is input... The superimposed amplitude is , frequency is For a sinusoidal disturbance signal, the mapping relationship between the input and output variables is as follows: ； In the formula, This is a real-time estimate of the unknown optimal input dilution rate. Let be the nonlinear mapping function from dilution rate to methane yield in the anaerobic digestion process. It is a continuous-time variable.

3. The reinforcement learning-based search for extreme methanogenic yield in an anaerobic digestion process method according to claim 1, characterized in that, In step S2, the pre-established sinusoidal disturbance signal amplitude structure of the exponential decay module is as follows: ； In the formula, The amplitude of the sinusoidal disturbance signal. The initial amplitude of the sinusoidal disturbance signal is given. This is a custom adjustment coefficient based on the characteristics of the anaerobic digestion process, used to control the decay rate of the disturbance signal amplitude. To extract the methane yield gradient information; the output value of the exponential decay module ranges from 0 to 1, and is related to the gradient information. The absolute value is positively correlated with the gradient information in the initial stage of optimization. When the disturbance signal is large, it maintains a high amplitude; when approaching the optimal value, the gradient information... Reduced, disturbance amplitude decays synchronously, gradient information in steady state Approaching 0, the disturbance amplitude decays to a negligible level.

4. The method for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning according to claim 1, characterized in that, In step S3, the DDPG algorithm includes an Actor policy network, a Critic value network, and an experience replay pool; The Actor policy network is used to adjust according to the real-time state of the environment. Output the action that maximizes value Q in this state. ; The Critic value network is used to analyze the current state. With action Perform a value assessment and output the corresponding value function. ; The experience replay pool is used to store interaction data... The data is stored in a specific format, and samples are randomly selected for network training. This is the reward value at the current moment. This represents the environmental state at the next moment.

5. The method for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning according to claim 1, characterized in that, In step S4, the state space required for training the DDPG algorithm is the real-time input dilution rate of the anaerobic digestion process. With real-time output methane yield The action space required for training the DDPG algorithm is the integrator gain of the extreme value search controller. The integrator gain is updated online adaptively using the DDPG algorithm.

6. The method for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning according to claim 5, characterized in that, In step S4, the reward function is obtained by a weighted sum of three parts: a high yield reward , a yield growth rate reward , and an input fluctuation penalty , and the specific expression is: ； in: ； ； ； In the formula, , , They are respectively time, time, methane yield at time t, for dilution rate at time, The sampling interval; , They are respectively time, Integrator gain at time step The penalty coefficient is... These are the weighting coefficients.

7. The method for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning according to claim 1, characterized in that, In step S5, the network parameters of the Actor network The gradient ascent algorithm is used for iterative updates. By adjusting the parameters, the long-term cumulative reward of the anaerobic digestion process is continuously increased. The update formula is as follows: ； in, For the first In the next iteration, the updated Actor network policy parameters For the first In the next iteration, the policy parameters of the Actor network, The learning rate of the Actor network. For the objective function For strategy parameters The gradient of is expressed as: ； In the formula, For expectation operations, represent the state. Obedience strategy State distribution under , Output action pair parameters for the Actor network gradient, In the strategy parameters Under, state The corresponding action output; For the output of the Critic network Value-pair action gradient, In the state Next action And follow the strategy The expected cumulative reward that can be obtained This indicates the action during gradient calculation. The value is the action output by the current policy network.

8. The method for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning according to claim 1, characterized in that, In step S5, the network parameters of the Critic network are... By iteratively updating by minimizing the mean square Bellman error, the update formula is: ； in, For the first In the next iteration, the updated Critic network parameters For the first In the next iteration, the parameters of the Critic network... The learning rate of the Critic network. The mean square Bellman error loss function Critic network parameters gradient, Let Bellman's mean square error loss function be expressed as follows: ； In the formula, The expectation operator indicates that the average loss is calculated over a batch of sampled data. For parameters Under these conditions, the Critic network considers the state. and actions Output value, This is the Bellman target value.

9. The method for searching the extreme value of methane yield in anaerobic digestion based on reinforcement learning according to claim 1, characterized in that, In step S6, the method for extreme value search and closed-loop optimization control of methane yield in the anaerobic digestion process based on the adaptive integrator gain, the sinusoidal perturbation signal with adaptive amplitude, and the extracted gradient information output by the updated DDPG algorithm specifically includes: Step S61: Initialize the input dilution rate for the anaerobic digestion process; Step S62: Superimpose a sinusoidal perturbation signal with adaptive amplitude onto the input dilution rate; Step S63: Measure the methane yield from the anaerobic digestion process in real time, using a cutoff frequency of... The high-pass filter and the cutoff frequency are A filter bank composed of low-pass filters is used for filtering to remove the DC component, retain the AC response, and extract the gradient information of methane yield. ; Step S64: The DDPG algorithm observes the input dilution rate in real time at the current moment. With output methane yield The output strategy action is the integrator gain. This enables online adaptive updating of the integrator gain. Step S65: Transfer gradient information With adaptive integrator gain Multiplication drives the dilution rate to converge toward the optimal value; Step S66: Determine gradient information If the value approaches 0, then proceed to step S62 to continue optimizing for the extreme value; if it does, then continue to output the current optimal dilution rate to maintain the steady-state maximum value of the methane yield.

10. A methane yield extremum search system based on reinforcement learning in an anaerobic digestion process, characterized in that, The system is used to implement the reinforcement learning-based methane yield extremum search method for anaerobic digestion as described in any one of claims 1 to 9, specifically including: The module for defining variables and constructing an extreme value search control framework for the anaerobic digestion process is used to determine the control input variables and controlled output variables of the anaerobic digestion process and to construct an extreme value search control framework for methane yield. The gradient information extraction and sinusoidal perturbation signal amplitude adaptive adjustment module is used to extract the gradient information of the output methane yield of the anaerobic digestion process through a filter bank based on the control framework, and input the gradient information into a pre-established exponential decay module to realize the adaptive update of the amplitude of the sinusoidal perturbation signal for extreme value search. The DDPG algorithm integrator parameter adaptive optimization architecture building module is used to construct the extreme value search integrator parameter adaptive optimization module using the DDPG algorithm in deep reinforcement learning. The DDPG algorithm adopts the Actor-Critic dual network architecture and stabilizes the network training process through the experience replay mechanism and the soft update mechanism. The DDPG algorithm training elements and interaction environment definition module is used to define the state space, action space and reward function required for DDPG algorithm training, and to take the anaerobic digestion process as the environment object for real-time interaction of the DDPG algorithm. The DDPG algorithm dual-network parameter iterative update module is used to iteratively update the network parameters of the Actor network and Critic network in the DDPG algorithm based on real-time running data of the anaerobic digestion process. The methane yield extremum search closed-loop optimization control module is used to perform extremum search and closed-loop optimization control on the methane yield of the anaerobic digestion process based on the adaptive integrator gain, the adaptive amplitude sinusoidal disturbance signal, and the extracted gradient information output by the updated DDPG algorithm.

Images