Self-adaptive adjustment method and system for difunctional material carbon capture system based on deep Q learning

Abstract

The invention provides a self-adaptive adjustment method and system for a bifunctional material carbon capture system based on deep Q learning, and the method comprises the steps: S1, constructing a state vector of a current time step based on the real-time operation parameters of the bifunctional material carbon capture system at the current time step; s2, inputting the state vector of the current time step into an adjustment decision model constructed based on a deep Q network algorithm, and outputting an adjustment decision of the current time step; s3, adjusting the bifunctional material carbon capture system based on the adjustment decision of the current time step, calculating a composite reward, and constructing a state vector of the next time step; and S3, repeating the steps S1 to S3, after each time step is finished, constructing an experience sample by combining the state vector of the time step, the adjustment decision, the composite reward and the state vector of the next time step of the time step, putting the experience sample into an experience pool, training an adjustment decision model based on the experience pool, and carrying out iterative updating.

Description

Technical Field

[0001] This invention belongs to the field of carbon emissions, specifically relating to an adaptive adjustment method and system for a dual-functional material carbon capture system based on deep Q-learning. Background Technology

[0002] The stable operation of dual-functional materials (DFMs) systems is highly dependent on the precise control of core parameters such as inlet flow rate, reaction temperature, and hydrogenation rate in order to balance CO2 capture efficiency, target product selectivity, and material cycle life. However, existing adjustment algorithms are difficult to adapt to dynamic operating conditions in industrial scenarios, which restricts the large-scale application of the technology.

[0003] Currently, the adjustment technologies most relevant to this invention are mainly divided into two categories, both of which have core defects that fail to meet engineering requirements:

[0004] One approach is the proportional-integral-derivative (PID) control algorithm, which is currently the mainstream adjustment scheme for DFMs carbon capture systems. It uses preset fixed parameters to construct linear feedback based on the deviation between the outlet CO2 concentration and the set value, thereby regulating actuators such as the inlet valve and heating module. While this technology is logically simple and has a fast response, making it suitable for small-scale stable scenarios, it presents significant problems in industrial applications: First, its parameter adaptability is poor; fluctuations in flue gas impurities and temperature (±5%~±10%) can cause a sharp drop in carbon capture rate of 10%~15%, or even trigger material sintering. Second, its single control objective, focusing solely on the capture rate, can easily lead to excessive hydrogen addition or temperature exceeding limits, shortening material lifespan by 20%~30% and increasing operating costs.

[0005] The second approach is a static prediction algorithm based on a single neural network. This algorithm trains models such as BP neural networks and basic LSTMs using historical data to construct a mapping relationship between operating conditions and control commands. While this technology attempts to compensate for the shortcomings of PID control, bottlenecks remain: First, its generalization ability is weak, with prediction errors exceeding 15% when faced with untrained extreme operating conditions (such as high-concentration impurity impacts); second, it lacks closed-loop feedback, meaning that real-time changes such as a 30% decrease in material adsorption performance cannot trigger parameter corrections, leading to an increase in CO2 escape rate; and third, it suffers from an economic imbalance, failing to consider material losses and energy consumption, resulting in a 10% to 20% reduction in operational economy.

[0006] The root cause of these shortcomings lies in the fact that DFMs carbon capture systems are complex nonlinear systems with strong multi-dimensional coupling of "material properties, operating parameters, and system output." Neither the linear control logic of the PID algorithm nor the static prediction mode of a single neural network can construct a control mechanism of "real-time perception, dynamic decision-making, and closed-loop feedback." Specifically, the PID algorithm cannot cope with nonlinear disturbances in the system, and the single neural network lacks dynamic learning and multi-objective optimization capabilities. Neither solves the three core problems of adaptive control to operating condition fluctuations, multi-parameter collaborative control, and balancing economic performance. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies in regulating DFMs carbon capture systems, which are difficult to adapt to dynamic operating conditions in industrial scenarios and thus restrict the large-scale application of the technology, this invention provides an adaptive regulation method and system for dual-functional material carbon capture systems based on deep Q-learning (DQN). The method constructs a high-dimensional state vector covering the key operating states of the DFMs carbon capture system, and combines it with the DQN algorithm to build a regulation decision model. Based on the experience pool, the regulation decision model is iteratively trained to construct a closed-loop architecture of "state perception-action decision-feedback update". This architecture can dynamically learn the changing patterns of real-time operating states and output the comprehensive optimal regulation strategy to achieve adaptive regulation of the DFMs carbon capture system.

[0008] According to one aspect of the present invention, an adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning is provided, comprising:

[0009] S1. Based on the real-time operating parameters of the bifunctional material carbon capture system at the current time step, construct the state vector at the current time step;

[0010] S2. Input the state vector of the current time step into the adjustment decision model constructed based on the deep Q-network algorithm, and output the adjustment decision of the current time step; the adjustment decision model constructs a discretized action space based on the adjustable actions of the actuators in the dual-function material carbon capture system; based on the main network and target network of the deep Q-network algorithm, output the Q value of all actions in the action space under the state vector, and select actions as adjustment decisions by combining the ε-greedy strategy.

[0011] S3. Adjust the bifunctional material carbon capture system based on the adjustment decision at the current time step, and construct the state vector of the adjusted bifunctional material carbon capture system.

[0012] Repeat steps S1-S3. After each time step, calculate the composite reward. Combine the state vector, adjustment decision, composite reward, and adjusted state vector of that time step to construct an experience sample and put it into the experience pool. Train the adjustment decision model based on the experience pool and update it iteratively.

[0013] As a further technical solution, the types of real-time operating parameters in the bifunctional material carbon capture system include: inlet flow rate, reaction temperature, hydrogenation rate, inlet CO2 concentration, outlet CO2 concentration, material adsorption pressure, material cycle number, and target product selectivity.

[0014] As a further technical solution, the process of constructing the discretized action space includes:

[0015] The adjustable parameters of the actuators in the bifunctional material carbon capture system are obtained; the actuators include: an inlet valve, a heating device and a hydrogenation pump, and the corresponding adjustable parameters are: inlet flow rate, reaction temperature and hydrogenation rate, respectively.

[0016] Based on the actual adjustable commands of each actuator, the discretized adjustment actions of each adjustable parameter are obtained. Cartesian product matching is performed on the discretized adjustment actions of different adjustable parameters to obtain the discretized action space.

[0017] As a further technical solution, the main network and target network based on the deep Q-network algorithm are neural networks with the same structure, including an input layer, a feature extraction layer, a fully connected layer and an output layer. The number of neurons in the output layer is equal to the number of all actions in the action space. The input layer receives the state vector and finally outputs the Q value of all actions in the action space under the state vector in the output layer.

[0018] As a further technical solution, the steps for selecting the action at the current time step using the ε-greedy strategy include:

[0019] Obtain the Q-values ​​of all actions in the action space under the state vector of the current time step output by the main network, and randomly select a random number from [0, 1) at each time step. When the random number is greater than the exploration rate ε, select the action with the largest Q-value from the action space as the action of the current time step; otherwise, randomly select an action from the action space as the action of the current time step.

[0020] As a further technical solution, the composite reward comprehensively considers four key indicators—CO2 capture efficiency, target product selectivity, material cycle life, and operating energy consumption—to set reward values, and introduces penalty values ​​to constrain abnormal operating conditions. The function is expressed as:

[0021]

[0022] In the formula, R represents the composite reward value; R1, R2, R3 and R4 represent the CO2 capture efficiency reward value, target product selectivity reward value, material cycle life reward value and operating energy consumption reward value, respectively; P is the abnormal operating condition penalty value; and ω1, ω2, ω3, ω4 and ω5 are the weighting coefficients of each reward value and penalty value.

[0023] As a further technical solution, the process of iteratively training the adjustment decision model based on the experience pool includes:

[0024] The experience samples in the experience pool are accumulated step by step. The batch size is set. When the number of accumulated samples in the experience pool meets the condition that it is not less than the batch size, the training is iterated in a loop with one time step as one iteration round until the training terminates. During each training round, a batch of samples is randomly drawn from the experience pool. The main network is used to calculate the current Q value, the target network is used to calculate the target Q value, the loss value is calculated based on the current Q value and the target Q value, and the main network is updated based on the loss value.

[0025] Every fixed number of training epochs, the network parameters of the main network are synchronized to the target network, and the target network is updated.

[0026] As a further technical solution, the criteria for determining training termination are as follows:

[0027] At the end of each training round, if the inlet flow rate, hydrogenation rate, and material adsorption pressure in the state vector of the adjusted bifunctional material carbon capture system all return to zero, then the bifunctional material carbon capture system is judged to have stopped operating, the training is terminated, and the network parameters of the adjustment decision model are saved; otherwise, the iterative training process continues.

[0028] According to another aspect of this specification, an adaptive adjustment system for a bifunctional material carbon capture system based on deep Q-learning is provided, for implementing an adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning, comprising:

[0029] The state perception module is used to construct a state vector based on the real-time operating parameters of the dual-functional material carbon capture system.

[0030] The DQN core decision module is used to input the state vector constructed by the state perception module into the adjustment decision model based on the deep Q-network algorithm, and output the adjustment decision. The adjustment decision model constructs a discretized action space based on the adjustable actions of the actuators in the dual-function material carbon capture system. Based on the main network and target network of the deep Q-network algorithm, it outputs the Q value of all actions in the action space under the state vector, and selects actions as the adjustment decision by combining the ε-greedy strategy.

[0031] The execution control module is used to adjust the dual-function material carbon capture system according to the adjustment decision output by the DQN core decision module;

[0032] The feedback update module is used to calculate the compound reward for each adjustment.

[0033] As a further technical solution, the DQN core decision module has a self-updating function, and the self-updating process includes:

[0034] In each adjustment process, the DQN core decision module records the state vector and adjustment decision, combines the composite reward calculated by the feedback update module and the adjusted state vector obtained by the state perception module to construct an experience sample, and put it into the experience pool. Based on the experience pool, the iterative adjustment decision model is trained to achieve self-updating.

[0035] Compared with existing technologies, the advantages of this invention are as follows: By constructing a high-dimensional state vector covering the key operating states of the DFMs carbon capture system and combining it with the DQN algorithm to build a regulation decision model, the coupling characteristic relationship between each operating state can be accurately captured. Its nonlinear decision logic is adapted to the complex characteristics of the DFMs carbon capture system, avoiding the regulation inaccuracy problem of traditional linear algorithms. Based on the iterative training of the regulation decision model using an experience pool, a closed-loop architecture of "state perception-action decision-feedback update" is constructed, which can dynamically learn the changing patterns of real-time operating states, enhance dynamic adaptability, and improve the generalization performance of operating conditions. By customizing a composite reward that covers carbon capture efficiency, target product selectivity, material cycle life, operating energy consumption reward, and penalty value for constraining abnormal operating conditions, the optimal regulation strategy is guided to be output, realizing the adaptive regulation of the DFMs carbon capture system and solving the shortcomings of traditional methods in multi-objective balance and economy. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating an adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning, provided in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the structure of the deep Q-learning neural network in an embodiment of the present invention;

[0039] Figure 3 This is a flowchart illustrating an example of an adaptive adjustment of a dual-functional material carbon capture system based on deep Q-learning in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the adaptive adjustment system of a dual-functional material carbon capture system based on deep Q-learning, provided as an embodiment of the present invention. Detailed Implementation

[0041] It should be noted that:

[0042] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0045] like Figure 1 As shown, an adaptive adjustment method for a bifunctional material (DFMs) carbon capture system based on deep Q-learning (DQN) includes:

[0046] Step 1: Based on the real-time operating parameters of the bifunctional material carbon capture system at the current time step, construct the state vector for the current time step;

[0047] Step 2: Input the state vector of the current time step into the adjustment decision model constructed based on the deep Q-network algorithm, and output the adjustment decision of the current time step; wherein, the adjustment decision model constructs a discretized action space based on the adjustable actions of the actuators in the dual-function material carbon capture system; based on the main network and target network of the deep Q-network algorithm, it outputs the Q value of all actions in the action space under the state vector, and selects actions as the adjustment decision by combining the ε-greedy strategy;

[0048] Step 3: Adjust the bifunctional material carbon capture system based on the adjustment decision at the current time step, and construct the state vector of the adjusted bifunctional material carbon capture system;

[0049] Repeat steps 1-3. After each time step, calculate the composite reward, combine the state vector, adjustment decision, composite reward, and adjusted state vector of that time step to construct an experience sample, and put it into the experience pool. Iteratively train the adjustment decision model based on the experience pool.

[0050] In step 1, the types of real-time operating parameters of the bifunctional material carbon capture system include: inlet flow rate, reaction temperature, hydrogenation rate, inlet CO2 concentration, outlet CO2 concentration, material adsorption pressure, material cycle number, and target product selectivity.

[0051] Optionally, the inlet gas flow rate (unit: m³ / h) is collected in real time by a pipeline flow meter with a sampling frequency of 1Hz; the reaction temperature (unit: °C) is collected by a temperature sensor built into the material bed with an error of ±0.1 °C; the hydrogenation rate (unit: mol / min) is obtained by the flow controller of the hydrogenation pump; the inlet CO2 concentration (unit: vol%) is collected by an infrared gas analyzer at the inlet, and the outlet CO2 concentration (unit: vol%) is collected by an online gas monitoring system at the outlet, serving as the core characterization of the capture efficiency; the material adsorption pressure (unit: MPa) is collected by a pressure sensor in the reactor; the number of material cycles is automatically accumulated through the system operation log, used to indirectly characterize the decay state of the material adsorption performance; and the target product selectivity (unit: %) is calculated by real-time detection of product components using a gas chromatograph.

[0052] Furthermore, the state vector of the current decision cycle is represented by a high-dimensional continuous variable vector, whose dimension corresponds one-to-one with the types of perception parameters, mathematically expressed as:

[0053]

[0054] In the formula, q is the inlet flow rate; T is the reaction temperature; r is the hydrogenation rate; c_in is the inlet CO2 concentration; c_out is the outlet CO2 concentration; p is the material adsorption pressure; η is the material cycle number; and φ is the target product selectivity.

[0055] Preferably, to improve the model convergence speed, all parameters in the state vector are normalized and mapped to the [0,1] interval to eliminate the influence of dimensional differences. The normalization formula is:

[0056]

[0057] Where x is the original parameter value; x_norm is the normalized parameter value; x_min and x_max are the industrial operating limit values ​​of the parameter (determined by the material properties of DFMs and system design parameters).

[0058] In step 2, the process of constructing the discretized action space includes:

[0059] The adjustable parameters of the actuators in the bifunctional material carbon capture system are obtained; the actuators include: an inlet valve, a heating device and a hydrogenation pump, and the corresponding adjustable parameters are: inlet flow rate, reaction temperature and hydrogenation rate, respectively.

[0060] Based on the actual adjustable instructions of each actuator, the discrete adjustment actions of each adjustable parameter are obtained. Cartesian product matching is performed on the discrete adjustment actions of different adjustable parameters to obtain the discrete action space of the agent.

[0061] Specifically, the action corresponds to the adjustment command of the actuator, which acts on the direct output of the DFMs carbon capture system. The action space is defined as a discretized set of adjustment commands. The mathematical representation of the action is:

[0062]

[0063] In the formula, A represents the action of the intelligent agent; a_q represents the intake flow rate adjustment action; A_t represents the reaction temperature adjustment action; and a_r represents the hydrogenation rate adjustment action.

[0064] Optionally, the intake flow rate adjustment action corresponds to the opening adjustment of the intake valve, which is discrete into 5 levels: "-10%, -5%, 0, +5%, +10%", representing the adjustment range based on the current flow rate.

[0065] The reaction temperature adjustment action corresponds to the power adjustment of the heating module, which is discrete into 5 levels: "-3℃, -1.5℃, 0℃, +1.5℃, +3℃", matching the detection accuracy of the temperature sensor and the temperature tolerance range of the material.

[0066] The hydrogenation rate adjustment action corresponds to the speed adjustment of the hydrogenation pump, which is discrete into 5 levels: "-8%, -4%, 0%, +4%, +8%", which are determined in combination with the stoichiometric ratio of the target product.

[0067] The discretization design of the action space ensures fine-grained adjustment and reduces the decision complexity of the algorithm. At the same time, each action is subject to boundary constraints to prevent the system from becoming unstable due to excessive adjustment (e.g., the temperature adjustment does not exceed the sintering temperature of the DFMs material).

[0068] It should be added that the number of discrete levels and the specific parameters of each level of the above actuators can be adjusted according to the actual situation. This is only an example and is not a unique limitation.

[0069] In step 2, the main network and the target network based on the deep Q-network algorithm are neural networks with the same structure, including an input layer, a feature extraction layer, a fully connected layer, and an output layer. The number of neurons in the output layer is equal to the number of all actions in the action space. The input layer receives the state vector and finally outputs the Q-values ​​of all actions in the action space under the state vector in the output layer.

[0070] Specifically, both the main network and the target network are deep Q-learning neural networks, such as... Figure 2 As shown, the network structure adopts a hybrid "convolutional-fully connected" structure, consisting of an input layer, a feature extraction layer, a fully connected layer, and an output layer. The specific design is as follows:

[0071] Input layer: Receives a normalized 8-dimensional state vector, with 8 neurons;

[0072] Feature extraction layer: It contains one convolutional layer and one pooling layer. The convolutional layer uses a 3×1 convolutional kernel (because the state parameters are one-dimensional data with temporal correlation), and the number of output channels is 16. The activation function is ReLU. The pooling layer uses a 2×1 max pooling kernel to reduce dimensionality and retain key features.

[0073] Fully connected layer: contains two hidden layers. The first hidden layer has 64 neurons and the second hidden layer has 32 neurons. Both use the ReLU activation function to avoid the gradient vanishing problem.

[0074] Output layer: The number of neurons and the total number of actions in the action space (the combination of actions corresponding to the specific settings of the three adjustment parameters mentioned above, which is 5×5×5=125). The activation function is a linear activation function, and the output layer is the Q value (action value) of all actions in the action space.

[0075] Furthermore, the steps for selecting the action at the current time step using the ε-greedy strategy include:

[0076] Obtain the Q-values ​​of all actions in the action space under the state vector of the current time step output by the main network, and randomly select a random number from [0, 1) at each time step. When the random number is greater than the exploration rate ε, select the action with the largest Q-value from the action space as the action of the current time step; otherwise, randomly select an action from the action space as the action of the current time step.

[0077] The ε-greedy strategy ensures exploration by randomly selecting actions. Optionally, in this invention, the initial value of the exploration rate (ε) is set to 0.9, and it decreases linearly to 0.1 as the number of training steps increases (the decrease rate is 0.001 / step), balancing the exploration and utilization of the algorithm.

[0078] The reward function is the core of guiding DQN to achieve multi-objective optimization. In step 4, the composite reward comprehensively considers four key indicators: CO2 capture efficiency, target product selectivity, material cycle life, and operating energy consumption to set the reward value, and introduces a penalty value for constraining abnormal operating conditions. The function is expressed as:

[0079]

[0080] In the formula, R represents the composite reward value; R1, R2, R3 and R4 represent the CO2 capture efficiency reward value, target product selectivity reward value, material cycle life reward value and operating energy consumption reward value, respectively; P is the abnormal operating condition penalty value; and ω1, ω2, ω3, ω4 and ω5 are the weighting coefficients of each reward value and penalty value.

[0081] In the above formula, the weight coefficients are determined by the Analytic Hierarchy Process (AHP). Preferably, they can be dynamically adjusted according to the priority of actual industrial needs to ensure that the reward function can accurately reflect the overall operating performance of the system.

[0082] Specifically, the meaning, calculation method, and weight settings of each component in the composite reward function are as follows:

[0083] Firstly, the CO2 capture efficiency bonus (R1) is calculated based on the deviation between the outlet CO2 concentration and a set threshold. The calculation formula is as follows:

[0084]

[0085] In the formula, c_set is the set value of the outlet CO2 concentration. When c_out≤c_set, the value of R1 is (0,1], and when c_out>c_set, the value of R1 is negative. The weighting coefficient ω1 is set to 0.4 (because the collection efficiency is the core indicator).

[0086] Secondly, the target product selectivity bonus value (R2) is calculated based on the deviation between the target product selectivity and the ideal value. The calculation formula is as follows:

[0087] R2=φ / φ0

[0088] In the formula, φ represents the selectivity of the target product; φ0 represents the ideal value of the selectivity of the target product.

[0089] When φ≥0.9φ0, R2 takes the value ≥0.9; when φ<0.7φ0, R2 takes the value 0; and the weighting coefficient ω2 is set to 0.25 (to match the needs of resource utilization).

[0090] Third, the material cycle life bonus (R3) is characterized by the ratio of the reaction temperature fluctuation amplitude to the material tolerance fluctuation ΔT_max:

[0091] ΔT = |T - T_opt|

[0092] In the formula, ΔT represents the reaction temperature fluctuation range; ΔT_max is the material's tolerance to reaction temperature fluctuation range; and T_opt is the material's optimal reaction temperature.

[0093] When ΔT ≤ 0.05ΔT_max, R3 = 1; when ΔT ≥ 0.1ΔT_max, R3 = 0. The weighting coefficient ω3 is set to 0.2 (to extend material life and reduce operating costs).

[0094] Fourth, the operating energy consumption bonus value (R4) is calculated as the ratio of the energy consumption E per unit carbon capture of the DFMs carbon capture system to the baseline energy consumption E0. The calculation formula is as follows:

[0095] R4=E0 / E

[0096] In the formula, E represents the energy consumption per unit of carbon capture by the DFMs carbon capture system; E0 represents the baseline energy consumption per unit of the DFMs carbon capture system.

[0097] When E≤E0, R4≥1; when E>1.2E0, R4=0; the weighting coefficient ω4 is set to 0.15 (in response to economic needs).

[0098] Fifth, the abnormal operating condition penalty value (P) is used to constrain abnormal operating conditions where each parameter in the state vector exceeds its limit. For each parameter type in the state vector, a safety range is set according to the limit value under normal operating conditions. If any state parameter exceeds the safety range, P=10 is assigned; otherwise, P=0, and the weight ω5 is set to 5 (strengthening safety constraints).

[0099] For example, T_sinter is the sintering temperature of the material, and the corresponding reaction temperature T is set to be no less than the sintering temperature of the material. When T≥T_sinter, P=10 is assigned.

[0100] Step 4, the process of iteratively training the adjustment decision model based on the experience pool, includes:

[0101] The experience samples in the experience pool are accumulated step by step. The batch size is set. When the number of accumulated samples in the experience pool meets the condition that it is not less than the batch size, the training is iterated in a loop with one time step as one iteration round until the training terminates. During each training round, a batch of samples is randomly drawn from the experience pool. The main network is used to calculate the current Q value, the target network is used to calculate the target Q value, the loss value is calculated based on the current Q value and the target Q value, and the main network is updated based on the loss value.

[0102] Every fixed number of training epochs, the network parameters of the main network are synchronized to the target network, and the target network is updated.

[0103] When the experience pool reaches its maximum capacity, the earliest stored sample is overwritten.

[0104] The criteria for terminating training are as follows:

[0105] At the end of each training round, if the inlet flow rate, hydrogenation rate, and material adsorption pressure in the state vector of the adjusted bifunctional material carbon capture system all return to zero, it is determined that the bifunctional material carbon capture system has stopped operating (e.g., due to a system shutdown command or the material needing replacement), the training terminates, and the network parameters of the adjustment decision model are saved; otherwise, the iterative training process continues.

[0106] Specifically, the training strategy for adjusting the decision-making model employs an experience replay mechanism and a target network separation technique:

[0107] Experience replay mechanism: Construct an experience pool with a capacity of 10,000 to store experience samples (including the state vector, adjustment decision, composite reward, and state vector of the next time step of each time step) during the adaptive adjustment of the dual-function material carbon capture system. The samples are simplified to "state-action-reward-next state". The system iterates in a loop with one time step as one iteration until training terminates. During training, a batch of samples (batch size of 32) is randomly drawn from the experience pool to break the temporal correlation between samples and improve training stability.

[0108] Target network separation: The main network updates parameters in real time, while the target network is used to calculate the target Q-value. Its parameters are synchronized from the main network every 100 steps to avoid excessive fluctuations in Q-value estimation and accelerate algorithm convergence. The loss function uses mean squared error loss, i.e. Where Q_target is the target Q-value (calculated by the target network), Q_main is the current Q-value output by the main network, and the optimizer is the Adam optimizer [an adaptive optimization algorithm for updating neural network parameters, whose core function is to minimize the loss function (mean squared error loss in the documentation) by adjusting network weights (such as the parameters of convolutional and fully connected layers in the DQN main network), thus achieving algorithm training convergence]. The learning rate [the "step size" by which the optimizer updates neural network parameters] is set to 0.001.

[0109] like Figure 3 As shown, in a specific embodiment, the steps of applying an adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning to a DFMs carbon capture system include:

[0110] Step 0: System Initialization: Start the DFMs carbon capture system, activate an adaptive adjustment method for a dual-functional material carbon capture system based on deep Q-learning, and complete the following initialization operations:

[0111] ① Initialize the parameters of the main network and the target network in the regulation decision model constructed based on the deep Q-network algorithm (using the Xavier initialization method).

[0112] ② Clear the experience pool and set the training parameters (experience pool capacity, batch size, learning rate, etc.);

[0113] ③ Set the safety range of the status parameters, the set value of the outlet CO2 concentration c_set, the ideal value of the target product selectivity φ0, and other benchmark parameters.

[0114] Step 1: State Awareness and Input. Eight-dimensional state parameters, including intake flow rate q and reaction temperature T, are collected in real time at a frequency of 1Hz. The collected parameters are filtered and denoised (using a moving average filter with a window size of 5) to remove measurement noise in the industrial setting. The parameters are then normalized into a state vector S_t.

[0115] Step 2: Action Decision. The main network of the adjustment decision model performs forward computation on the input state vector S_t, outputting the Q-values ​​of all actions in the discretized action space. An ε-greedy strategy is used to select action A_t: when the random number is greater than ε (exploration rate), the action with the largest Q-value is selected as the adjustment decision; when the random number is less than or equal to ε, an action is randomly selected to ensure exploration. The initial value of ε is set to 0.9, and it linearly decays to 0.1 with the increase of training steps (decay rate of 0.001 / step), balancing the exploration and utilization of the algorithm.

[0116] Step 3: Action Execution and Status Feedback. Based on the adjustment decision, specific execution commands (such as intake valve opening adjustment signals and heating module power adjustment signals) are executed to drive the corresponding actuators within the DFMs carbon capture system to complete parameter adjustments. After adjustment, the status parameters of the adjusted DFMs carbon capture system are collected to obtain the adjusted vector S'_t.

[0117] Step 4: Reward Calculation and Experience Storage. Based on the adjustment results of Step 3, calculate indicators such as capture efficiency and product selectivity. Then, calculate the composite reward value R_t for this adjustment according to the composite reward function expression. Store the experience sample "S_t, A_t, R_t, S'_t" into the experience pool. If the experience pool is full, overwrite the earliest stored sample.

[0118] Step 5: DQN network training. When the number of samples in the experience pool reaches the batch size (32), start network training:

[0119] ① 32 samples were randomly selected from the experience pool;

[0120] ② Calculate the target Q value using the target network: ;

[0121] In the formula, γ is the discount factor, set to 0.9, representing the weight of future rewards; This indicates that the target network calculates the target Q-value; Indicates candidate actions; Represents the network parameters of the target network;

[0122] ③ Calculate the current Q value through the main network: ;

[0123] In the formula, This represents the current Q-value calculated by the main network; This represents the function for calculating the Q value. This represents the network parameters of the main network.

[0124] ④ Calculate the loss function and update the main network parameters through backpropagation;

[0125] ⑤ Synchronize the main network parameters to the target network every 100 steps.

[0126] Step 6: Termination Judgment. Determine whether the DFMs carbon capture system has stopped operating (e.g., system shutdown command or material replacement required). If it has not stopped, proceed to the next round of the "perception-decision-execution-feedback" loop; if it has stopped, save the DQN network parameters and the algorithm ends.

[0127] In the above steps, the system responds to fluctuations in operating conditions in real time through each step of the loop. At the same time, the network parameters are continuously optimized through online training, so that the output adjustment decision gradually adapts to the nonlinear characteristics of the system, thereby improving the control accuracy and stability.

[0128] In summary, the present invention provides the following technical advantages through the above embodiments:

[0129] 1. Overcoming the limitations of linear control and adapting to complex system characteristics—Existing PID algorithms rely on linear feedback logic, which cannot cope with the multi-dimensional intertwined nonlinear problems of "material-parameter-output" in dual-function material carbon capture systems. This invention constructs a high-dimensional state space covering the key operating states of the system, and combines it with the feature extraction capabilities of the DQN deep neural network to accurately capture the coupling relationships between various elements. Its nonlinear decision logic can naturally adapt to the complex characteristics of the system, avoiding the control inaccuracy problem of traditional linear algorithms.

[0130] 2. Enhanced Dynamic Adaptability and Improved Generalization Performance – Single neural network algorithms, due to their static prediction mode, are prone to deviations when faced with complex working conditions that have not been trained, and lack real-time correction mechanisms. This invention relies on a closed-loop architecture of "state perception - action decision - feedback update," which collects the adjusted system state in real time and updates the training samples, enabling the algorithm to dynamically learn the patterns of working condition changes. Even when encountering sudden fluctuations in working conditions, it can quickly adapt through online optimization, significantly improving the algorithm's generalization ability and response flexibility.

[0131] 3. Achieving multi-objective collaborative optimization to balance performance and economy—Traditional algorithms generally suffer from a single control objective, either focusing only on capture efficiency while neglecting material loss, or failing to consider operating energy consumption, resulting in poor economic efficiency. This invention, through the design of a customized composite reward function, incorporates core objectives such as carbon capture efficiency, product selectivity, material lifetime, and operating energy consumption into the optimization system, guiding the regulation decision model to output the comprehensive optimal regulation strategy, fundamentally solving the shortcomings of traditional algorithms in balancing multi-objectives and economy.

[0132] The key points and protection points of this invention are as follows:

[0133] 1. A DQN-driven closed-loop control architecture of "perception-decision-execution-feedback" enables dynamic adaptation;

[0134] 2. A high-dimensional state space encompassing the system's core parameters supports accurate state representation;

[0135] 3. Integrate multi-objective and security-constrained composite rewards to guide comprehensive optimal decision-making.

[0136] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide an adaptive adjustment system for a dual-functional material carbon capture system based on deep Q-learning. This system is used to execute an adaptive adjustment method for a dual-functional material carbon capture system based on deep Q-learning from the above method embodiments.

[0137] See Figure 4 The system includes:

[0138] The state perception module is used to construct a state vector based on the real-time operating parameters of the dual-functional material carbon capture system.

[0139] The DQN core decision module is used to input the state vector constructed by the state perception module into the adjustment decision model based on the deep Q-network algorithm, and output the adjustment decision. The adjustment decision model constructs a discretized action space based on the adjustable actions of the actuators in the dual-function material carbon capture system. Based on the main network and target network of the deep Q-network algorithm, it outputs the Q value of all actions in the action space under the state vector, and selects actions as the adjustment decision by combining the ε-greedy strategy.

[0140] The execution control module is used to adjust the dual-function material carbon capture system according to the adjustment decision output by the DQN core decision module;

[0141] The feedback update module is used to calculate the composite reward for each adjustment and feed it back to the DQN core decision module for iterative updates.

[0142] Specifically, the state perception module, as the "sensing end", is responsible for collecting multi-dimensional operating parameters of the DFMs carbon capture system in real time, constructing a state vector that reflects the real operating conditions of the DFMs carbon capture system, and providing data support for decision-making.

[0143] The DQN core decision module acts as the "brain," outputting actions based on the state information input from the state perception module and a regulatory decision model built on a deep Q-network algorithm, thereby achieving dynamic regulatory decisions under multiple objectives.

[0144] Execution control module: As the "execution end", it receives adjustment commands output by the DQN core decision module and drives the intake valve, heating module, hydrogen pump and other actuators to complete the adjustment of adjustable parameters;

[0145] Feedback Update Module: As the "learning end", it monitors the adjustment process in real time, calculates the compound reward of adjustment and feeds it back to the DQN core decision module, realizing the online update of network parameters and algorithm performance iteration of the adjustment decision model in the DQN core decision module.

[0146] This architecture, through a closed-loop design of "perception-decision-execution-feedback", can dynamically adapt to the fluctuations in working conditions in industrial scenarios. At the same time, through the deep learning capabilities of DQN, it can build a complex mapping relationship between multi-dimensional parameters and system output, thus solving the limitations of linear logic and static prediction in traditional algorithms.

[0147] The core decision module of DQN is the core of this invention. Its design is based on the fusion of traditional Q-learning and deep neural networks. Through the experience replay mechanism and the target network separation strategy, it improves the convergence and stability of the algorithm in complex nonlinear systems.

[0148] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning, characterized in that, include: S1. Based on the real-time operating parameters of the bifunctional material carbon capture system at the current time step, construct the state vector at the current time step; S2. Input the state vector of the current time step into the adjustment decision model constructed based on the deep Q-network algorithm, and output the adjustment decision of the current time step; the adjustment decision model is based on the adjustable action of the actuator in the dual-function material carbon capture system, and constructs a discretized action space. Based on the deep Q-network algorithm, the main network and target network output the Q-values ​​of all actions in the action space under the state vector, and combine the ε-greedy strategy to select actions as adjustment decisions; S3. Adjust the bifunctional material carbon capture system based on the adjustment decision at the current time step, and construct the state vector of the adjusted bifunctional material carbon capture system. Repeat steps S1-S3. After each time step, calculate the composite reward. Combine the state vector, adjustment decision, composite reward, and adjusted state vector of that time step to construct an experience sample and put it into the experience pool. Train the adjustment decision model based on the experience pool and update it iteratively.

2. The adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning as described in claim 1, characterized in that, In the dual-functional carbon capture system, the types of real-time operating parameters include: inlet flow rate, reaction temperature, hydrogenation rate, inlet CO2 concentration, outlet CO2 concentration, material adsorption pressure, material cycle number, and target product selectivity.

3. The adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning as described in claim 1, characterized in that, The process of constructing the discretized action space includes: The adjustable parameters of the actuators in the bifunctional material carbon capture system are obtained; the actuators include: an inlet valve, a heating device and a hydrogenation pump, and the corresponding adjustable parameters are: inlet flow rate, reaction temperature and hydrogenation rate, respectively. Based on the actual adjustable commands of each actuator, the discretized adjustment actions of each adjustable parameter are obtained. Cartesian product matching is performed on the discretized adjustment actions of different adjustable parameters to obtain the discretized action space.

4. The adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning as described in claim 1, characterized in that, The main network and target network based on the deep Q-network algorithm are neural networks with the same structure, including an input layer, a feature extraction layer, a fully connected layer, and an output layer. The number of neurons in the output layer is equal to the number of all actions in the action space. The input layer receives the state vector and finally outputs the Q-values ​​of all actions in the action space under the state vector in the output layer.

5. The adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning as described in claim 4, characterized in that, The steps for selecting the action at the current time step using the ε-greedy strategy include: Obtain the Q-values ​​of all actions in the action space under the state vector of the current time step output by the main network, and randomly select a random number from [0, 1) at each time step. When the random number is greater than the exploration rate ε, select the action with the largest Q-value from the action space as the action of the current time step; otherwise, randomly select an action from the action space as the action of the current time step.

6. The adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning as described in claim 1, characterized in that, The composite reward comprehensively considers four key indicators—CO2 capture efficiency, target product selectivity, material cycle life, and operating energy consumption—to set reward values, and introduces penalty values ​​for constraining abnormal operating conditions. The function is expressed as: ； In the formula, R represents the composite reward value; R1, R2, R3 and R4 represent the CO2 capture efficiency reward value, target product selectivity reward value, material cycle life reward value and operating energy consumption reward value, respectively; P is the abnormal operating condition penalty value; and ω1, ω2, ω3, ω4 and ω5 are the weighting coefficients of each reward value and penalty value.

7. The adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning as described in claim 1, characterized in that, The process of iteratively training the adjustment decision model based on the experience pool includes: The experience samples in the experience pool are accumulated step by step. The batch size is set. When the number of accumulated samples in the experience pool meets the condition that it is not less than the batch size, the training is iterated in a loop with one time step as one iteration round until the training terminates. During each training round, a batch of samples is randomly drawn from the experience pool. The main network is used to calculate the current Q value, the target network is used to calculate the target Q value, the loss value is calculated based on the current Q value and the target Q value, and the main network is updated based on the loss value. Every fixed number of training epochs, the network parameters of the main network are synchronized to the target network, and the target network is updated.

8. The adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning as described in claim 7, characterized in that, The criteria for terminating training are as follows: At the end of each training round, if the inlet flow rate, hydrogenation rate, and material adsorption pressure in the state vector of the adjusted bifunctional material carbon capture system all return to zero, then the bifunctional material carbon capture system is judged to have stopped operating, the training is terminated, and the network parameters of the adjustment decision model are saved; otherwise, the iterative training process continues.

9. An adaptive adjustment system for a bifunctional material carbon capture system based on deep Q-learning, used to implement the adaptive adjustment method for a bifunctional material carbon capture system based on deep Q-learning as described in any one of claims 1-8, characterized in that, include: The state perception module is used to construct a state vector based on the real-time operating parameters of the dual-functional material carbon capture system. The DQN core decision module is used to input the state vector constructed by the state perception module into the adjustment decision model constructed based on the deep Q-network algorithm, and output the adjustment decision; the adjustment decision model is based on the adjustable actions of the actuators in the dual-function material carbon capture system, and constructs a discretized action space. Based on the deep Q-network algorithm, the main network and target network output the Q-values ​​of all actions in the action space under the state vector, and combine the ε-greedy strategy to select actions as adjustment decisions; The execution control module is used to adjust the dual-function material carbon capture system according to the adjustment decision output by the DQN core decision module; The feedback update module is used to calculate the compound reward for each adjustment.

10. The adaptive adjustment system for a dual-functional material carbon capture system based on deep Q-learning as described in claim 9, characterized in that, The DQN core decision module has a self-updating function, and the self-updating process includes: In each adjustment process, the DQN core decision module records the state vector and adjustment decision, combines the composite reward calculated by the feedback update module and the adjusted state vector obtained by the state perception module to construct an experience sample, and put it into the experience pool. Based on the experience pool, the iterative adjustment decision model is trained to achieve self-updating.

Images