Combustor multi-target dynamic optimization control system based on reinforcement learning

Abstract

The invention discloses a combustor multi-target dynamic optimization control system based on reinforcement learning. The combustor multi-target dynamic optimization control system comprises a multi-modal acquisition module, a causal relationship generation module, a variable identification module and a layered reinforcement learning module. The multi-mode acquisition module is used for acquiring and fusing flame image sequence data and sound wave signal data from the optical sensor and the acoustic sensor to form a state representation vector; the causal relationship generation module performs conditional independence test on the state representation vector by using a PC algorithm, and identifies a key variable by analyzing a directed path pointing to a core target in a graph; and the identification module is used for calculating a dynamic safety boundary for the identified key variable. According to the method, a control system fusing multi-modal perception, causal reasoning and hierarchical reinforcement learning is constructed, conflicts among competitive targets can be coordinated, multi-target optimization balance can be maintained, and combustion oscillation and flameout risks are thoroughly eradicated through a dynamic safety boundary mechanism.

Description

Technical Field

[0001] This invention relates to the field of burner optimization technology, and more specifically to a multi-objective dynamic optimization control system for burners based on reinforcement learning. Background Technology

[0002] Large industrial burners play a crucial role in key equipment such as power plant boilers and industrial kilns. To meet the heat energy supply requirements of large-scale production, these burners typically employ a design combining multiple burner units in parallel. Each burner unit has an independent fuel supply system, air conditioning device, and flame monitoring and control components, which are arranged and installed according to a specific layout and spacing. During actual operation, multiple burner units work collaboratively, ensuring complete combustion of fuel and releasing stable and efficient heat energy by precisely controlling the fuel flow, air ratio, and ignition sequence of each unit. This collectively accomplishes complex combustion tasks and ensures the continuous and stable operation of industrial production processes. Therefore, a multi-objective dynamic optimization control system for burners is required.

[0003] For example, a control system for optimizing combustion of a burner, authorized by Chinese patent publication number CN 106247396 B, discloses an optimized control system that controls the air volume of the burner to ensure operation at a low excess air coefficient, thereby reducing flue gas heat loss while effectively improving fuel burnout rate, reducing the reducing atmosphere of the boiler, improving coking and corrosion phenomena, and achieving a safe, efficient, and low-pollution combustion process.

[0004] However, while most existing technologies can control the burner's airflow, the control objective is singular. When multiple control objectives conflict, only a single correlation adjustment can be made, resulting in low optimization efficiency. Furthermore, the combustion operating condition boundary is a fixed threshold; approaching the optimal operating condition boundary can easily lead to safety accidents such as combustion oscillation or flameout. Summary of the Invention

[0005] To address these issues, this invention provides a multi-objective dynamic optimization control system for burners based on reinforcement learning.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multi-objective dynamic optimization control system for burners based on reinforcement learning, including a multi-modal acquisition module, a causal relationship generation module, a variable identification module, and a hierarchical reinforcement learning module;

[0008] The multimodal acquisition module is used to acquire and fuse flame image sequence data and acoustic signal data from optical and acoustic sensors to form a state representation vector.

[0009] The causal relationship generation module uses the PC algorithm to test the conditional independence of the state representation vector, gradually constructs a directed acyclic graph of causal relationships, and identifies key variables by analyzing the directed paths pointing to the core target in the graph, and then calculates the average influence of the key variables on the target variable.

[0010] The identification module is used to calculate the dynamic safety boundary for the identified key variables in order to determine the upper and lower limit operating range of the dynamic safety boundary;

[0011] The hierarchical reinforcement learning module is constructed from an upper-layer meta-policy network and a lower-layer policy executor. It can output corrected action instructions based on state representation vectors and dynamic safety boundaries, and send them to the physical executor.

[0012] Furthermore, the multimodal acquisition module also includes a preprocessing unit and a fusion unit;

[0013] The preprocessing unit is built on image processing algorithms and fast Fourier transform algorithms. The image processing algorithm can calculate the flame area, aspect ratio, and brightness fluctuation characteristic parameters in the flame image sequence data in real time. The fast Fourier transform algorithm is used to convert the sound wave signal data to the frequency domain and extract the sound pressure level of the preset frequency band as the acoustic spectrum feature.

[0014] The fusion unit can assign a unified timestamp to the flame image sequence data and the acoustic signal data to ensure data alignment. It also employs a feature-weighted fusion algorithm to assign weights based on the known correlation between each feature parameter and the combustion state, and combine them into a state representation vector.

[0015] Furthermore, the specific content of the causal relationship generation module is as follows:

[0016] 1) Cause-effect graph construction based on conditional independence test

[0017] First, input the state representation vector to obtain the complete dataset. Then, use the PC algorithm to analyze the dataset based on the conditional independence test and calculate the independence statistic. The calculation formula is as follows:

[0018] in, For the monitored spectral characteristics, The preset expected frequency;

[0019] 2) Identify key variables using directed acyclic graphs of causal relationships.

[0020] After constructing the cause-effect graph, analyze all directed paths in the graph that point to the core objective, identify key variables with strong causal influence, and then calculate the average influence of the key variables on the target variable. The formula for calculating the average influence is as follows:

[0021] Where Y is the target variable, This is a preset fixed value for the key variable X under the first intervention condition. This is a pre-set fixed value for the key variable X under the second intervention condition. Let Y be the expected value of the target variable after the first intervention. This represents the expected value of the target variable Y after the first intervention.

[0022] Furthermore, the specific content of the identification module is as follows:

[0023] 1) First, query historical safe operation data, find historical data records with similar operating conditions under the current similar operating conditions, and extract data points where the target variable is within the preset safety range to form a set of safety values;

[0024] 2) Calculate the statistical quantiles of the safety value set to determine the operational range of the dynamic safety boundary; the calculation formula is as follows:

[0025]

[0026] in, For a set of safe values, This serves as a preset lower limit for the dynamic safety boundary. This is the preset upper limit for the dynamic safety boundary.

[0027] Furthermore, the upper-layer meta-policy network adopts a deep neural network architecture to receive state representation vectors. Simultaneously, a target value function is established within the deep neural network architecture to calculate the long-term expected value of each optimization objective. .

[0028] Furthermore, the long-term expected value The calculation formula is as follows:

[0029] in, For state representation vectors, Let K be the weight, and K be the target quantity. Let be the advantage value for the k-th objective.

[0030] Furthermore, the lower-level policy executor receives the dynamic safety boundary from the identification module and decodes the upper-level policy instructions into specific control actions, using the Lagrange multiplier method to transform the dynamic safety boundary into constraints for the optimization problem.

[0031] Furthermore, the hierarchical reinforcement learning module also has a causal inference unit, which can generate a directed acyclic graph provided by the causal relationship generation module to virtually intervene in the initial actions.

[0032] This invention has the following advantages: By constructing a control system that integrates multimodal perception, causal reasoning, and hierarchical reinforcement learning, it effectively solves the core problems of single objective and rigid safety boundaries in traditional combustion control. Specifically, the upper-level meta-policy network can coordinate conflicts between competing objectives; the lower-level policy executor, combining causal graph deduction and dynamic safety boundaries, decodes the policy into control actions that satisfy both multi-objective optimization and safety, thereby fundamentally avoiding safety accidents caused by approaching fixed thresholds and achieving multi-objective dynamic optimal control within the safety domain.

[0033] Meanwhile, by replacing traditional correlation adjustment with causal intervention and inference, the accuracy and efficiency of optimization decision-making are significantly improved. The causal generation module identifies key variables and dynamic boundaries, and the hierarchical learning module makes optimization decisions under safety constraints, enabling the system to maintain a multi-objective optimization balance. Furthermore, the dynamic safety boundary mechanism completely eliminates the risk of combustion oscillation and flameout, achieving a simultaneous improvement in control precision and operational safety.

[0034] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0035] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0036] Figure 1 This is a block diagram of the multi-objective dynamic optimization control system for burners based on reinforcement learning, as described in this invention. Detailed Implementation

[0037] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these embodiments are merely for further explanation of the present invention and should not be construed as limiting the scope of protection of the present invention. Technical engineers in the field can make some non-essential improvements and adjustments to the present invention based on the above-described content. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Please seeFigure 1 The multi-objective dynamic optimization control system for burners based on reinforcement learning includes a multi-modal acquisition module, a causal relationship generation module, a variable identification module, and a hierarchical reinforcement learning module.

[0039] The multimodal acquisition module is used to acquire and fuse flame image sequence data and sound wave signal data from optical and acoustic sensors to form a unified, high-dimensional state representation vector, providing comprehensive and accurate real-time operating condition information for the entire system.

[0040] Optical sensors (such as high-speed cameras) are pointed at the flame area to continuously capture a sequence of flame image data at a specific frame rate, thereby obtaining visual information about the flame morphology, brightness distribution, and flicker frequency. Acoustic sensors are installed on the inner wall of the combustion chamber to continuously collect broadband acoustic signal data generated by combustion noise and vibration.

[0041] The multimodal acquisition module also includes a preprocessing unit and a fusion unit. The preprocessing unit is built upon image processing algorithms and a fast Fourier transform algorithm. The image processing algorithm can calculate the flame area, aspect ratio, and brightness fluctuation characteristic parameters in the flame image sequence data in real time. Details are as follows:

[0042] 1) A binary image of the burned area is obtained by segmenting the unburned area or using a temperature threshold range. This image is then further refined through expansion and etching processes. Finally, the number of white dots in the binary image, representing the number of pixels in the burned area, is counted to calculate the flame area.

[0043] 2) Calculate the aspect ratio: After acquiring the flame image, determine its width and height pixel values, and divide the width by the height to get the flame's aspect ratio.

[0044] 3) Calculate brightness fluctuation: Calculate the average brightness of all pixels in the image or the RGB value of each pixel to determine the image brightness.

[0045] The Fast Fourier Transform algorithm is used to convert acoustic signal data to the frequency domain and extract the sound pressure level of a preset frequency band as acoustic spectral features.

[0046] The aforementioned image processing algorithms and fast Fourier transform algorithms refer to mathematical tools and computational methods used for feature extraction from flames and acoustic signals. Among them, image processing algorithms can also include edge detection, gray-level co-occurrence matrix analysis, etc., which are mainly used to extract morphological parameters (such as area and aspect ratio) and dynamic characteristics (such as brightness fluctuations) from flame image sequences. The fast Fourier transform algorithm can also be wavelet transform, which is mainly used to convert the acquired acoustic signals from the time domain to the frequency domain and extract the sound pressure level or energy distribution characteristics of specific frequency bands to achieve multi-dimensional monitoring and analysis of the combustion state.

[0047] The fusion unit can assign a unified timestamp to all flame image sequence data and acoustic signal data to ensure data alignment and that they reflect the same operating conditions at the same moment. Subsequently, a feature-weighted fusion algorithm is used to assign weights based on the known correlation between each feature parameter and the combustion state, combining all data into a unified, high-dimensional numerical system state representation vector.

[0048] The core principle of the feature-weighted fusion algorithm is to dynamically allocate weights based on the correlation strength between each feature and the target working condition, and generate a high-dimensional comprehensive representation vector through linear combination.

[0049] For example, if the influence of flame area on combustion stability is twice that of aspect ratio, then the weights should be assigned to 0.67 and 0.33 respectively.

[0050] The causal relationship generation module first uses the PC algorithm to test the conditional independence of the state representation vectors, gradually constructing a directed acyclic graph of causal relationships. By analyzing the directed paths pointing to the core objective in the graph, it identifies key variables with strong causal influence. Finally, it quantifies the degree of influence by calculating the average causal effect of the key variables, providing a quantitative basis for setting dynamic safety boundaries. This facilitates the identification of key variables and the quantification of their impact through causal discovery techniques, providing an interpretable basis for safety constraints on the system. The specific content of the causal relationship generation module is as follows:

[0051] 1) Cause-effect graph construction based on conditional independence test

[0052] First, the input state representation vector is used to obtain the complete dataset. The PC algorithm (causal discovery algorithm) is then used to analyze the dataset based on the conditional independence test, calculating the independence statistic. This statistic assesses whether two variables are independent given a specific dataset. If the conditional independence holds, the edge between the two variables is considered movable. The formula for calculating the independence statistic is as follows:

[0053] in, For the monitored spectral characteristics, The preset expected frequency,

[0054] By iteratively performing this test, the algorithm gradually eliminates redundant edges and determines the causal direction, ultimately outputting a directed acyclic graph that depicts the causal relationships between variables.

[0055] 2) Identify key variables using directed acyclic graphs of causal relationships.

[0056] After constructing the causal graph, all directed paths in the graph pointing to the core objective are analyzed to identify key variables with strong causal influence. The core principle is: if a variable can influence the target through more and more direct causal paths, and there are fewer mediating variables along the paths, then that variable is considered a key variable.

[0057] Next, the average impact value of the key variables on the target variable is calculated. This average impact value serves two purposes: first, it compares the influence of different key variables to determine their priority; second, it serves as an important input for subsequent calculations of the dynamic safety boundary. For example, if the ACE of a key variable indicates a significant positive impact on parameter generation, its safety boundary will be set more strictly to limit its fluctuation range, thus providing precise and quantifiable constraints for subsequent identification modules. The formula for calculating the average impact value is as follows:

[0058] Where Y is the target variable, This is a preset fixed value for the key variable X under the first intervention condition. This is a pre-set fixed value for the key variable X under the second intervention condition. Let Y be the expected value of the target variable after the first intervention. This represents the expected value of the target variable Y after the first intervention.

[0059] Preset fixed value of key variable X The calculation formula is as follows;

[0060] in, Let Y be the value of the target variable, and P be the coefficient.

[0061] The identification module calculates dynamic safety boundaries for identified critical variables to determine the upper and lower limits of the dynamic safety boundary's operating range, ensuring that the set boundaries always remain within the safe zone. It generates dynamic safety operating boundaries for critical variables that adapt to changes in operating conditions. The specific content of the identification module is as follows:

[0062] 1) First, query historical safe operation data to find historical data records with similar operating characteristics under current similar operating conditions (such as similar load and fuel type). From the filtered records, extract data points where the target variable is within the preset safety range, and further identify the values ​​of the key variables corresponding to the data points, thus forming a set of safe values ​​for the key variables under current similar operating conditions.

[0063] 2) Statistical quantile calculations are performed on the set of safety values ​​to determine the operational range of the dynamic safety boundary, ensuring that the operational boundaries set for key variables always remain within historically validated safety zones. The calculation formula is as follows:

[0064]

[0065] in, For a set of safe values, This serves as a preset lower limit for the dynamic safety boundary. This is the preset upper limit for the dynamic safety boundary.

[0066] The hierarchical reinforcement learning module consists of an upper-layer meta-policy network and a lower-layer policy executor. It outputs corrected action commands based on state representation vectors and dynamic safety boundaries, and distributes these commands to the physical actuators, achieving multi-objective dynamic optimization under safety constraints. While ensuring absolute system safety, it intelligently coordinates and resolves conflicts between multiple competing objectives during combustion, achieving dynamic optimization control.

[0067] The upper-level meta-policy network employs a deep neural network architecture to receive state representation vectors. Simultaneously, an objective value function is established within the deep neural network architecture to calculate the long-term expected value of each optimization objective. Long-term expected value The calculation formula is as follows:

[0068] in, For state representation vectors, Let K be the weight, and K be the target quantity. The advantage value for the k-th objective is calculated using the following formula:

[0069]

[0070] in, For preset action values, Let k be the action value function for target k.

[0071] The lower-level policy executor receives the dynamic safety boundary from the identification module and decodes the upper-level policy instructions into specific control actions. It uses the Lagrange multiplier method to transform the dynamic safety boundary into the constraints of the optimization problem and solves it through quadratic programming to ensure that the output action satisfies all safety constraints.

[0072] The dynamic safety boundary defines the upper and lower limits of allowed operations for each state vector, forming a multi-dimensional safety operation space. The core idea of ​​the constraint handling framework based on the Lagrange multiplier method is to cleverly integrate strict boundary constraints into an unconstrained optimization objective. Specifically, it creates a corresponding "Lagrange multiplier" for each safety boundary constraint. This multiplier is essentially a dynamically adjusted penalty coefficient. If an initial action approaches or exceeds a certain boundary, its corresponding multiplier value automatically increases, thus imposing a significant penalty on the out-of-bounds behavior in subsequent optimizations; conversely, if the action is far from the boundary, the multiplier value decreases to avoid unnecessary performance limitations.

[0073] Then, the lower-level policy executor uses the solver to iteratively calculate and find a modified action command that is closest to the original preliminary action plan but also fully satisfies all dynamic safety boundary constraints, and outputs it.

[0074] The hierarchical reinforcement learning module also includes a causal inference unit. This unit generates a directed acyclic graph based on causal relationships, virtually intervenes in initial actions, and infers the impact of these actions on various target variables along the causal path. This process quantifies the causal effects of actions, rather than merely statistical correlations, thereby avoiding erroneous decisions due to variable confounding and significantly improving optimization efficiency.

[0075] The aforementioned multi-objective dynamic optimization control system for burners, based on reinforcement learning, effectively addresses the core issues of single objective and rigid safety boundaries in traditional combustion control by constructing a control system that integrates multimodal perception, causal reasoning, and hierarchical reinforcement learning. Specifically, the upper-level meta-policy network coordinates conflicts between competing objectives; the lower-level policy actuator, combining causal graph deduction and dynamic safety boundaries, decodes the policy into control actions that satisfy both multi-objective optimization and safety, thereby fundamentally avoiding safety accidents caused by approaching fixed thresholds and achieving multi-objective dynamic optimal control within the safety domain.

[0076] Meanwhile, by replacing traditional correlation adjustment with causal intervention and inference, the accuracy and efficiency of optimization decision-making are significantly improved. The causal generation module identifies key variables and dynamic boundaries, and the hierarchical learning module makes optimization decisions under safety constraints, enabling the system to maintain a multi-objective optimization balance. Furthermore, the dynamic safety boundary mechanism completely eliminates the risk of combustion oscillation and flameout, achieving a simultaneous improvement in control precision and operational safety.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-objective dynamic optimization control system for burners based on reinforcement learning, characterized in that, It includes a multimodal acquisition module, a causal relationship generation module, a variable identification module, and a hierarchical reinforcement learning module; The multimodal acquisition module is used to acquire and fuse flame image sequence data and acoustic signal data from optical and acoustic sensors to form a state representation vector. The causal relationship generation module uses the PC algorithm to test the conditional independence of the state representation vector, gradually constructs a directed acyclic graph of causal relationships, and identifies key variables by analyzing the directed paths pointing to the core target in the graph, and then calculates the average influence value of the key variables on the target variable. The identification module is used to calculate the dynamic safety boundary for the identified key variables in order to determine the upper and lower limit operating range of the dynamic safety boundary; The hierarchical reinforcement learning module is constructed from an upper-layer meta-policy network and a lower-layer policy executor. It can output corrected action instructions based on state representation vectors and dynamic safety boundaries, and send them to the physical executor.

2. The multi-objective dynamic optimization control system for burners based on reinforcement learning according to claim 1, characterized in that, The multimodal acquisition module also includes a preprocessing unit and a fusion unit; The preprocessing unit is built on image processing algorithms and fast Fourier transform algorithms. The image processing algorithm can calculate the flame area, aspect ratio, and brightness fluctuation characteristic parameters in the flame image sequence data in real time. The fast Fourier transform algorithm is used to convert the sound wave signal data to the frequency domain and extract the sound pressure level of the preset frequency band as the acoustic spectrum feature. The fusion unit can assign a unified timestamp to the flame image sequence data and the acoustic signal data, and use a feature-weighted fusion algorithm to assign weights according to the known correlation between each feature parameter and the combustion state, and combine them into a state representation vector.

3. The multi-objective dynamic optimization control system for burners based on reinforcement learning according to claim 1, characterized in that, The specific content of the causal relationship generation module is as follows: 1) Cause-effect graph construction based on conditional independence test First, the input state representation vector is analyzed using the PC algorithm based on the conditional independence test, and the independence statistic is calculated; the calculation formula is as follows: ; in, For the monitored spectral characteristics, The preset expected frequency; 2) Identify key variables using directed acyclic graphs of causal relationships. After constructing the cause-effect graph, analyze all directed paths in the graph that point to the core objective, identify key variables with strong causal influence, and then calculate the average influence of the key variables on the target variable. The formula for calculating the average influence is as follows: ; Where Y is the target variable, This is a preset fixed value for the key variable X under the first intervention condition. This is a pre-set fixed value for the key variable X under the second intervention condition. Let Y be the expected value of the target variable after the first intervention. This represents the expected value of the target variable Y after the first intervention.

4. The multi-objective dynamic optimization control system for burners based on reinforcement learning according to claim 1, characterized in that, The specific contents of the identification module are as follows: 1) First, query historical safe operation data, find historical data records with similar operating conditions under the current similar operating conditions, and extract data points where the target variable is within the preset safety range to form a set of safety values; 2) Calculate the statistical quantiles of the safety value set to determine the operational range of the dynamic safety boundary; the calculation formula is as follows: ； in, For a set of safe values, This serves as a preset lower limit for the dynamic safety boundary. This is the preset upper limit for the dynamic safety boundary.

5. The multi-objective dynamic optimization control system for burners based on reinforcement learning according to claim 1, characterized in that, The upper-layer meta-policy network employs a deep neural network architecture to receive state representation vectors. Simultaneously, an objective value function is established within the deep neural network architecture to calculate the long-term expected value of each optimization objective. .

6. The multi-objective dynamic optimization control system for burners based on reinforcement learning according to claim 5, characterized in that, The long-term expected value The calculation formula is as follows: ; in, For state representation vectors, Let K be the weight, and K be the target quantity. Let be the advantage value for the k-th objective.

7. The multi-objective dynamic optimization control system for burners based on reinforcement learning according to claim 1, characterized in that, The lower-level policy executor receives the dynamic safety boundary from the identification module and decodes the upper-level policy instructions into specific control actions. It then uses the Lagrange multiplier method to transform the dynamic safety boundary into constraints for the optimization problem.

8. The multi-objective dynamic optimization control system for burners based on reinforcement learning according to claim 1, characterized in that, The hierarchical reinforcement learning module also has a causal inference unit, which can generate a directed acyclic graph provided by the causal relationship module to virtually intervene in the initial actions.

Images