Multi-subgraph fusion reinforcement learning control method and system for multi-input multi-output system

Abstract

The embodiment of the invention provides a multi-subgraph fusion reinforcement learning control method and system for a multi-input multi-output system. The method comprises the following steps: extracting various variable data in a multi-input multi-output system mechanism model, constructing a global directed graph for structurally representing a variable data coupling relationship in the multi-input multi-output system based on the various variable data, and reconstructing the global directed graph into a plurality of mutually independent sub-graphs for representing deep features; and inputting the sub-graph into a reinforcement learning agent constructed by a strategy network and an evaluation network, in the strategy network, generating a control action paying attention to local dynamic and global coordinated optimization according to the sub-graph, and in the evaluation network, determining a multi-level causal fusion expression adaptive to fusion deep features through a multi-level causal path so as to obtain a value score. According to the embodiment of the invention, the interpretability and generalization ability of the strategy are improved, and the complex local and global coupling relationship among multiple variables in the multiple-input multiple-output system can be depicted more finely and more structurally.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligent control technology, and in particular to a multi-subgraph fusion reinforcement learning control method and system for multi-input multi-output systems, as well as a reinforcement learning agent training method and system for multi-subgraph fusion of multi-input multi-output systems. Background Technology

[0002] Multiple-input multiple-output (MIMO) control systems are widely found in complex industrial and engineering systems. Their typical characteristic is the simultaneous and significant interaction and coupling between multiple manipulated variables and multiple controlled variables. These systems typically involve multiple physical mechanisms such as mass transfer, energy conversion, and dynamic evolution, exhibiting strong nonlinearity, strong coupling, and parameter uncertainty. During operation, changes in a single manipulated variable often simultaneously affect multiple controlled variables, which in turn constrain each other through internal system mechanisms, making mutual interference between control loops unavoidable. In MIMO systems, the control variables and state variables are usually subject to conservation relationships or intrinsic constraints, resulting in multi-loop, multi-timescale dynamic characteristics. For example, adjusting an input variable not only directly affects the corresponding output but may also indirectly affect other output variables through internal coupling channels, triggering a chain reaction of changes in the overall operating state. This complex coupling structure leads to significant differences in the system's dynamic characteristics under different operating conditions, greatly increasing the control difficulty.

[0003] Currently, the actual operation of multiple-input multiple-output (MIMO) control systems still widely relies on traditional PID control or its improved cascade and feedforward-feedback composite control. However, these types of PID controllers often struggle to achieve ideal control performance under different operating conditions when facing strong nonlinearity and strongly coupled dynamics. They typically exhibit large dynamic process oscillations, long settling times, steady-state errors, and significant deterioration in control quality under large load changes or strong disturbances. With the advent of data-driven intelligent control methods, such as reinforcement learning (RL), the potential has emerged to autonomously learn optimal strategies through trial and error in complex systems without relying on precise models. Deep reinforcement learning can handle high-dimensional state and continuous action spaces, and is theoretically suitable for the control optimization of MIMO systems.

[0004] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology: Applying standard deep reinforcement learning algorithms directly to industrial control still faces several unresolved challenges: First, training requires massive amounts of interactive data, while exploratory interaction with real industrial systems is extremely costly and risky. Second, standard algorithms typically treat system states as flat feature vectors, making it difficult to effectively capture and utilize the complex topological relationships and causal dependencies between state variables determined by physical mechanisms, resulting in low learning efficiency and insufficient coordination of the learned policies. Third, the training process is unstable, policy convergence is difficult to guarantee, and the trained policies may have significant steady-state errors due to issues such as network output normalization and reward function design. Finally, "black box" decision models lack interpretability, making it difficult to gain the trust of operators in high-reliability power production scenarios. Even using Graph Reinforcement Learning (GRL), which combines Graph Neural Networks (GNNs) with reinforcement learning to improve reinforcement learning's ability to perceive system structure, the graph structures are mostly built based on statistical correlations of data, lacking deep embedding of the object's intrinsic physical mechanisms. The constructed graphs may not accurately reflect the real physical causal chains, limiting further improvements in control performance and the reliability of the method. Summary of the Invention

[0005] In order to at least address the problems existing in the aforementioned technologies.

[0006] In a first aspect, embodiments of the present invention provide a multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems, comprising: Extract various variable data from the mechanism model of a multiple-input multiple-output system, construct a global directed graph based on the various variable data to structurally represent the coupling relationship of variable data in the multiple-input multiple-output system, and reconstruct the global directed graph into multiple independent subgraphs representing deep features; The subgraph is input into a reinforcement learning agent constructed from a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled to obtain a multi-level causal path from the execution of the control action to the controlled output of the MIMO system. A multi-level causal fusion representation that adapts to and integrates deep features is determined through the multi-level causal path to obtain a value score. The value score is used to guide the training of the reinforcement learning agent, enabling the reinforcement learning agent to perform reinforcement learning control on the MIMO system.

[0007] Secondly, embodiments of the present invention also provide a reinforcement learning agent training method for multi-subgraph fusion in multiple-input multiple-output systems, comprising: Obtain the mechanism model of the multiple-input multiple-output system, construct a corresponding global directed graph as a graph representation based on the coupling relationship between the variable data in the mechanism model, and decompose the global directed graph into multiple independent subgraphs by the category of variable data for training reinforcement learning agents; The subgraph is input into a reinforcement learning agent constructed by a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled and a value score is determined. The policy gradient reward function is determined based on the error between the value score and the preset controlled state setting. The policy gradient reward function is used to constrain the value score of the control action corresponding to a single variable data to enhance the local control effect. The sum of the value scores of the control actions corresponding to all variable data is constrained by the additional reward to enhance the global control effect. The reinforcement learning agent is iteratively trained through the reward function until the error converges.

[0008] Thirdly, embodiments of the present invention provide a multi-subgraph fusion reinforcement learning control system for multiple-input multiple-output systems, comprising: The mechanism model and graph construction module is used to extract multiple variable data within the mechanism model of the multi-input multi-output system, construct a global directed graph that structurally represents the coupling relationship of variable data within the multi-input multi-output system based on the multiple variable data, and reconstruct the global directed graph into multiple independent subgraphs representing deep features; A reinforcement learning agent module is used to input the subgraph into a reinforcement learning agent constructed from a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled to obtain a multi-level causal path from the execution of the control action to the controlled output of the multi-input multi-output system. A multi-level causal fusion representation that adapts and integrates deep features is determined through the multi-level causal path to obtain a value score. The value score is used to guide the training of the reinforcement learning agent, enabling the reinforcement learning agent to perform reinforcement learning control on the multi-input multi-output system.

[0009] Fourthly, embodiments of the present invention provide a reinforcement learning agent training system for multi-subgraph fusion in multiple-input multiple-output systems, comprising: The mechanism model and graph construction module is used to obtain the mechanism model of the multiple-input multiple-output system, construct a corresponding global directed graph as a graph representation based on the coupling relationship between the variable data in the mechanism model, and decompose the global directed graph into multiple independent subgraphs by the category of variable data for the training of reinforcement learning agents. A reinforcement learning agent module is used to input the subgraph into a reinforcement learning agent constructed by a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled and a value score is determined. The steady-state error compensation module is used to receive the output of the policy network, calculate the deviation between the controlled variable and the preset target value in real time, generate a dynamic compensation signal, and use the dynamic compensation signal to compensate the output in order to continuously correct the steady-state deviation that the reinforcement learning policy has failed to completely eliminate. The reinforcement learning training module is used to determine the policy gradient reward function based on the error between the value score and the preset controlled state setting value. The policy gradient reward function is used to constrain the value score of the control action corresponding to a single variable data to enhance the local control effect. The additional reward is used to constrain the sum of the value scores of the control actions corresponding to all variable data to enhance the global control effect. The reinforcement learning agent is iteratively trained through the reward function until the error converges.

[0010] Fifthly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems and the reinforcement learning agent training method for multi-subgraph fusion of multi-input multi-output systems according to any embodiment of the present invention.

[0011] In a sixth aspect, embodiments of the present invention provide a storage medium storing a computer program, characterized in that, when the program is executed by a processor, it implements the steps of the multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems and the reinforcement learning agent training method for multi-subgraph fusion of multi-input multi-output systems according to any embodiment of the present invention.

[0012] In a seventh aspect, embodiments of the present invention provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems and the reinforcement learning agent training method for multi-subgraph fusion of multi-input multi-output systems according to any embodiment of the present invention.

[0013] The beneficial effects of this invention are as follows: By constructing a directed graph of the mechanism and embedding physical constraints into the graph structure, the reinforcement learning agent strictly follows the physical laws of the system while learning in a data-driven manner, thus improving the interpretability and generalization ability of the strategy; by adopting a multi-subgraph partitioning and fusion strategy, combined with hierarchical causal modeling, the complex local and global coupling relationships between multiple variables in a multi-input multi-output system can be characterized more precisely and structurally; the introduction of an integral compensation module creatively combines the advantages of traditional control theory and modern reinforcement learning, effectively eliminating the steady-state error that may exist in the reinforcement learning strategy and ensuring the high accuracy of the system in long-term operation. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems provided in an embodiment of the present invention; Figure 2 This is a flowchart of a DC boiler-turbine unit related to a multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the construction of a directed graph and the partitioning of multiple subgraphs in a multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems provided in an embodiment of the present invention. Figure 4 This is a flowchart of a reinforcement learning agent training method for multi-subgraph fusion in a multi-input multi-output system, provided by an embodiment of the present invention. Figure 5 This is a flowchart of the overall reinforcement learning control training process for a reinforcement learning agent training method for multi-subgraph fusion in a multi-input multi-output system, provided by an embodiment of the present invention. Figure 6 This is a training reward convergence curve of a reinforcement learning agent training method for multi-subgraph fusion in a multi-input multi-output system provided by an embodiment of the present invention during the training process. Figure 7 This is a schematic diagram illustrating the tracking control effect of the main steam pressure of a multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems provided in an embodiment of the present invention. Figure 8This is a schematic diagram illustrating the tracking control effect of intermediate point vapor enthalpy in a multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems provided in an embodiment of the present invention. Figure 9 This is a schematic diagram illustrating the tracking control effect of unit load using a multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems, provided in an embodiment of the present invention. Figure 10 This is a schematic diagram of the structure of a multi-subgraph fusion reinforcement learning control system for a multi-input multi-output system provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of a reinforcement learning agent training system for multi-subgraph fusion in a multi-input multi-output system according to an embodiment of the present invention; Figure 12 This is a schematic diagram of an embodiment of an electronic device for multi-subgraph fusion reinforcement learning control of a multiple-input multiple-output system, as provided in one embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] In the following embodiments, the multiple-input multiple-output system includes a flow boiler-turbine unit and a multi-stage continuous stirred tank, etc. Specifically, taking the OTBT (Once-Through Boiler-Turbine Unit) system as an example, the reinforcement learning optimization controller training method and system of the multiple-input multiple-output system proposed in this invention will be described in detail. However, this invention is not limited to this application scenario. Similarly, it can also be applied to scenarios such as multi-stage continuous stirred tanks.

[0018] like Figure 1 The diagram shown is a flowchart of a multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems according to an embodiment of the present invention, including the following steps: S11: Extract multiple variable data from the mechanism model of the multiple input multiple output system, construct a global directed graph that structurally represents the coupling relationship of variable data in the multiple input multiple output system based on the multiple variable data, and reconstruct the global directed graph into multiple independent subgraphs representing deep features; S12: The subgraph is input to a reinforcement learning agent constructed from a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled to obtain a multi-level causal path from the execution of the control action to the controlled output of the MIMO system. A multi-level causal fusion representation that adapts to and integrates deep features is determined through the multi-level causal path to obtain a value score. The value score is used to guide the training of the reinforcement learning agent, enabling the reinforcement learning agent to perform reinforcement learning control on the MIMO system.

[0019] In this embodiment, the method constructs a mechanism-driven directed graph model to structurally represent the coupling relationships between system variables. The mechanism model is a mathematical model built based on a multi-input multi-output system, predicting behavior by describing the system's inherent mechanisms (such as control variables, intermediate state variables, and controlled variables). A multi-subgraph fusion reinforcement learning architecture and integral compensation mechanism are designed for training, thereby achieving more accurate, robust, and adaptive intelligent control of the boiler-turbine unit system.

[0020] For step S11, multiple variable data are extracted from the existing multiple-input multiple-output system mechanism model. As an implementation method, the categories of the multiple variable data include: control variables, intermediate state variables and controlled variables, and there is a clear directional relationship between the variable data.

[0021] This approach constructs a global directed graph representing the mechanism of a multi-input multi-output (MIMO) control system. As one implementation method, the various variable data are used as nodes, and the dynamic causal relationships between these variables are used as directed edges. This constructs a structured global directed graph that represents the coupling relationships between different variable data categories within the MIMO system. A directed graph is a graphical model that uses directed edges to represent causal or dependency relationships between variables. It explicitly indicates the directional relationships between variables through directed edges (arrows), thereby describing the inherent working methods and interaction rules of each element in the MIMO control system structure.

[0022] Specifically, for multiple-input multiple-output systems, this method uses a once-through boiler-turbine unit as an example for explanation. The process flow of the once-through boiler-turbine unit is as follows: Figure 2 As shown, fuel (such as coal in the diagram) is burned in the boiler furnace to produce high-temperature flue gas. As this flue gas flows through the furnace, it transfers heat to the water-cooled walls, superheaters, reheaters, and other heating surfaces, gradually heating the water into high-temperature, high-pressure steam.

[0023] Economizer stage: Feedwater first enters the economizer for preheating, and then enters the water-cooled wall.

[0024] Water-cooled wall stage: Water absorbs heat in the water-cooled wall, forming a water-air mixture.

[0025] Separation and superheating: The steam-water mixture is separated by a separator, and the steam enters the superheater for further heating into superheated steam.

[0026] Reheater stage: After the exhaust steam from the high-pressure cylinder enters the reheater for heating, it returns to the intermediate-pressure cylinder to continue doing work.

[0027] Superheated steam enters the high-pressure cylinder of the turbine, expands, and performs work, driving the rotor to rotate. The reheated steam enters the intermediate-pressure and low-pressure cylinders to continue performing work, ultimately driving the generator to produce electricity.

[0028] Extract the variable data from the above steps, including: control variables (e.g., coal feed rate, water feed rate, turbine valve opening), intermediate state variables (e.g., furnace pulverized coal flow rate, separator steam pressure, etc.), controlled variables (e.g., main steam pressure, intermediate point steam enthalpy, unit load), and the dynamic causal relationships between setpoints (e.g., as the coal feed rate increases, the furnace pulverized coal flow rate also increases). Based on the above data, construct a global mechanism directed graph, where nodes represent the aforementioned variable data and directed edges represent influence paths.

[0029] Specifically, taking a 600MW supercritical once-through boiler-turbine generator unit as an example, the key variables of the system are determined based on the principles of conservation of mass, energy, and momentum. These include, as the manipulated variable, the coal feed rate. Water supply and turbine valve opening Unit load as a controlled variable Main steam pressure and the enthalpy of vapor at the midpoint The dynamic characteristics of a multiple-input multiple-output system are described by the following state matrix equation (1): (1) in, , , , For the state matrix of a multiple-input multiple-output system, = , = , The intermediate state variables of the system are the pulverized coal flow rate in the furnace. Separator steam pressure Enthalpy of vapor at midpoint .

[0030] A directed graph is constructed based on the dynamic causal relationships between these variables. Nodes in the directed graph represent the variables and their set values, while directed edges represent the direct influence relationships between variables, such as a path from the coal feed rate node to the pulverized coal flow rate node, or from the pulverized coal flow rate node to the separator steam pressure node. Its topological relationships are represented by an adjacency matrix. This indicates that the matrix elements are represented as follows: This indicates the existence of slave nodes. To the node A directed edge is defined if it is a given value, otherwise it is 0. The adjacency matrix representing the topological relationships in a directed graph is as follows: Figure 3 Displayed from left to center.

[0031] Similarly, directed graphs of the global mechanism of other types of multi-input multi-output control systems can be constructed, such as multi-stage continuous stirred tank reactors. Simply put, this involves reactants entering at a constant flow rate from the first stage, with products simultaneously exiting, maintaining stable volumes at each stage. A stirrer ensures rapid mixing of materials, resulting in uniform concentration and temperature within each stage. The materials then sequentially pass through multiple reactors, with each stage's reaction conditions (such as temperature and concentration) independently controlled. This approach is suitable for complex reactions or high-conversion requirements, and the multi-stage design reduces backmixing, avoids side reactions, and improves selectivity. The variable data from the above steps are extracted, including: control variables (reactants, rotor speed), intermediate state variables (reactant state), controlled variables (temperature, concentration, liquid level, pressure), and the corresponding established dynamic causal relationships. Based on this data, a directed graph of the mechanism is constructed, where nodes represent the variable data and directed edges represent influence paths, which will not be elaborated further here.

[0032] Furthermore, based on the degree of physical coupling and functional relevance, the global directed graph is divided into several internally tightly connected subgraphs, forming a multi-subgraph structure, such as... Figure 3 The diagram from center to right shows the process. This step transforms the complex, coupled network, which is difficult to handle, into a series of sub-problems with clear structures and well-defined physical meanings. Based on the sparsity of the adjacency matrix and the physical coupling relationships between nodes, this method divides the original mechanism graph into several independent, tightly connected subgraphs according to their coupling degree. This division preserves the physical relationships between variables and provides structured input for subsequent feature extraction. Thus, in subsequent steps, by constructing a directed mechanism graph and embedding physical constraints into the graph structure, the reinforcement learning agent can strictly adhere to the physical laws of the system while learning data-driven processes, laying the groundwork for improved policy interpretability and generalization ability.

[0033] In step S12, the complex coupled network, such as the multi-input multi-output system, which is difficult to handle, is transformed into a series of sub-problems (multiple subgraphs of a directed graph) with clear structure and explicit physical meaning, as obtained in step S11. At this point, a multi-subgraph fusion policy-evaluation network is constructed based on the multi-subgraphs of the directed graph. This method designs a novel reinforcement learning agent architecture, including a policy network and an evaluation network, both of which are deeply integrated with the multi-subgraph structure determined in step S11. Simply put, the policy network is responsible for "generating control actions," and the value network is responsible for "evaluating the quality of control actions." Working together, they enable the agent to make better decisions in complex environments.

[0034] As one implementation, the control actions generated by the policy network based on the plurality of independent subgraphs, focusing on local dynamics and global coordination optimization, include: The variable data corresponding to multiple independent subgraphs are split into multiple subsets according to categories; Independent graph attention networks are used to extract features from multiple subgraphs rich in local coupling relationships within the subsets; By mapping the fused features of the multiple subgraphs using a multilayer perceptron, control actions are generated to control the control variables within the multiple-input multiple-output system.

[0035] In this implementation, the policy network is responsible for generating control actions based on the current system state. The core of this approach is as follows: First, the enhanced state vector, containing the controlled variable, intermediate states, and setpoints, is split into multiple subsets according to variable categories. Then, for each subset, a separate graph attention network branch is used to extract features rich in local coupling relationships from its corresponding subgraph. Finally, the features from all subgraphs are fused and mapped using a multilayer perceptron to generate the final control action. This method allows the policy network to simultaneously focus on local dynamic details and global coordination relationships.

[0036] Specifically, the policy network reception status As input, where For controlled output, This is an intermediate state. This is a set value. The policy network first decomposes the input state into feature subsets corresponding to the three subgraphs mentioned above. Each feature subset is input into an independent graph attention network branch. Each branch uses its corresponding subgraph adjacency matrix as a topological constraint and performs multi-head attention computation to aggregate neighbor node information. Specifically, for each attention head... ,node In the Layer characteristics Update using the following formula (2): (2) in, Represents a node The neighborhood group, The normalized attention coefficient, For learnable weight matrix, To focus on the number of heads, This represents a vector concatenation operation, used to combine the vectors of multiple attention heads to obtain a higher-dimensional feature. ELU is the exponential linear unit activation function. Each GAT (Graph Attention Network) dynamically calculates the importance between nodes through attention coefficients. The node features output by the branches are then average-pooled to obtain the subgraph-level feature vector. , , These feature vectors are concatenated and mapped through a multilayer perceptron to ultimately output the normalized original control action. .

[0037] As one implementation method, in the evaluation network, the control action is modeled to obtain a multi-level causal path from the execution of the control action by the multiple-input multiple-output system to the controlled output. A multi-level causal fusion representation that adapts and integrates deep features is determined through the multi-level causal path to obtain a value score, including: The evaluation network models the control action, and determines the indirect influence of the control variable on the intermediate state variable based on the control action through hierarchical causal modeling, as well as the deep internal coupling influence from the intermediate state variable to the controlled variable. Based on the indirect influence and the deep influence, a multi-level causal path is determined. The information of the multi-level causal path is superimposed and fused to obtain a multi-level causal fusion representation; The deep features of the multi-level causal fusion representation are extracted using a graph attention network, and the value score of the deep features is determined by a multilayer perceptron.

[0038] In this implementation, the evaluation network is responsible for assessing the quality of the control actions generated by the policy network. Its core lies in a hierarchical causal modeling module. This module does not simply treat all variables as equal nodes, but explicitly models the control actions. To the controlled output The multi-level causal path between them includes direct influence, through intermediate states, etc. The indirect effects and the deep effects through internal coupling in intermediate states are also considered. After these multi-level causal information are fused, they are combined with graph attention networks for feature extraction, thereby providing the policy network with more accurate value assessment signals that better reflect the system's internal physical logic.

[0039] The evaluation network constructed in the above manner is used to assess the value of state-action pairs. This network introduces multi-level causal influence modeling: based on the global adjacency matrix. Calculate the first-order direct effect of the control action on the controlled output. Second-order indirect influence and third-order depth effects These three levels of influence are defined by calculation formula (3) as follows: (3) Taking into account the cumulative effect of multi-level paths, this method fuses the multi-level transmission effects through superposition, resulting in the following formula (4) multi-level causal fusion representation: (4) in, This is a matrix representing the hierarchical causal superposition effect.

[0040] Current action and The input is fed into a GAT module, which uses the causal strength information contained in the matrix to guide attention calculation and extract the structured feature representation of the action. .Will Compared with the current state The data is then concatenated and output through another multilayer perceptron, where the Q-value is a value score for generating the control action. This value score guides the training of the reinforcement learning agent, enabling it to perform reinforcement learning control of the MIMO system. More specifically, based on this score, the parameters of the policy network are updated, thereby allowing the policy network to provide more valuable control signals.

[0041] As can be seen from this implementation, this method, by constructing a directed graph of the mechanism and embedding physical constraints into the graph structure, enables the reinforcement learning agent to strictly follow the physical laws of the system while learning in a data-driven manner, thereby improving the interpretability and generalization ability of the strategy. Simultaneously, by employing a multi-subgraph partitioning and fusion strategy, combined with hierarchical causal modeling, it can more precisely and structurally characterize the complex local and global coupling relationships among multiple variables in the boiler-turbine unit system, achieving high-performance control.

[0042] like Figure 4 The diagram shows a flowchart of a reinforcement learning agent training method for multi-subgraph fusion in a multiple-input multiple-output system according to an embodiment of the present invention, including the following steps: S21: Obtain the mechanism model of the multiple-input multiple-output system, construct a corresponding global directed graph as a graph representation based on the coupling relationship between the variable data in the mechanism model, and decompose the global directed graph into multiple independent subgraphs by the category of variable data for training reinforcement learning agents. S22: The subgraph is input into a reinforcement learning agent constructed by a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled and a value score is determined. S23: Determine the policy gradient reward function based on the error between the value score and the preset controlled state setting value. Use the policy gradient reward function to constrain the value score of the control action corresponding to a single variable data to enhance the local control effect. Use additional rewards to constrain the sum of the value scores of the control actions corresponding to all variable data to enhance the global control effect. Iterate the reinforcement learning agent through the reward function until the error converges.

[0043] In this embodiment, the reinforcement learning control steps described above illustrate a multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output (MIMO) systems, which utilizes a reinforcement learning agent constructed from a fusion policy-evaluation network. Before applying the reinforcement learning agent to the real-time control system, it needs to be trained. The method is characterized by further including end-to-end training of the reinforcement learning agent with dual-delay deep deterministic policy gradients before applying it to the MIMO system.

[0044] In this embodiment, steps S21 and S22 are similar to steps S11 to S12 described above. They involve extracting various variable data from the MIMO system mechanism model, constructing a structured global directed graph representing the coupling relationships between the variable data within the MIMO system based on this data, and reconstructing the global directed graph into multiple independent subgraphs representing deep features. Furthermore, the policy network outputs control actions in the reinforcement learning agent, and the evaluation network provides value scores. The specific processes have been explained in the previous steps and will not be repeated here.

[0045] For step S23, reinforcement learning agent training and deployment, in the boiler-turbine unit system environment, stable training techniques such as experience playback and target network can be used to train the policy network and evaluation network constructed above end-to-end.

[0046] As one implementation method, iterative training until the error converges includes: The intermediate state variables, control actions, and constraints of this training round are stored in the training experience replay pool as historical data. In the next training round, a portion of historical data is randomly sampled from the training experience replay pool to train the reinforcement learning agent, thereby breaking the temporal correlation of reinforcement learning. The training process is guided by a piecewise reward function, which not only rewards and penalizes the tracking accuracy of individual variables in segments but also includes a collaborative reward term. When all controlled variables simultaneously achieve high-precision tracking, an additional reward is given, thus encouraging the controller to achieve coordinated optimization among multiple variables.

[0047] Specifically, the policy and evaluation network is trained using a dual-delay deep deterministic policy gradient algorithm, coupled with an experience replay mechanism. During training, the agent interacts with a boiler-turbine unit environment constructed based on a mechanistic model, learning the optimal control strategy by optimizing a multi-objective reward function. The reward function comprehensively considers the tracking accuracy of each controlled variable, the collaborative adjustment effect, and system safety constraints. Tracking accuracy formula (5): (5) Formula for synergistic regulation effect (6): (6) System safety constraint formula (7): (7) Among them, the The controlled state and the controlled state setpoint of the state are in The normalization bias at time is The design of the reward function consists of two parts: first, constructing a single-index reward for each controlled object. This is used to characterize the effect of local control; based on this, an additional reward is introduced according to the overall deviation of all controlled variables. Additional rewards are given when all system states fall within the allowable error range. Finally, the sum of these individual rewards is used as the total reward for each step, guiding the strategy to achieve precise control while ensuring overall coordination.

[0048] In the initial stage of training, the agent outputs actions based on the current policy network, and adds random noise to the actions to enhance its exploration capabilities. The actions are then applied to the environment to obtain system state transitions and immediate rewards. After each interaction, the agent stores information such as its state, action, reward, and next state in the experience replay pool. During training, small batches of historical data are randomly sampled from the experience replay pool to break the temporal correlation between samples, thereby improving training stability and sample utilization efficiency.

[0049] As one implementation method, modeling the control action to determine a value score includes: Simultaneously, two independent evaluation networks are constructed to determine the value of the control action, resulting in two value scores; The value score with the smallest value among the two value scores is selected as the final value score of the control action to reduce overestimation bias.

[0050] In this embodiment, the method employs two independent Critic networks. By using the "minimum value of the two Critic networks," the overestimation of Q-values ​​is effectively suppressed, thereby improving the stability of policy learning and control performance. When updating the Critic network, the target policy network is used to generate the action for the next time step, and smooth noise is added to this action to enhance the policy's robustness to small perturbations. Unlike the frequent updates of the Critic network, a delayed update mechanism is adopted, i.e., a policy update is performed only after several Critic network updates, in order to reduce the variance of policy gradient estimation and improve overall training stability. The goal of updating the policy network is to maximize the Critic network's value assessment of the current policy output action. During training, the target network gradually follows the changes in the parameters of the main network through soft updates to ensure the stability of the target value calculation. Through the above interactive sampling, value assessment, and policy iterative update process, the agent can gradually learn a stable control policy suitable for continuous control tasks. Finally, the reinforcement learning control policy after training convergence is deployed to a complex multi-input multi-output control system.

[0051] The trained policy network parameters are integrated into the real-time control system. During online operation (optionally, the trained policy network parameters can also be combined with the integral compensation module), the control system collects the current state in each sampling period, obtains the original action through policy network inference (and then corrects it through the integral compensation module), and finally sends it to each execution unit of the system to achieve coordinated and optimized control of the controlled state of the system. This is roughly the process of applying reinforcement learning control after training in S11-S12.

[0052] Through the above training, the method of the present invention significantly outperforms traditional PID and mainstream deep reinforcement learning algorithms in key indicators such as dynamic response speed, overshoot, settling time and steady-state accuracy in terms of load tracking and disturbance rejection.

[0053] As one implementation method, this method also introduces an integral compensator at the output of the policy network. The reinforcement learning agent further includes an integral compensator, which is used to receive the output of the policy network, calculate the deviation between the controlled variable and the preset target value in real time, generate a dynamic compensation signal, and use the dynamic compensation signal to compensate the output in order to continuously correct the steady-state deviation that the reinforcement learning policy has failed to completely eliminate.

[0054] In this embodiment, the module calculates the deviation between the controlled variable and the set value in real time. For example, in a DC boiler-turbine unit, it calculates the deviation between the main steam pressure, intermediate point steam enthalpy, and unit load and the set values ​​under ideal conditions in real time (if applied to a multi-stage continuous stirred tank, it calculates the deviation between the temperature, concentration, liquid level, and pressure during stirring and the set values ​​under ideal conditions in real time. Similarly, when applied to other multiple-input multiple-output control systems, it calculates the deviation between the controlled variable and the set value of the corresponding control system in real time), and performs an integral operation on the deviation to generate a dynamic compensation signal.

[0055] This compensation signal is compared with the original control action output by the policy network. These components, when combined, form the final control command that acts on the actual system. This design effectively inherits the advantage of traditional control's integral element in eliminating steady-state errors, while complementing the dynamic optimization capabilities of reinforcement learning.

[0056] Specifically, in the multi-subgraph fusion strategy - evaluation network output of the original control action. Then, an integral compensation module is introduced to eliminate steady-state error. The integrator calculates and accumulates the deviation between the controlled output and the set value in real time, and finally obtains the output compensation. The calculation formula (8) for the output compensation is: (8) in, Represents the controlled variable Its set value The real-time deviation vector between them This is an adjustable integral gain coefficient matrix. This integral compensation mechanism can further and continuously correct the small steady-state deviations that the reinforcement learning strategy failed to completely eliminate, significantly improving the control accuracy of the system in long-term operation. The overall structure of all the above steps in this method is as follows: Figure 5 As shown.

[0057] As can be seen from this implementation method, the introduction of an integral compensation module creatively combines the advantages of traditional control theory and modern reinforcement learning, effectively eliminating the steady-state error that may exist in reinforcement learning strategies and ensuring the high accuracy of the system during long-term operation.

[0058] The actual effects of this method are explained, such as... Figure 6 The changes in the average reward curve during training are shown. It can be seen that within approximately 200 training epochs, the reward value rapidly increases from an initial negative value and stabilizes at a high level, indicating that the controller has successfully learned an effective control strategy. After training is complete, the network parameters are saved for online deployment.

[0059] To verify the control performance, load tracking tests were conducted on the DC boiler-turbine unit.

[0060] like Figure 7 The effectiveness of this method in regulating main steam pressure is demonstrated. The reinforcement learning agent exhibits excellent dynamic characteristics, achieving minimal overshoot and rapid convergence. The pressure response curve smoothly transitions to the setpoint, effectively suppressing the impact of pressure fluctuations on the boiler system and ensuring the operational stability of the main steam system. MSIRL is used to comprehensively evaluate the tracking control effect.

[0061] like Figure 8 The effectiveness of this method in controlling the intermediate-point steam enthalpy is demonstrated, and the proposed algorithm exhibits excellent tracking performance. The steam enthalpy response is stable, with no drastic oscillations during the dynamic process, and the steady-state deviation is controlled within a small range. This precise enthalpy control helps maintain the thermal load balance of the boiler's heating surfaces and improves the overall thermal efficiency of the unit.

[0062] like Figure 9 This paper demonstrates the load regulation process of the proposed method, showcasing that the controller achieves a smooth load response characteristic. The unit load rises steadily to the setpoint, overshoot is significantly reduced, and the load is quickly stabilized near the target value. This smooth load regulation capability helps reduce frequency impacts on the power grid and improves the grid compatibility of the unit.

[0063] The experimental results above demonstrate that the control system provided by this method can achieve high-performance control of a DC boiler-turbine unit, meeting industrial operation requirements in terms of dynamic response, steady-state accuracy, and anti-interference capability. Similarly, the reinforcement learning agent of this method can be applied to other multiple-input multiple-output control systems to achieve the same results.

[0064] like Figure 10 The diagram shown is a structural schematic of a multi-subgraph fusion reinforcement learning control system for multi-input multi-output systems according to an embodiment of the present invention. The system can execute the multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems described in any of the above embodiments and is configured in a terminal.

[0065] This embodiment provides a multi-subgraph fusion reinforcement learning control system 10 for multiple-input multiple-output systems, which includes: a mechanism model and graph construction module 11 and a reinforcement learning agent module 12.

[0066] The mechanism model and graph construction module 11 is used to extract various variable data within the mechanism model of the multi-input multi-output system, construct a global directed graph that structurally represents the coupling relationship of variable data within the multi-input multi-output system based on the various variable data, and reconstruct the global directed graph into multiple independent subgraphs representing deep features. The reinforcement learning agent module 12 is used to input the subgraphs into a reinforcement learning agent constructed by a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraphs. In the evaluation network, the control actions are modeled to obtain a multi-level causal path from the execution of the control action to the controlled output of the multi-input multi-output system. A multi-level causal fusion representation that adapts to and integrates deep features is determined through the multi-level causal path to obtain a value score. The value score is used to guide the training of the reinforcement learning agent, enabling the reinforcement learning agent to perform reinforcement learning control on the multi-input multi-output system.

[0067] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems in any of the above method embodiments. In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: Extract various variable data from the mechanism model of a multiple-input multiple-output system, construct a global directed graph based on the various variable data to structurally represent the coupling relationship of variable data in the multiple-input multiple-output system, and reconstruct the global directed graph into multiple independent subgraphs representing deep features; The subgraph is input into a reinforcement learning agent constructed from a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled to obtain a multi-level causal path from the execution of the control action to the controlled output of the MIMO system. A multi-level causal fusion representation that adapts to and integrates deep features is determined through the multi-level causal path to obtain a value score. The value score is used to guide the training of the reinforcement learning agent, enabling the reinforcement learning agent to perform reinforcement learning control on the MIMO system.

[0068] like Figure 11 The diagram shown is a structural schematic of a reinforcement learning agent training system for multi-subgraph fusion of multiple-input multiple-output systems according to an embodiment of the present invention. The system can execute the reinforcement learning agent training method for multi-subgraph fusion of multiple-input multiple-output systems as described in any of the above embodiments and is configured in a terminal.

[0069] This embodiment provides a reinforcement learning agent training system 20 for multi-subgraph fusion of multiple-input multiple-output systems, which includes: a mechanism model and graph construction module 21, a reinforcement learning agent module 22, a steady-state error compensation module 23, and a reinforcement learning training module 24.

[0070] The mechanism model and graph construction module 21 is used to acquire the mechanism model of the multiple-input multiple-output system, construct a corresponding global directed graph as a graph representation based on the coupling relationship between the variable data in the mechanism model, and decompose the global directed graph into multiple independent subgraphs by the category of the variable data for training the reinforcement learning agent; the reinforcement learning agent module 22 is used to input the subgraphs into a reinforcement learning agent constructed by a policy network and an evaluation network, in which the policy network generates control actions that focus on local dynamics and global coordination optimization based on the subgraphs, and in the evaluation network, the control actions are modeled and a value score is determined; the steady-state error compensation module 23 is used to receive the output of the policy network. The system calculates the deviation between the controlled variable and the preset target value in real time, generates a dynamic compensation signal, and uses the dynamic compensation signal to compensate the output to continuously correct the steady-state deviation that the reinforcement learning strategy failed to completely eliminate. The reinforcement learning training module 24 is used to determine the policy gradient reward function based on the error between the value score and the preset controlled state setting value. The policy gradient reward function is used to constrain the value score of the control action corresponding to a single variable data to enhance the local control effect. The additional reward is used to constrain the sum of the value scores of the control actions corresponding to all variable data to enhance the global control effect. The reinforcement learning agent is iteratively trained through the reward function until the error converges.

[0071] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the methods in any of the above method embodiments; In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: Obtain the mechanism model of the multiple-input multiple-output system, construct a corresponding global directed graph as a graph representation based on the coupling relationship between the variable data in the mechanism model, and decompose the global directed graph into multiple independent subgraphs by the category of variable data for training reinforcement learning agents; The subgraph is input into a reinforcement learning agent constructed by a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled and a value score is determined. The policy gradient reward function is determined based on the error between the value score and the preset controlled state setting. The policy gradient reward function is used to constrain the value score of the control action corresponding to a single variable data to enhance the local control effect. The sum of the value scores of the control actions corresponding to all variable data is constrained by the additional reward to enhance the global control effect. The reinforcement learning agent is iteratively trained through the reward function until the error converges.

[0072] Figure 12 This is a schematic diagram of the hardware structure of an electronic device for a multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems, as provided in another embodiment of this application. Figure 12 As shown, the device includes: One or more processors 1210 and memory 1220, Figure 12 Taking a processor 1210 as an example, the device for a multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems may further include an input device 1230 and an output device 1240.

[0073] The processor 1210, memory 1220, input device 1230, and output device 1240 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0074] The memory 1220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-subgraph fusion reinforcement learning control method for MIMO systems in this embodiment. The processor 1210 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 1220, thereby implementing the multi-subgraph fusion reinforcement learning control method for MIMO systems described in the above embodiment.

[0075] The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1220 may optionally include memory remotely located relative to the processor 1210, and these remote memories may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0076] Input device 1230 can receive input numerical or character information. Output device 1240 may include display devices such as a display screen.

[0077] The one or more modules are stored in the memory 1220. When executed by the one or more processors 1210, they execute the multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems in any of the above method embodiments.

[0078] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0079] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0080] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the multi-subgraph fusion reinforcement learning control method for multi-input multi-output systems according to any embodiment of this invention.

[0081] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0082] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0083] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0084] (4) Other electronic devices with data processing functions.

[0085] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-subgraph fusion reinforcement learning control method for multiple-input multiple-output systems, comprising: Extract various variable data from the mechanism model of a multiple-input multiple-output system, construct a global directed graph based on the various variable data to structurally represent the coupling relationship of variable data in the multiple-input multiple-output system, and reconstruct the global directed graph into multiple independent subgraphs representing deep features; The subgraph is input into a reinforcement learning agent constructed from a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled to obtain a multi-level causal path from the execution of the control action to the controlled output of the MIMO system. A multi-level causal fusion representation that adapts to and integrates deep features is determined through the multi-level causal path to obtain a value score. The value score is used to guide the training of the reinforcement learning agent, enabling the reinforcement learning agent to perform reinforcement learning control on the MIMO system.

2. The multi-subgraph fusion reinforcement learning control method according to claim 1, characterized in that, The extraction of multiple variable data within the mechanistic model of the multiple-input multiple-output system, and the construction of a global directed graph based on the multiple variable data to structurally represent the coupling relationships of the variable data within the multiple-input multiple-output system, includes: Using the various variable data as nodes and the dynamic causal relationships between the various variable data as directed edges, a global directed graph is constructed to structurally represent the coupling relationships of the variable data categories within the multi-input multi-output system. The categories of the variable data include: control variables, intermediate state variables, and controlled variables.

3. The multi-subgraph fusion reinforcement learning control method according to claim 2, characterized in that, The generation of control actions that focus on local dynamics and global coordination optimization based on the subgraph in the policy network includes: The variable data corresponding to multiple independent subgraphs are split into multiple subsets according to categories; Independent graph attention networks are used to extract features from multiple subgraphs rich in local coupling relationships within the subsets; By mapping the fused features of the multiple subgraphs using a multilayer perceptron, control actions are generated to control the control variables within the multiple-input multiple-output system.

4. The multi-subgraph fusion reinforcement learning control method according to claim 2, characterized in that, In the evaluation network, the control action is modeled to obtain a multi-level causal path from the execution of the control action by the multiple-input multiple-output system to the controlled output. A multi-level causal fusion representation that adapts and integrates deep features is determined through this multi-level causal path to obtain a value score, including: The evaluation network models the control action, and determines the indirect influence of the control variable on the intermediate state variable based on the control action through hierarchical causal modeling, as well as the deep internal coupling influence from the intermediate state variable to the controlled variable. Based on the indirect influence and the deep influence, a multi-level causal path is determined. The information of the multi-level causal path is superimposed and fused to obtain a multi-level causal fusion representation; The deep features of the multi-level causal fusion representation are extracted using a graph attention network, and the value score of the deep features is determined by a multilayer perceptron.

5. A reinforcement learning agent training method for multi-subgraph fusion in multiple-input multiple-output systems, comprising: Obtain the mechanism model of the multiple-input multiple-output system, construct a corresponding global directed graph as a graph representation based on the coupling relationship between the variable data in the mechanism model, and decompose the global directed graph into multiple independent subgraphs by the category of variable data for training reinforcement learning agents; The subgraph is input into a reinforcement learning agent constructed by a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled and a value score is determined. The policy gradient reward function is determined based on the error between the value score and the preset controlled state setting. The policy gradient reward function is used to constrain the value score of the control action corresponding to a single variable data to enhance the local control effect. The sum of the value scores of the control actions corresponding to all variable data is constrained by the additional reward to enhance the global control effect. The reinforcement learning agent is iteratively trained through the reward function until the error converges.

6. The reinforcement learning agent training method for multi-subgraph fusion according to claim 5, characterized in that, The step of iteratively training the reinforcement learning agent using the reward function until the error converges includes: The intermediate state variables, control actions, and constraints of this training round are stored in the training experience replay pool as historical data. In the next training round, a portion of the historical data is randomly sampled from the training experience replay pool to train the reinforcement learning agent, thereby achieving reinforcement learning that breaks the temporal correlation.

7. The reinforcement learning agent training method for multi-subgraph fusion according to claim 5, characterized in that, In the evaluation network, modeling the control action to determine a value score includes: Simultaneously, two independent evaluation networks are constructed to determine the value of the control action, resulting in two value scores; The value score with the smallest value among the two value scores is selected as the final value score of the control action to reduce overestimation bias.

8. The reinforcement learning agent training method for multi-subgraph fusion according to claim 5, characterized in that, The reinforcement learning agent also includes an integral compensator, which receives the output of the policy network, calculates the deviation between the controlled variable and the preset target value in real time, generates a dynamic compensation signal, and uses the dynamic compensation signal to compensate the output in order to continuously correct the steady-state deviation that the reinforcement learning policy has failed to completely eliminate.

9. The method according to any one of claims 1-8, characterized in that, The multiple input multiple output system includes a boiler-turbine unit, and the categories of the variable data include control variables, intermediate state variables, and controlled variables. The control variables include: coal feed rate, water feed rate, and turbine valve opening; The intermediate state variables include: furnace pulverized coal flow rate and separator steam pressure; The controlled variables include: main steam pressure, intermediate point steam enthalpy, and unit load.

10. A multi-subgraph fusion reinforcement learning control system for multiple-input multiple-output systems, comprising: The mechanism model and graph construction module is used to extract multiple variable data within the mechanism model of the multi-input multi-output system, construct a global directed graph that structurally represents the coupling relationship of variable data within the multi-input multi-output system based on the multiple variable data, and reconstruct the global directed graph into multiple independent subgraphs representing deep features; A reinforcement learning agent module is used to input the subgraph into a reinforcement learning agent constructed from a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled to obtain a multi-level causal path from the execution of the control action to the controlled output of the multi-input multi-output system. A multi-level causal fusion representation that adapts and integrates deep features is determined through the multi-level causal path to obtain a value score. The value score is used to guide the training of the reinforcement learning agent, enabling the reinforcement learning agent to perform reinforcement learning control on the multi-input multi-output system.

11. A reinforcement learning agent training system for multi-subgraph fusion in multiple-input multiple-output systems, comprising: The mechanism model and graph construction module is used to obtain the mechanism model of the multiple-input multiple-output system, construct a corresponding global directed graph as a graph representation based on the coupling relationship between the variable data in the mechanism model, and decompose the global directed graph into multiple independent subgraphs by the category of variable data for the training of reinforcement learning agents. A reinforcement learning agent module is used to input the subgraph into a reinforcement learning agent constructed by a policy network and an evaluation network. In the policy network, control actions that focus on local dynamics and global coordination optimization are generated based on the subgraph. In the evaluation network, the control actions are modeled and a value score is determined. The steady-state error compensation module is used to receive the output of the policy network, calculate the deviation between the controlled variable and the preset target value in real time, generate a dynamic compensation signal, and use the dynamic compensation signal to compensate the output in order to continuously correct the steady-state deviation that the reinforcement learning policy has failed to completely eliminate. The reinforcement learning training module is used to determine the policy gradient reward function based on the error between the value score and the preset controlled state setting value. The policy gradient reward function is used to constrain the value score of the control action corresponding to a single variable data to enhance the local control effect. The additional reward is used to constrain the sum of the value scores of the control actions corresponding to all variable data to enhance the global control effect. The reinforcement learning agent is iteratively trained through the reward function until the error converges.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-8.

13. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-8.

14. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-8.

Images