A Deep Reinforcement Learning-Based Method and System for Hydraulic Balance Optimization of Secondary Pipeline Networks

Abstract

This application provides a method and system for optimizing the hydraulic balance of a secondary pipe network based on deep reinforcement learning, relating to the field of intelligent heating control technology. The method first acquires multi-source heterogeneous data and generates standardized feature vectors. Then, a deep reinforcement learning model is used to perform multi-objective collaborative optimization to generate output commands. These output commands are then sent to the target controlled equipment in the heating system to dynamically adjust the system's operating state. Furthermore, based on real-time collected pipe network system data, the system performance is dynamically evaluated in a digital twin environment, and the model parameters are continuously iteratively optimized through online training. This method can transform the linkage relationships of multiple controlled devices in the secondary pipe network into state transition probabilities and combine topological features for multi-objective collaborative optimization to generate a globally optimal solution. It can also dynamically evaluate system performance in a digital twin environment and continuously iteratively optimize model parameters through online training, thereby improving the hydraulic balance optimization effect of the secondary pipe network.

Description

Technical Field

[0001] This application relates to the field of intelligent heating control technology, and in particular to a method and system for optimizing the hydraulic balance of secondary pipe networks based on deep reinforcement learning. Background Technology

[0002] In centralized heating control, the heating circulation pipeline can be divided into primary and secondary networks based on the transmission and distribution of the heat medium. The primary network refers to the high-temperature, high-pressure pipeline network from the heat source outlets (such as power plants and district boiler rooms) to the inlets of various heat exchange stations. The secondary network refers to the pipeline network from the heat exchange station outlets to user-side heat dissipation equipment such as radiators, underfloor heating coils, and fan coil units. To improve the overall operating efficiency, energy consumption, and user experience of the heating system, hydraulic balance optimization of both the primary and secondary networks is necessary.

[0003] Hydraulic balance optimization aims to minimize the total circulating water volume of the heating system while ensuring that all terminals receive the design flow rate, thereby reducing the energy consumption of circulation control equipment such as pumps. The hydraulic balance optimization process begins with collecting operational data and conducting diagnostic analysis on actual flow ratios, imbalance rates, and pump operating conditions through qualitative judgment or quantitative calculation. Then, starting from the most unfavorable loop in the system, adjustments are made step-by-step towards the heat source, setting the operating conditions of control equipment at each stage. Finally, based on load forecasts, the frequency of variable frequency pumps or the opening of electric valves are automatically adjusted to achieve dynamic and adaptive hydraulic balance.

[0004] However, during the hydraulic balance optimization of secondary pipe networks, heat energy consumption exceeds user demand, resulting in high energy consumption per unit area. Manual adjustment takes 3-7 days, leading to low optimization efficiency and difficulty in handling dynamic loads. Furthermore, linear models using PID control perform poorly when dealing with strongly coupled nonlinear systems like secondary pipe networks, with overshoot exceeding 15%, further hindering the hydraulic balance optimization of the secondary pipe network. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method and system for optimizing the hydraulic balance of secondary pipe networks based on deep reinforcement learning, addressing the problem of poor hydraulic balance optimization performance in secondary pipe networks.

[0006] According to a first aspect of this application, a method for optimizing the hydraulic balance of a secondary pipe network based on deep reinforcement learning is provided, the method comprising: Acquire multi-source heterogeneous data, which includes at least one of the following: flow data, pressure data, temperature data, valve opening status, and key parameters of PID regulation collected from the user side of the secondary pipeline network and heat exchange station in the heating system. Generate standardized feature vectors based on the multi-source heterogeneous data; A deep reinforcement learning model is used to perform multi-objective collaborative optimization based on the standardized feature vectors to generate output instructions; the deep reinforcement learning model is a deep reinforcement learning engine constructed by fusing pipeline topology features using proximal policy optimization or flexible actor-critic algorithm; the deep reinforcement learning model is used to model the dynamic characteristics of the heating system through Markov decision process; the output instructions include at least one of valve control signals and pump frequency conversion signals; The output command is sent to the target controlled device in the heating system so that the target controlled device can dynamically adjust the system operating status according to the output command; the operating status includes at least one of the following: branch flow distribution, total flow of secondary pipe network and system pressure difference, and global pipe network impedance curve. The system performance is dynamically evaluated in a digital twin environment based on real-time collected pipeline system data, and the model parameters of the deep reinforcement learning model are continuously iteratively optimized through online training based on the system performance evaluation results; the pipeline system data includes multi-source physical data collected by the target controlled equipment for the heating system after executing the output command.

[0007] In some embodiments, acquiring multi-source heterogeneous data includes: In response to the optimized execution instruction, a data acquisition request is generated, the data acquisition request including a first acquisition request and a second acquisition request; The first acquisition request is sent to the sensor device of the environmental perception layer in the heating system to obtain the real-time physical data collected by the sensor device; The second acquisition request is sent to the PID adjustment history database of the environmental perception layer in the heating system to obtain the key parameters of the PID adjustment. Historical PID operating characteristics are generated based on the aforementioned key PID adjustment parameters; The multi-source heterogeneous data is obtained by fusing the real-time physical data and the historical PID operating characteristics.

[0008] In some embodiments, generating a standardized feature vector based on the multi-source heterogeneous data includes: Sensor heterogeneous data is extracted from the multi-source heterogeneous data; the sensor heterogeneous data includes user-side data and heat exchange station data. The heterogeneous sensor data is fused through spatiotemporal alignment; The Z-value normalization algorithm is used to normalize the fused heterogeneous sensor data; The topology of the heating system's pipe network was analyzed using a graph neural network. By combining the normalized heterogeneous sensor data and the pipeline topology, the standardized feature vector is generated by adjusting the controller.

[0009] In some embodiments, a deep reinforcement learning model is used to perform multi-objective collaborative optimization based on the standardized feature vectors to generate output instructions, including: The decision objective is set to distribute the branch flow by adjusting the real-time opening of the user-side regulating valve and to coordinate the control of the total flow and pressure gradient of the pipeline network by optimizing the speed of the secondary pipeline circulation pump, so as to achieve system-level hydraulic balance. The standardized feature vector is input into the deep reinforcement learning model; The dynamic characteristics of the heating system are modeled using a Markov decision process. The optimal control strategy is calculated based on the standardized feature vector and the dynamic characteristics using either proximal strategy optimization or a flexible actor critic algorithm. The output command is generated based on the optimal control strategy.

[0010] In some embodiments, sending the output command to a target controlled device in the heating system, so that the target controlled device dynamically adjusts the system operating state according to the output command, includes: The output command is sent to the execution control layer of the heating system; The execution control layer converts the output command into physical operation signals, which include valve control signals and water pump frequency conversion signals. The physical operation signal is sent to the target controlled device.

[0011] In some embodiments, dynamically evaluating system performance in a digital twin environment based on real-time acquired pipeline system data includes: Receive the execution receipt signal fed back by the target controlled device in response to the physical operation signal; Obtain the cumulative duration for which the physical operation signal is sent to the target controlled device; In response to receiving the execution receipt signal, or when the accumulated time reaches a preset time threshold, the pipeline system data is acquired; Extract control result parameters from the pipeline system data; By comparing the control result parameters with the key PID adjustment parameters, a system performance evaluation result is generated.

[0012] In some embodiments, dynamically evaluating system performance in a digital twin environment based on real-time acquired pipeline system data includes: A hydraulic-thermal coupling model is constructed based on the pipeline network topology and the key parameters of the PID regulation; the boundary conditions of the hydraulic-thermal coupling model are the multi-source heterogeneous data. A digital twin environment is constructed based on the aforementioned hydraulic-thermal coupling model; Set the alarm threshold for the digital twin environment; When the multi-source heterogeneous data meets the alarm threshold, simulation is automatically triggered to obtain simulation result information; The system performance evaluation results are generated based on the simulation results.

[0013] In some embodiments, the model parameters of the deep reinforcement learning model are continuously and iteratively optimized through online training based on system performance evaluation results, including: Obtain the physical parameters of the control target; Extract real-time monitoring physical parameters from the pipeline system data; The model output error is calculated by comparing the physical parameters of the control target with the real-time monitored physical parameters. The model parameters of the deep reinforcement learning model are continuously optimized iteratively by backpropagating the model output error.

[0014] In some embodiments, the method further includes: Obtain a model training dataset, which includes sample data with parameter labels; the sample data includes heterogeneous multi-source sample data extracted from a PID regulation history database; the parameter labels are set according to the key parameters of the PID regulation. Input the training dataset of the model into the deep reinforcement learning model to obtain the training output instructions output by the deep reinforcement learning model; Extract training instruction parameters from the training output instructions; The model training loss is calculated by comparing the training instruction parameters and the parameter labels. If the model training loss is greater than a preset loss threshold, the model parameters of the deep reinforcement learning model are adjusted according to the model training loss. If the training loss of the model is less than or equal to a preset loss threshold, the model parameters of the deep reinforcement learning model are output.

[0015] According to a second aspect of this application, a secondary pipe network hydraulic balance optimization system based on deep reinforcement learning is provided, the system comprising: The environmental perception layer is used to acquire multi-source heterogeneous data, which includes at least one of the following: flow data, pressure data, temperature data, valve opening status, and key parameters of PID regulation collected from the user side of the secondary pipeline network and heat exchange station in the heating system. The data processing layer is used to generate standardized feature vectors based on the multi-source heterogeneous data; A deep reinforcement learning decision layer is used to perform multi-objective collaborative optimization based on the standardized feature vector using a deep reinforcement learning model to generate output instructions; the deep reinforcement learning model is a deep reinforcement learning engine constructed by fusing network topology features using proximal policy optimization or a flexible actor-critic algorithm; the deep reinforcement learning model is used to model the dynamic characteristics of the heating system through a Markov decision process; the output instructions include at least one of valve control signals and pump frequency conversion signals. An execution control layer is used to send the output command to the target controlled equipment in the heating system, so that the target controlled equipment dynamically adjusts the system operating state according to the output command; the operating state includes at least one of branch flow distribution, total flow of secondary pipe network and system pressure difference, and global pipe network impedance curve. The feedback optimization layer is used to dynamically evaluate system performance in a digital twin environment based on real-time collected pipeline system data, and to continuously iteratively optimize the model parameters of the deep reinforcement learning model through online training based on the system performance evaluation results; the pipeline system data includes multi-source physical data collected by the target controlled device for the heating system after executing the output command.

[0016] By employing the above technical solutions, this application provides a method and system for optimizing the hydraulic balance of a secondary pipe network based on deep reinforcement learning. The method first acquires multi-source heterogeneous data and generates standardized feature vectors. Then, a deep reinforcement learning model is used to perform multi-objective collaborative optimization to generate output instructions. These output instructions are then sent to the target controlled equipment in the heating system to dynamically adjust the system's operating state. Furthermore, based on real-time collected pipe network system data, the system performance is dynamically evaluated in a digital twin environment, and the model parameters are continuously iteratively optimized through online training. This method can transform the linkage relationships of multiple controlled devices in the secondary pipe network into state transition probabilities and use graph neural networks to encode topological features for multi-objective collaborative optimization, generating a globally optimal solution. It can also dynamically evaluate system performance in a digital twin environment and continuously iteratively optimize model parameters through online training, thereby improving the hydraulic balance optimization effect of the secondary pipe network.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the heating system structure provided in an embodiment of this application; Figure 2 A schematic diagram of the process for optimizing the hydraulic balance of a secondary pipe network based on deep reinforcement learning, provided in an embodiment of this application; Figure 3 This is a schematic diagram of the system hierarchy provided in the embodiments of this application; Figure 4 This is a schematic diagram of the deep reinforcement learning training process provided in an embodiment of this application; Figure 5 A schematic diagram of the secondary pipeline hydraulic balance optimization system based on deep reinforcement learning provided in this application embodiment. Detailed Implementation

[0019] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0020] In this application embodiment, the heating system is also referred to as a district heating system or a centralized heating system, which can transmit heat energy from centralized or decentralized heat sources to multiple buildings or users through a pipeline network to meet the heating and domestic hot water needs of users in the area.

[0021] A heating system can include heat sources, heating networks, heat exchange stations, and user-side terminal equipment. Among them, heat sources refer to facilities that produce heat energy, such as combined heat and power plants, district boiler rooms, industrial waste heat, geothermal energy, and heat pump stations.

[0022] A heating network is a pipeline network in a heating system used to transport heat media such as high-temperature water or steam. It can connect heat sources and end-user devices. A heating network can include primary and secondary pipeline networks. The primary pipeline network refers to the high-temperature, high-pressure pipeline network from the outlet of heat sources such as power plants and district boiler rooms to the inlet of each heat exchange station. The secondary pipeline network refers to the pipeline network from the outlet of the heat exchange station to user-side heat dissipation equipment such as radiators, underfloor heating coils, and fan coil units.

[0023] A heat exchange station, also known as a heating station, is a node used to connect primary and secondary pipe networks. Heat exchange stations facilitate heat exchange and provide hydraulic and thermal isolation between systems. User-side terminal equipment refers to the heat dissipation terminal equipment and indoor piping deployed inside the user's building, such as radiators, floor coils, and fan coil units.

[0024] To improve the overall operating efficiency, energy consumption, and user experience of the heating system, hydraulic balance optimization of the primary and secondary pipe networks is necessary. Hydraulic balance optimization aims to minimize the total circulating water volume of the heating system while ensuring that all terminals receive the design flow rate, thereby reducing the energy consumption of circulation control equipment such as pumps.

[0025] Hydraulic balance optimization can begin by collecting operational data and conducting diagnostic analysis on actual flow ratios, imbalance rates, and pump operating conditions through qualitative judgment or quantitative calculation. Then, starting from the most unfavorable loop in the system, adjustments are made step by step towards the heat source, and the operating conditions of control equipment are set step by step. Finally, based on load forecasts, the frequency of variable frequency pumps or the opening of electric valves are automatically adjusted to achieve dynamic and adaptive hydraulic balance.

[0026] In some embodiments, hydraulic balance optimization of the heating network can be achieved through manual adjustment. This involves acquiring heating network operation data through manual inspections or IoT data reporting, performing statistical analysis on the data to determine whether the current operating status of the heating network meets heating demand, analyzing potential operational risks, and then adjusting circulation control equipment such as valves and pumps in the heating network based on the analysis results. This aims to reduce the energy consumption of circulation control equipment while ensuring that all terminals receive the design flow rate.

[0027] Clearly, optimizing the hydraulic balance of the heating network through manual adjustment is inefficient, taking 3-7 days to complete. Furthermore, during the secondary network hydraulic balance optimization process, heat consumption exceeds user demand, resulting in high energy consumption per unit area. Manual adjustment typically targets a small number of circulation control devices, making it difficult to handle the dynamic load of the secondary network.

[0028] In some embodiments, a linear model of proportional-integral-differential (PID) control can also be used to optimize the hydraulic balance of the heating network, so as to ensure that each parallel branch obtains the required flow rate while the return water temperature approaches a consistent set value through the PID control curve.

[0029] When implementing PID control, a linearized model can be constructed first, and an independent PID controller can be designed for each branch. Then, the initial PID parameters are calculated through parameter tuning. Next, all valves are set to a certain opening degree, and after the system stabilizes, the initial return water temperature of each branch is recorded. The error is then calculated based on the return water temperature, and a PID control curve is generated based on this error to achieve dynamic adjustment. For example, if the return water temperature is too low, it indicates that the flow rate is too small, and the PID controller will output a signal to open the valve wider. Conversely, if the return water temperature is too high, it indicates that the flow rate is too large, and the PID controller will output a signal to close the valve.

[0030] However, the PID control process has poor adaptability and poor optimization effect when dealing with strongly coupled nonlinear systems such as secondary pipe networks. The overshoot exceeds 15%, resulting in poor hydraulic balance optimization effect of secondary pipe networks.

[0031] To address the problem of poor hydraulic balance optimization in secondary pipe networks, this application provides a hydraulic balance optimization method for secondary pipe networks based on deep reinforcement learning (DRL) in some embodiments. The method can perform autonomous hydraulic balance optimization of heating secondary pipe networks based on the DRL model, and achieve dynamic optimization and control by constructing a DRL decision model that integrates the topological characteristics of the pipe network.

[0032] By deploying the aforementioned method, a secondary pipe network hydraulic balance optimization system based on deep reinforcement learning can be formed in the heating system. For example... Figure 1 As shown, the system mainly includes an environmental perception layer, a data processing layer, a deep reinforcement learning decision-making layer, an execution control layer, and a feedback optimization layer. The environmental perception layer is equipped with controllers, temperature sensors, pressure sensors, electromagnetic flowmeters, and intelligent regulating valves for real-time data acquisition and understanding of the pipeline network topology. The execution control layer can deploy circulation control equipment such as intelligent regulating valves and secondary pipeline circulation pumps. These intelligent regulating valves and secondary pipeline circulation pumps can receive data signals from the execution control layer for real-time adjustment. The remaining layers are all built as intelligent data platforms to realize the serial operation of the entire system.

[0033] The method can also be applied to electronic devices that establish a communication connection with an intelligent data platform and have data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. For ease of description, the intelligent data platform is used as the execution subject of the method in this embodiment. It should be understood that the method can also be applied to other types of execution subjects, which are not illustrated in this embodiment. Figure 2 As shown, the method includes: S101. Obtain multi-source heterogeneous data.

[0034] When performing hydraulic balance optimization of the secondary pipeline network, multi-source heterogeneous data can be acquired first. This multi-source heterogeneous data includes at least one of the following: flow data, pressure data, temperature data, valve opening status, and key PID control parameters collected from the user side and heat exchange stations of the secondary pipeline network in the heating system.

[0035] For example, a multi-source heterogeneous data acquisition system can be built on the user side (A~N) and heat exchange station by deploying an environmental sensing layer. The multi-source heterogeneous data acquisition system can include sensor devices such as electromagnetic flowmeters, pressure transmitters, temperature sensors, and smart control valves.

[0036] When acquiring multi-source heterogeneous data, flow data can be captured in real time through electromagnetic flowmeters in each branch; pressure transmitters can monitor the pressure difference in the supply or return water network, i.e., pressure data; temperature sensors can synchronously collect supply or return water temperature, as well as user room temperature data, and other temperature data; and intelligent regulating valves can dynamically provide feedback on valve opening status. Furthermore, during the acquisition of multi-source heterogeneous data, physical linkage of key equipment can be achieved, i.e., the primary heating network exchanges heat energy with the secondary network via plate heat exchangers, and the secondary network water flow is driven by a circulating pump.

[0037] In some embodiments, multi-source heterogeneous data can also synchronously integrate key parameters from the PID regulation historical database, including historical valve opening curves, pump frequency regulation records, and temperature or pressure response data under corresponding operating conditions, to output an enhanced dataset that integrates real-time physical parameters and historical PID operating characteristics, including flow rate, pressure, temperature, valve opening, and circulating pump frequency, providing structured input for upper-level data preprocessing.

[0038] like Figure 3 As shown, when acquiring multi-source heterogeneous data, the system can first receive an optimization execution command input by the user to initiate hydraulic balance optimization. Upon receiving the optimization execution command, the system generates a data acquisition request in response. This data acquisition request includes a first acquisition request and a second acquisition request. The first acquisition request is used to acquire physical monitoring data from the heating system. The intelligent data platform can obtain real-time physical data collected by the sensor devices in the environmental perception layer of the heating system by sending the first acquisition request to them.

[0039] The second acquisition request is used to obtain PID control-related data. Specifically, the intelligent data platform can send the second acquisition request to the historical PID regulation database of the environmental perception layer in the heating system to obtain key PID regulation parameters. After obtaining the key PID regulation parameters, historical PID operating characteristics can be generated based on these parameters. By fusing real-time physical data and historical PID operating characteristics, an enhanced dataset is formed, thereby obtaining multi-source heterogeneous data.

[0040] For example, when collecting data, the intelligent data platform can use temperature sensors (1~n), pressure transmitters (1~n), and electromagnetic flow meters (1~n) installed on the user side (A~N) in the environmental perception layer to collect raw data such as flow rate, pressure, temperature, and valve opening in the physical pipeline network in real time, so as to obtain real-time physical data containing the set of raw physical parameters. That is, real-time physical data can include physical parameters such as flow rate, pressure, temperature, valve opening, and pump frequency collected in real time.

[0041] The intelligent data platform can also access a historical database of PID control and extract key PID control parameters, such as PID control curves, from this database. Then, using data statistical tools or feature extraction models, features are extracted from these key PID control parameters to obtain historical PID operating characteristics. These characteristics can include multi-dimensional feature vectors composed of control parameter maxima, minima, control frequency, and other feature values. Finally, these historical PID operating characteristics are fused with real-time physical data to create an enhanced dataset.

[0042] S102. Generate standardized feature vectors based on multi-source heterogeneous data.

[0043] After acquiring multi-source heterogeneous data, the data processing layer can be used to preprocess the data to generate standardized feature vectors. Standardized feature vectors refer to the scaling and centering of each variable, dimension, and column of features in the multi-source heterogeneous data during data processing, so that the data conforms to a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0044] To obtain standardized feature vectors, in some embodiments, when generating standardized feature vectors from multi-source heterogeneous data, the data processing algorithm application of the data processing layer can be called first. This data processing algorithm application can be used to perform outlier cleaning, multi-source spatiotemporal alignment, and feature fusion after receiving the raw data, and use a graph neural network (GNN) to analyze the pipeline topology of the heating system, and finally output a normalized state vector, providing a high-quality input data stream with continuous time and uniform scale for the DRL decision layer.

[0045] Based on the invoked data processing algorithm application, the intelligent data platform can extract heterogeneous sensor data from multi-source heterogeneous data. This heterogeneous sensor data includes user-side data and heat exchange station data. The fused heterogeneous sensor data is then fused through spatiotemporal alignment and normalized using a Z-score standardization algorithm.

[0046] Then, a Graph Neural Network (GNN) is used to analyze the pipe network topology of the heating system. This topology can be used to enable digital twinning and optimization processes. To map the physical pipe network structure to a mathematical topological graph, topological elements such as nodes, edges, adjacency matrices, and global graph features can be defined based on the physical pipe network structure data. Nodes can correspond to node features, i.e., feature vectors constituting the nodes, used to represent physical entities in the pipe network such as pipe intersections, user access points, heat sources, heat exchange stations, pump stations, storage tanks, and sensor installation points. Edges can form edge features, i.e., feature vectors constituting the edges, used to represent physical entities in the pipe network such as pipes, valves, and pumps. The adjacency matrix represents the matrix of connections between nodes. Global graph features can be used to describe the characteristics of the entire system, such as outdoor temperature, total network load, and operating mode.

[0047] Based on the GNN model architecture, a message passing process for the topology is set up, allowing each node to exchange information with its neighboring nodes through edges and update its own state representation. By stacking multiple GNN layers, nodes can receive information from multiple neighboring nodes, thereby capturing a wider range of network dependencies. After parsing the network topology of the heating system, and combining the normalized heterogeneous sensor data with the network topology, a standardized feature vector is generated by adjusting the controller.

[0048] For example, if the data processing layer is used to construct a transformation path from multi-source heterogeneous data to decision input, the intelligent data platform can, based on the data processing algorithms deployed in the data processing layer, first fuse heterogeneous sensor data from the user side and heat exchange station through spatiotemporal alignment, integrating dynamic parameters such as flow rate, pressure, temperature, and valve opening as feature dimensions into the same multi-dimensional feature vector. Then, the Z-score normalization method is used to normalize the dynamic parameters such as flow rate, pressure, and temperature, eliminating dimensional differences. Finally, a graph neural network is used to analyze the pipeline topology, thereby generating normalized feature vectors (NFVs) through the control controller, providing the DRL decision layer with a high-quality input data stream that is temporally continuous and scale-uniform.

[0049] S103. Using a deep reinforcement learning model, perform multi-objective collaborative optimization based on standardized feature vectors to generate output instructions.

[0050] After generating standardized feature vectors, a deep reinforcement learning model can be used to make decisions in the hydraulic balance optimization process. That is, a deep reinforcement learning model is used to perform multi-objective collaborative optimization based on the standardized feature vectors to generate output instructions.

[0051] The deep reinforcement learning model employs either Proximal Policy Optimization (PPO) or Soft Actor-Critic (SAC) algorithms, integrating network topology features to construct a deep reinforcement learning engine. The PPO algorithm, when updating the policy, limits the magnitude of each update through methods such as pruning, ensuring the new policy doesn't deviate too far from the old one, thus achieving stable and efficient training. The SAC algorithm, on the other hand, adds additional policy entropy to the standard goal of maximizing cumulative reward, encouraging the policy to explore further and avoiding premature convergence to local optima.

[0052] Deep reinforcement learning models are also used to model the dynamic characteristics of heating systems using Markov Decision Processes (MDPs). MDPs treat data as having Markov property, meaning the next state of the system depends only on the current state and the current action, and is independent of historical states. The decision process describes how an agent achieves its goal by making a series of decisions in the environment.

[0053] When using Markov decision processes to model the dynamic characteristics of a heating system, the secondary network of the heating system can be regarded as an intelligent agent. By sensing the system status such as temperature and pressure, it can execute control actions such as regulating valves and pump speeds, and obtain feedback information such as energy consumption and comfort from environmental data such as physical network data and user data, thereby learning a strategy that maximizes long-term cumulative rewards.

[0054] The output command is the model output of the deep reinforcement learning model, used to control the operating status of the circulation control equipment in the secondary pipe network. Since the main circulation control equipment in the secondary pipe network is valves and pumps, the output command can include at least one of valve control signals and pump frequency conversion signals. Specifically, the valve control signal controls the opening degree of the valves in the secondary pipe network; the pump frequency conversion signal controls the pumping speed of the pumps.

[0055] For example, after generating standardized feature vectors at the data processing layer, intelligent multi-objective collaborative optimization can be performed through the DRL decision layer to obtain the optimal control strategy. As the core of the entire optimization system, the DRL decision layer takes the preprocessed state vector as input, calculates the optimal control strategy through the PPO or SAC algorithm, and outputs the valve opening adjustment amount (±5%) and the pump frequency adjustment amount (±5Hz).

[0056] In some embodiments, when using a deep reinforcement learning model to perform multi-objective collaborative optimization based on standardized feature vectors, a decision objective can be set first. The decision objective is used to distribute branch flow by adjusting the real-time opening of the user-side regulating valve and to coordinate the control of the total flow and pressure gradient of the pipeline network by optimizing the rotational speed of the secondary pipeline circulation pump, thereby achieving system-level hydraulic balance.

[0057] The standardized feature vectors are then input into a deep reinforcement learning model, and the dynamic characteristics of the heating system are modeled using a Markov decision process. Then, a proximal policy optimization or flexible actor-critic algorithm is employed to calculate the optimal control strategy based on the standardized feature vectors and dynamic characteristics, thereby generating output commands based on the optimal control strategy.

[0058] For example, the DRL decision layer, as the intelligent hub of the architecture, can use proximal policy optimization (PPO) or flexible actor-commentator (SAC) algorithms to build a deep reinforcement learning engine, and model the dynamic characteristics of the heating system through Markov decision process (MDP) to achieve dual-objective collaborative optimization.

[0059] Among them, the decision objectives of the dual-objective collaborative optimization are to adjust the opening of the intelligent regulating valves on the user side (A~N) in real time to achieve precise distribution of branch flow, and to optimize the speed of the secondary network circulation pump, and to coordinate the control of the total flow and pressure gradient of the pipeline network to achieve system-level hydraulic balance.

[0060] Through dual-objective collaborative optimization, the DRL decision layer can use a deep reinforcement learning model to output instructions, including valve control signals and pump frequency conversion signals. The valve control signals generate valve opening instructions for each branch valve, used to adjust the intelligent regulating valves (1~n); the pump frequency conversion signals output the pump speed setpoint, used to control the circulating pump frequency converter.

[0061] S104. Send the output command to the target controlled device in the heating system so that the target controlled device can dynamically adjust the system operating status according to the output command.

[0062] After obtaining the output instructions from the deep reinforcement learning model, the control layer can send these instructions to the target controlled equipment in the heating system, enabling the target controlled equipment to dynamically adjust the system's operating state according to the output instructions. The target controlled equipment refers to the cyclic control devices associated with the output instructions in the secondary pipe network, such as valves and pumps. The operating state includes at least one of the following: branch flow distribution, total flow rate of the secondary pipe network versus system pressure difference, and global pipe network impedance curve.

[0063] To perform optimized control, in some embodiments, the execution control layer converts the output command corresponding to the optimal decision result into physical operation signals to drive intelligent regulating valves and variable frequency pumps to perform precise adjustments. Simultaneously, the execution effect is monitored in real time and compared with the PID control curve for analysis. Therefore, when the execution layer sends the output command to the target controlled device in the heating system so that the target controlled device dynamically adjusts the system operating state according to the output command, the output command can first be sent to the execution control layer of the heating system. The execution control layer then converts the output command into physical operation signals, which include valve control signals and pump frequency conversion signals. These physical operation signals are then sent to the target controlled device.

[0064] For example, the execution control layer can be used to achieve deterministic conversion of digital commands into physical actions. To meet the valve execution requirements, the opening command output by the DRL model can be converted into the valve position stroke of the regulating valve. For instance, any opening value within the 0-100% range can be converted into a drive current within the 4-20mA range, used to dynamically change the branch flow distribution through the valve. Similarly, to meet the pump execution requirements, a frequency converter can be used to convert the speed command into a motor stator frequency within the 30-50Hz range, used to regulate the total flow of the secondary network and the system pressure difference.

[0065] During instruction conversion, multiple cyclic control devices can be coupled for energy efficiency to minimize the total system power consumption through their coordinated operation. For example, by adjusting the local hydraulic characteristics through valve opening and simultaneously changing the global pipeline impedance curve through pump speed regulation, the valves and pumps can operate in coordination to minimize the total system power consumption.

[0066] S105. Based on real-time collected pipeline system data, dynamically evaluate system performance in a digital twin environment, and continuously iterate and optimize the model parameters of the deep reinforcement learning model through online training based on the system performance evaluation results.

[0067] After sending output commands to the target controlled equipment in the heating system, the intelligent data platform can continuously monitor the system's operational status after responding to the output commands through a feedback optimization layer. The system's operational status can be comprehensively evaluated using pipeline system data, which includes multi-source physical data collected from the heating system after the target controlled equipment executes the output commands. Therefore, system performance can be dynamically evaluated in a digital twin environment based on real-time collected pipeline system data.

[0068] A digital twin environment is a comprehensive application platform that runs in a virtual computer environment, integrating data, models, algorithms, interaction, and visualization functions. Through real-time or near-real-time virtual-real interaction, it dynamically and faithfully mirrors, simulates, analyzes, predicts, and optimizes the entire lifecycle of the physical entity corresponding to the system model in a virtual space.

[0069] In some embodiments, when dynamically evaluating system performance in a digital twin environment based on real-time acquired pipeline system data, the system can first receive execution feedback signals from the target controlled device in response to physical operation signals, and obtain the cumulative duration for which physical operation signals were sent to the target controlled device. Upon receiving the execution feedback signal, the pipeline system data can be acquired in response. Alternatively, the pipeline system data can be acquired when the cumulative duration reaches a preset duration threshold. Control result parameters are then extracted from the pipeline system data, and system performance evaluation results are generated by comparing the control result parameters with key PID control parameters.

[0070] For example, after modeling the dynamic characteristics of a heating system, a core quintuple can be defined, including the state space (S), action space (A), transition function (T), reward function (R), and discount factor (γ). The optimal policy is found by maximizing the expected cumulative discount reward. This can be achieved by combining Flexible Actor-Critic (SAC) and Proximal Policy Optimization (PPO) algorithms, allowing the agent to learn the policy through extensive trial and error interactions with the digital twin environment. Then, within the digital twin, various historical or simulated scenarios are used as training environments to run the RL algorithm for a preset number of steps, learning an initial policy. Independent test scenarios are then set up within the digital twin to evaluate the performance, robustness, and safety of the learned policy.

[0071] Then, through deployment and online learning, operating in shadow mode, the learned strategies are initially run in parallel with existing control strategies, and the results are compared in the digital twin to increase decision-making confidence. Actions, such as new pump speed setpoints, are selected based on strategy π and then distributed to the circulation control equipment. Furthermore, strategies are fine-tuned using new data from system operation, employing online RL or periodic offline updates, to adapt to changes in the pipeline network.

[0072] In some embodiments, when dynamically evaluating system performance in a digital twin environment based on real-time acquired pipeline system data, a hydraulic-thermal coupling model can be constructed based on the pipeline topology and key parameters of PID control. The boundary conditions of this hydraulic-thermal coupling model are multi-source heterogeneous data. A digital twin environment is then constructed based on this model, and alarm thresholds are set for the digital twin environment. When the multi-source heterogeneous data meets the alarm thresholds, simulation is automatically triggered to obtain simulation results, which are then used to generate system performance evaluation results.

[0073] For example, to perform feedback optimization, a hydrothermal (HT) coupled model can be constructed using the multi-species extension (EPANET-MSX) or EPANET external coupling strategy, and a static twin can be obtained through model calibration of the hydrothermal coupled model.

[0074] Then, through real-time data fusion, hydraulic data such as pump status, valve status, flow meter, and pressure data, as well as thermal data such as heat source outlet temperature, key node temperature sensors, ambient temperature, and user heat load are integrated into the digital twin environment.

[0075] Then, for the digital twin environment, the latest sensor data is used as boundary conditions to drive the HT coupled model for simulation at a real-time or near-real-time rate (e.g., every 5 minutes). This allows for the deduction of the entire system's full-view state and the acquisition of simulation results, such as temperature field and heat loss distribution. The results can also be displayed in a visual format, such as displaying the real-time temperature distribution and heat loss intensity of the pipeline network on a map in the form of a heat map.

[0076] Based on the alarm thresholds set in the digital twin environment, combined hydraulic and thermal diagnostics can be performed, including identifying abnormal areas, such as an abnormally low temperature in a certain area, which may be due to pipe leaks or insulation failure. Furthermore, by assessing the thermal equilibrium state, the thermal imbalance of each branch of the system can be analyzed, providing a basis for adjusting the balancing valves.

[0077] The simulation results obtained can be used to generate system performance evaluation results for prediction and optimization. For example, by combining weather forecasts and user behavior, future heat loads can be predicted, and the output of heat sources and pump scheduling strategies can be optimized to achieve on-demand heating and load prediction and scheduling. Furthermore, by predicting delay times, the hydraulic and thermal delay times from the heat source to the end user can be accurately calculated to optimize early start-up and shutdown times.

[0078] It is evident that the feedback optimization layer can build a digital twin environment based on the EPANET hydraulic-thermal coupling model, inject fault scenarios to enhance the robustness of DRL, and update the strategy parameters online based on the feedback from the physical system, forming a self-evolving closed loop of "perception-decision-execution-evaluation".

[0079] After dynamically evaluating the system performance based on real-time collected pipeline system data in a digital twin environment, online training can be continuously iterated based on the system performance evaluation results obtained from the dynamic evaluation to adjust and optimize the model parameters of the deep reinforcement learning model.

[0080] In some embodiments, when performing continuous iterative optimization of the deep reinforcement learning model parameters through online training based on system performance evaluation results, the physical parameters of the control target can be obtained first, and then the real-time monitoring physical parameters can be extracted from the pipeline system data. Then, by comparing the control target physical parameters and the real-time monitoring physical parameters, the model output error is calculated, and the model parameters of the deep reinforcement learning model are continuously iteratively optimized by backpropagating the model output error.

[0081] For example, by acquiring the physical parameters of the control target, the ideal state that the agent needs to track can be determined. These parameters could include the target return water temperature and the target pressure range. The control target physical parameters can be encoded in the reward function or input as part of the state to the agent. Additionally, real-time monitoring physical parameters can be extracted from pipeline system data as the current state perceived by the agent. These real-time monitoring physical parameters can include the return water temperature of each end branch, the pressure at key nodes, the main pipe flow rate, the current valve opening, and the ambient temperature.

[0082] By comparing the physical parameters of the control target with the real-time monitored physical parameters, the model output error is calculated to reflect the gap between the current policy's performance in the environment and the target. For example, the agent calculates the error based on the current state s. t The action of adjusting the opening of the output valve a t Actions are performed in a digital twin environment or a real system, and the environment transitions to a new state. t+1 and return an instant reward r t Among them, the reward r t It can serve as a quantitative result comparing the target with real-time parameters. Furthermore, the temporal difference error within the DRL algorithm can be used to measure the accuracy of the current value function estimation, serving as a basis for policy optimization.

[0083] The DRL model parameters are then iteratively optimized by backpropagating the model output error. This involves sampling from experience, calculating the loss, and updating the network weights through backpropagation. During backpropagation, the loss value can be calculated based on the DRL model's loss function, such as the loss for the actor and critic networks in DDPG. Then, through backpropagation, the gradient of the loss with respect to the network parameters is calculated using automatic differentiation. Finally, optimizers such as Adam are used to update the DRL model parameters.

[0084] Therefore, the feedback optimization layer can dynamically evaluate system performance in a digital twin environment, continuously iteratively optimize DRL model parameters through online training, and form a closed-loop control loop of "perception-decision-execution-evaluation". Each layer is tightly coupled through data flow and feedback mechanisms, enabling the system's autonomous evolution and continuous optimization.

[0085] By applying the technical solutions of the above embodiments, the aforementioned deep reinforcement learning-based secondary pipe network hydraulic balance optimization method can transform the linkage relationship of multiple controlled devices in the secondary pipe network into state transition probabilities, and use graph neural networks to encode topological features for multi-objective collaborative optimization to generate a globally optimal solution. Furthermore, it can dynamically evaluate system performance in a digital twin environment, continuously iteratively optimize model parameters through online training, and improve the hydraulic balance optimization effect of the secondary pipe network.

[0086] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a method for optimizing the hydraulic balance of secondary pipe networks based on deep reinforcement learning. The difference between this method and the above embodiments is that it can generate a model training dataset based on PID control and train a deep reinforcement learning model. Figure 4 As shown, the method includes: S201. Obtain the model training dataset; S202. Input the model training dataset into the deep reinforcement learning model to obtain the training output instructions output by the deep reinforcement learning model. S203. Extract training instruction parameters from training output instructions; S204. Calculate the model training loss by comparing the training instruction parameters and parameter labels; S205. If the model training loss is greater than the preset loss threshold, adjust the model parameters of the deep reinforcement learning model according to the model training loss. S206. If the model training loss is less than or equal to the preset loss threshold, output the model parameters of the deep reinforcement learning model.

[0087] To facilitate continuous iterative online training, a model training dataset can be acquired before training the model. This dataset includes labeled sample data, which comprises heterogeneous multi-source data extracted from a historical PID control database. The parameter labels are set based on key PID control parameters. A sizable training dataset can be quickly generated using key PID control parameters from the historical database.

[0088] The model training dataset is then split into training and validation sets. The training set is input into the deep reinforcement learning model to obtain the training output commands. Training command parameters, such as valve opening and pump cycle frequency, are extracted from the training output commands and compared with the validation set to calculate the model training loss.

[0089] Regarding the model training loss, a preset loss threshold can be used to determine whether the model has been trained to convergence. When the model training loss is greater than the preset loss threshold, it means that the current model has not yet been trained to convergence. Therefore, the model parameters of the deep reinforcement learning model can be adjusted according to the model training loss, and iterative training can be performed until the model training loss is less than or equal to the preset loss threshold, or the preset maximum number of iterations is reached, at which point the model parameters of the deep reinforcement learning model are output.

[0090] In some embodiments, as a specific implementation of the deep reinforcement learning-based secondary pipe network hydraulic balance optimization method described in the above embodiments, some embodiments of this application also provide a deep reinforcement learning-based secondary pipe network hydraulic balance optimization system, such as... Figure 5 As shown, the system includes: The environmental perception layer is used to acquire multi-source heterogeneous data, which includes at least one of the following: flow data, pressure data, temperature data, valve opening status, and key parameters of PID regulation collected from the user side of the secondary pipeline network and heat exchange station in the heating system. The data processing layer is used to generate standardized feature vectors based on the multi-source heterogeneous data; A deep reinforcement learning decision layer is used to perform multi-objective collaborative optimization based on the standardized feature vector using a deep reinforcement learning model to generate output instructions; the deep reinforcement learning model is a deep reinforcement learning engine constructed using proximal policy optimization or a flexible actor-critic algorithm; the deep reinforcement learning model is used to model the dynamic characteristics of the heating system through a Markov decision process; the output instructions include at least one of valve control signals and pump frequency conversion signals; An execution control layer is used to send the output command to the target controlled equipment in the heating system, so that the target controlled equipment dynamically adjusts the system operating state according to the output command; the operating state includes at least one of branch flow distribution, total flow of secondary pipe network and system pressure difference, and global pipe network impedance curve. The feedback optimization layer is used to dynamically evaluate system performance in a digital twin environment based on real-time collected pipeline system data, and to continuously iteratively optimize the model parameters of the deep reinforcement learning model through online training based on the system performance evaluation results; the pipeline system data includes multi-source physical data collected by the target controlled device for the heating system after executing the output command.

[0091] By applying the technical solutions of the above embodiments, the aforementioned secondary pipe network hydraulic balance optimization system based on deep reinforcement learning can first acquire multi-source heterogeneous data and generate standardized feature vectors. Then, a deep reinforcement learning model is used to perform multi-objective collaborative optimization to generate output instructions. These output instructions are then sent to the target controlled equipment in the heating system to dynamically adjust the system's operating state. Furthermore, based on real-time collected pipe network system data, the system performance is dynamically evaluated in a digital twin environment, and online training is used to continuously iteratively optimize model parameters. The system can transform the linkage relationships of multiple controlled devices in the secondary pipe network into state transition probabilities and use graph neural networks to encode topological features for multi-objective collaborative optimization, generating a globally optimal solution. It can also dynamically evaluate system performance in a digital twin environment and continuously iteratively optimize model parameters through online training, thereby improving the hydraulic balance optimization effect of the secondary pipe network.

[0092] It should be noted that other corresponding descriptions of the functional units involved in the secondary pipe network hydraulic balance optimization system based on deep reinforcement learning provided in the embodiments of this application can be found in the corresponding descriptions in the secondary pipe network hydraulic balance optimization method based on deep reinforcement learning provided in the above embodiments, and will not be repeated here.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A method for optimizing the hydraulic balance of a secondary pipe network based on deep reinforcement learning, characterized in that, The method includes: Acquire multi-source heterogeneous data, which includes at least one of the following: flow data, pressure data, temperature data, valve opening status, and key parameters of PID regulation collected from the user side of the secondary pipeline network and heat exchange station in the heating system. Generate standardized feature vectors based on the multi-source heterogeneous data; A deep reinforcement learning model is used to perform multi-objective collaborative optimization based on the standardized feature vectors to generate output instructions; the deep reinforcement learning model is a deep reinforcement learning engine constructed by fusing pipeline topology features using proximal policy optimization or flexible actor-critic algorithm; the deep reinforcement learning model is used to model the dynamic characteristics of the heating system through Markov decision process; the output instructions include at least one of valve control signals and pump frequency conversion signals; The output command is sent to the target controlled device in the heating system so that the target controlled device can dynamically adjust the system operating status according to the output command; the operating status includes at least one of the following: branch flow distribution, total flow of secondary pipe network and system pressure difference, and global pipe network impedance curve. The system performance is dynamically evaluated in a digital twin environment based on real-time collected pipeline system data, and the model parameters of the deep reinforcement learning model are continuously iteratively optimized through online training based on the system performance evaluation results; the pipeline system data includes multi-source physical data collected by the target controlled equipment for the heating system after executing the output command.

2. The method according to claim 1, characterized in that, Acquiring multi-source heterogeneous data, including: In response to the optimized execution instruction, a data acquisition request is generated, the data acquisition request including a first acquisition request and a second acquisition request; The first acquisition request is sent to the sensor device of the environmental perception layer in the heating system to obtain the real-time physical data collected by the sensor device; The second acquisition request is sent to the PID adjustment history database of the environmental perception layer in the heating system to obtain the key parameters of the PID adjustment. Historical PID operating characteristics are generated based on the aforementioned key PID adjustment parameters; The multi-source heterogeneous data is obtained by fusing the real-time physical data and the historical PID operating characteristics.

3. The method according to claim 1, characterized in that, Generating standardized feature vectors based on the multi-source heterogeneous data includes: Sensor heterogeneous data is extracted from the multi-source heterogeneous data; the sensor heterogeneous data includes user-side data and heat exchange station data. The heterogeneous sensor data is fused through spatiotemporal alignment; The Z-value normalization algorithm is used to normalize the fused heterogeneous sensor data; The topology of the heating system's pipe network was analyzed using a graph neural network. By combining the normalized heterogeneous sensor data and the pipeline topology, the standardized feature vector is generated by adjusting the controller.

4. The method according to claim 1, characterized in that, Using a deep reinforcement learning model, multi-objective collaborative optimization is performed based on the standardized feature vectors to generate output instructions, including: The decision objective is set to distribute the branch flow by adjusting the real-time opening of the user-side regulating valve and to coordinate the control of the total flow and pressure gradient of the pipeline network by optimizing the speed of the secondary pipeline circulation pump, so as to achieve system-level hydraulic balance. The standardized feature vector is input into the deep reinforcement learning model; The dynamic characteristics of the heating system are modeled using a Markov decision process. The optimal control strategy is calculated based on the standardized feature vector and the dynamic characteristics using either proximal strategy optimization or a flexible actor critic algorithm. The output command is generated based on the optimal control strategy.

5. The method according to claim 1, characterized in that, Sending the output command to the target controlled device in the heating system, so that the target controlled device dynamically adjusts the system operating state according to the output command, includes: The output command is sent to the execution control layer of the heating system; The execution control layer converts the output command into physical operation signals, which include valve control signals and water pump frequency conversion signals. The physical operation signal is sent to the target controlled device.

6. The method according to claim 5, characterized in that, Dynamically evaluate system performance in a digital twin environment based on real-time acquired pipeline system data, including: Receive the execution receipt signal fed back by the target controlled device in response to the physical operation signal; Obtain the cumulative duration for which the physical operation signal is sent to the target controlled device; In response to receiving the execution receipt signal, or when the accumulated time reaches a preset time threshold, the pipeline system data is acquired; Extract control result parameters from the pipeline system data; By comparing the control result parameters with the key PID adjustment parameters, a system performance evaluation result is generated.

7. The method according to claim 1, characterized in that, Dynamically evaluate system performance in a digital twin environment based on real-time acquired pipeline system data, including: A hydraulic-thermal coupling model is constructed based on the pipeline network topology and the key parameters of the PID regulation; the boundary conditions of the hydraulic-thermal coupling model are the multi-source heterogeneous data. A digital twin environment is constructed based on the aforementioned hydraulic-thermal coupling model; Set the alarm threshold for the digital twin environment; When the multi-source heterogeneous data meets the alarm threshold, simulation is automatically triggered to obtain simulation result information; The system performance evaluation results are generated based on the simulation results.

8. The method according to claim 1, characterized in that, Based on the system performance evaluation results, the model parameters of the deep reinforcement learning model are continuously and iteratively optimized through online training, including: Obtain the physical parameters of the control target; Extract real-time monitoring physical parameters from the pipeline system data; The model output error is calculated by comparing the physical parameters of the control target with the real-time monitored physical parameters. The model parameters of the deep reinforcement learning model are continuously optimized iteratively by backpropagating the model output error.

9. The method according to claim 1, characterized in that, The method further includes: Obtain a model training dataset, which includes sample data with parameter labels; the sample data includes heterogeneous multi-source sample data extracted from a PID regulation history database; the parameter labels are set according to the key parameters of the PID regulation. Input the training dataset of the model into the deep reinforcement learning model to obtain the training output instructions output by the deep reinforcement learning model; Extract training instruction parameters from the training output instructions; The model training loss is calculated by comparing the training instruction parameters and the parameter labels. If the model training loss is greater than a preset loss threshold, the model parameters of the deep reinforcement learning model are adjusted according to the model training loss. If the training loss of the model is less than or equal to a preset loss threshold, the model parameters of the deep reinforcement learning model are output.

10. A secondary pipe network hydraulic balance optimization system based on deep reinforcement learning, characterized in that, The system includes: The environmental perception layer is used to acquire multi-source heterogeneous data, which includes at least one of the following: flow data, pressure data, temperature data, valve opening status, and key parameters of PID regulation collected from the user side of the secondary pipeline network and heat exchange station in the heating system. The data processing layer is used to generate standardized feature vectors based on the multi-source heterogeneous data; A deep reinforcement learning decision layer is used to perform multi-objective collaborative optimization based on the standardized feature vector using a deep reinforcement learning model to generate output instructions; the deep reinforcement learning model is a deep reinforcement learning engine constructed by fusing network topology features using proximal policy optimization or a flexible actor-critic algorithm; the deep reinforcement learning model is used to model the dynamic characteristics of the heating system through a Markov decision process; the output instructions include at least one of valve control signals and pump frequency conversion signals. An execution control layer is used to send the output command to the target controlled equipment in the heating system, so that the target controlled equipment dynamically adjusts the system operating state according to the output command; the operating state includes at least one of branch flow distribution, total flow of secondary pipe network and system pressure difference, and global pipe network impedance curve. The feedback optimization layer is used to dynamically evaluate system performance in a digital twin environment based on real-time collected pipeline system data, and to continuously iteratively optimize the model parameters of the deep reinforcement learning model through online training based on the system performance evaluation results; the pipeline system data includes multi-source physical data collected by the target controlled device for the heating system after executing the output command.

Images