Hydraulic power plant equipment intelligent operation auxiliary decision-making method and system based on reinforcement learning

Abstract

The invention discloses a hydraulic power plant equipment intelligent operation auxiliary decision-making method and system based on reinforcement learning, and belongs to the technical field of hydraulic power plant intelligent operation and management. A state compression and abstraction method based on reinforcement learning is adopted, multi-source heterogeneous data is combined, and a real-time data processing and dynamic adjustment mechanism is adopted; the real-time performance and the expandability of the model are enhanced; establishing a dynamic mapping relation between a target variable and a control variable by using a deep reinforcement learning and multi-step control optimization method; the generalization ability of the model under different scenes and targets is improved through knowledge base construction and knowledge migration technologies; and through real-time data acquisition, intelligent analysis and decision support, the operation efficiency, safety and reliability of the hydraulic power plant are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent operation and management of hydropower plants, and in particular to a method and system for intelligent operation decision-making assistance of hydropower plant equipment based on reinforcement learning. Background Art

[0002] With the development of the power industry, hydropower plants, as important energy production units, are crucial to the stability of the entire power system in terms of operating efficiency and safety. Traditional hydropower plant operation and management relies on manual experience and fixed processes, which are inefficient, slow to respond, and difficult to adapt to complex and changing operating environments. In traditional hydropower plant operation and management, the following problems exist: (1) The state space dimension explodes, making it difficult to effectively process and analyze large amounts of data. (2) Data quality varies, utilization is low, and it is difficult to form effective decision support. (3) There is a lack of real-time data processing and dynamic adjustment mechanisms, and the real-time and scalability of the model are insufficient. (4) There is a lack of effective dynamic mapping relationships between target variables and control variables. (5) The model has insufficient generalization capabilities and is difficult to adapt to different scenarios and goals. Therefore, it is urgent to design a technology for intelligent operation and decision-making assistance for hydropower plant equipment to solve at least one of the problems existing in traditional hydropower plant operation and management. Summary of the invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide a hydropower plant equipment intelligent operation auxiliary decision-making technology based on reinforcement learning to solve the problems existing in the prior art in the operation management of hydropower plant equipment.

[0004] In order to achieve the above technical objectives, the present application provides a method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning, comprising the following steps:

[0005] Collect relevant operation data of the hydropower station, perform data preprocessing, generate data sets, and obtain abnormal state attributes, where the state attribute set includes equipment delay time, abnormal operating temperature, cooling water pressure, and related equipment status information during the operation of the hydropower unit;

[0006] The intelligent agent is set to perceive the abnormal state attributes. According to the fluctuation degree of normal operating parameters, the decision-making behavior of the intelligent agent that can achieve the maximum return under various abnormal states of the hydropower unit is screened. By setting up a reinforcement learning mechanism, the intelligent agent is controlled to judge whether there is any abnormality in the actual operating state of the current hydropower unit.

[0007] Preferably, in the process of obtaining relevant operating data of the hydropower station, a data access service is constructed by combining direct database interaction and standard industrial protocol communication to obtain relevant operating data of the hydropower station, which is used to ensure real-time access to online monitoring of original waveform data, characteristic spectrum structure and characteristic value parameters.

[0008] Preferably, in the process of data preprocessing, data preprocessing is performed by data completion and error correction, data cleaning and data fusion, wherein:

[0009] Data completion and error correction, including:

[0010] Missing value processing: For analog quantities, linear interpolation is used for processing, and for switch quantities, the previous value method is used to fill in the missing values;

[0011] Outlier processing: The mean and standard deviation method of the upper and lower limit methods are used to process outliers;

[0012] Data cleaning, including: cleaning of erroneous data, cleaning of contaminated data and cleaning of redundant data;

[0013] Data fusion, including time synchronization technology, working condition synchronization technology and event synchronization technology.

[0014] Preferably, in the process of obtaining the degree of fluctuation, based on the parameter x0 of the normal operation of the hydropower unit in the initial state, the degree of fluctuation is obtained according to the actual parameter information x of the hydropower unit after running for t time, which is expressed as:

[0015] Preferably, in the process of setting up the agent, the prediction function of the agent is expressed as:

[0016]

[0017] In the formula, m represents the agent, [σ,σ+1] represents each learning iteration process, and x m (t) is the iterative function of the abnormal state attributes of the hydropower unit.

[0018] Preferably, in the process of setting the mandatory learning mechanism, the mandatory learning mechanism includes:

[0019] The agent decision strategy of balancing local search and global search is adopted to improve the learning effect of the agent in the virtual environment;

[0020] The fully trained intelligent agent is applied to the actual production process to make autonomous decisions based on the current operating conditions of the hydropower equipment, and to optimize the uncertain abnormal factors in the hydropower units through intelligent decision-making.

[0021] Preferably, in the process of setting the forced learning mechanism, the forced learning mechanism is optimized by the updating method of Speedy Q-learning, wherein the Speedy Q-learning algorithm performs partial sampling on the subspace of the next state joint action space and obtains the maximum Q value.

[0022] Preferably, in the process of setting up the intelligent agent, data conversion instructions are set for the intelligent agent using RDF strategy, the status data file received in WVD format is made into an RDF file, and then the WVD-RDF data is used to integrate the corresponding parameter information into the form of SVG graphics, and the data is converted into a matrix form with elements as the basic units after EEMD decomposition.

[0023] The present invention discloses a hydropower plant equipment intelligent operation auxiliary decision-making system based on reinforcement learning, which is used for the above-mentioned hydropower plant equipment intelligent operation auxiliary decision-making method based on reinforcement learning, comprising:

[0024] The data processing module is used to collect the relevant operation data of the hydropower station, perform data preprocessing, generate data sets, and obtain abnormal state attributes, where the state attribute set includes equipment delay time, abnormal operation temperature, cooling water pressure, and related equipment status information during the operation of the hydropower unit;

[0025] The prediction module is set in the intelligent agent to assist the intelligent agent in perceiving the attributes of abnormal states. According to the degree of fluctuation of normal operating parameters, the intelligent agent is screened to make decisions that can achieve the maximum return under various abnormal states of the hydropower unit. By setting up a reinforcement learning mechanism, the intelligent agent is controlled to judge whether there is any abnormality in the actual operating state of the current hydropower unit.

[0026] Preferably, the system adopts a microservice layered architecture design method to build a remote service publishing system based on a restful architecture. When adding, reading, modifying, and deleting data, the service requester places the requested data in the HTTP document body and sends a request to the server through the first URI. After the server completes processing according to the method parameters, it returns the result to the service requester in the format of XML, hypertext markup language, or lightweight data through the second URI.

[0027] The present invention discloses the following technical effects:

[0028] (1) Efficiency of data access and management: Through hybrid access, combined with direct database connection and standard industrial protocol communication, real-time and stable access to hydropower station operation data is achieved, including the original waveform data, characteristic spectrum structure and characteristic value parameters of the online monitoring system, thereby improving the real-time and accuracy of the data. In addition, data completion and error correction technologies, such as linear interpolation and mean standard deviation method, effectively handle missing values ​​and outliers, and improve data quality.

[0029] (2) Advanced intelligent state recognition: By adopting a multi-agent reinforcement learning mechanism, combined with Q-Learning and W-Learning, the agent can autonomously learn and optimize decision-making behavior to adapt to the real-time operating status of the hydropower unit. This method reduces the amount of calculation in the learning phase, improves the convergence speed of the algorithm, and makes the state recognition of the hydropower unit more accurate and efficient.

[0030] (3) Optimization of abnormal state identification process: By converting the data in the form of Wigner-Ville distribution (WVD) into RDF files and combining them with SVG graphics, the abnormal state of the hydropower unit can be quickly identified and the fault location can be accurately located. At the same time, the data storage mode in the form of EEMD decomposition and element-based unit matrix is ​​adopted to improve the efficiency of data processing and the economy of storage.

[0031] (4) In-depth analysis of special items and key indicators: The intelligent report generation model and multi-parameter state quantity conversion mode provided by the system meet the needs of various parameter statistics and automatically provide qualitative evaluation results through logical judgment rules. In addition, the automatic alarm and early warning method of equipment failure based on status and performance realizes intelligent analysis of system abnormalities through dynamic threshold analysis and similar influencing factor trend analysis.

[0032] (5) Comprehensiveness of equipment status evaluation: Through the formulation of equipment condition evaluation standards and health status prediction standards, an intelligent equipment condition analysis module was developed to achieve accurate assessment of the overall health status level of the equipment. This provides a scientific decision-making basis for the development of equipment maintenance work.

[0033] (6) Innovation in microservice design: The remote service publishing system based on the RESTful architecture is adopted to achieve comprehensive microservices in the application architecture. This design improves the service response speed and system maintainability, while reducing the coupling between services. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is the maintenance and control process of hydropower units based on multi-agents described in the present invention;

[0036] Figure 2 is the response model of the RESTful Web service of the present invention;

[0037] Figure 3 This is the platform bus message routing and service management described in the present invention. DETAILED DESCRIPTION

[0038] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0039] The present invention provides a technology for intelligent operation decision-making assistance of hydropower plant equipment based on reinforcement learning, which specifically includes the following contents:

[0040] 1. Water machine monitoring data access and underlying data management:

[0041] Responsible for collecting data from various equipment and systems in the hydropower plant in real time, and performing necessary preprocessing, such as denoising and normalization, to improve data quality.

[0042] 1) Data access:

[0043] The real-time access service for hydropower station-related operating data mainly covers information systems closely related to the analysis of hydropower units, including computer monitoring systems, dam monitoring systems, water regulation and water regime systems, and online monitoring systems, among which the online monitoring system is the core business of this system access.

[0044] For data access services, a hybrid access method will be used to achieve stable data access, and data access services will be built by combining database direct interaction and standard industrial protocol communication. For the system data currently integrated in the hydropower company's integrated platform, real-time acquisition of system data will be carried out through database direct connection. For the online monitoring system, it is planned to build a high-speed acquisition service based on the manufacturer's communication protocol to ensure real-time access to online monitoring original waveform data, characteristic spectrum structure and characteristic value parameters.

[0045] 2) Data completion and error correction:

[0046] During data preprocessing, functions such as quality analysis, format unification, missing value processing, and outlier processing need to be considered.

[0047] (1) Missing value processing:

[0048] Simulation: Linear interpolation method, using programming tools to identify the location of missing values ​​in the data set, and then analyzing the distribution of missing values ​​and their impact on data analysis. If the missing value is between two valid data points and there is an obvious linear relationship between the two, linear interpolation is used.

[0049] Switching quantity: Switching quantity uses the previous value method to fill missing values. When dealing with time series data or any data set such as switching quantity (i.e., discrete values ​​such as 0 and 1, which are usually used to represent a state of on or off), missing values ​​are often encountered. Missing values ​​may be caused by equipment failure, data transmission interruption, or errors in the data recording process.

[0050] (2) Outlier processing:

[0051] The mean and standard deviation method with upper and lower limit methods is used. The data follows a normal distribution, and data points outside the mean plus or minus n times the standard deviation (n is usually 2 or 3) are considered outliers.

[0052] 3) Data cleaning:

[0053] (1) Error data cleaning:

[0054] Error detection: Data cleaning should be able to use error detection models to automatically detect status data errors caused by sensor damage (including sensor failure, sensor power failure, sensor wiring failure, etc.), environmental interference (including electromagnetic, dust, temperature and humidity, etc.), acquisition equipment failure, communication anomalies, and communication point errors.

[0055] The error detection model is an error detection model established for the characteristics of each physical state (electrical state, mechanical state, temperature state, etc.) and its error mode (deviation, mutation, jitter, etc.).

[0056] Fault tolerance: Redundant data resources are used to reconstruct the cleaned erroneous state data to ensure that subsequent applications are not affected. Redundant state data includes: state data of repeated measurements of multiple systems, high time density data, and state data calculated based on associated data using motion characteristic formulas or historical signature tables.

[0057] Report: The cleansed data should be reported to inform subsequent application systems and facilitate maintenance.

[0058] (2) Polluted data cleaning:

[0059] Contaminated data cleaning should be able to remove noise data coupled in the real state data and even separate multiple signals coupled together.

[0060] The methods for cleaning contaminated data include traditional mathematical filtering methods (wash out filter) and local methods based on signal pattern recognition (pattern recognition).

[0061] In order to facilitate correlation analysis, the cleaning of contaminated data should not cause obvious delay (phase lag) of state quantities.

[0062] (3) Redundant data cleaning:

[0063] Redundant data includes repeated measurement data and steady-state process data, etc. Redundant data cleaning is to discard "useless" data on the basis of saving all required data, which can not only reduce storage space but also reduce analysis workload.

[0064] Repeated measurement data: measurement data of the same physical state from multiple systems. In order to ensure the uniformity of input data for subsequent applications, one of them is selected or integrated through the model and mutual backup is achieved.

[0065] Steady-state process data: The key is intelligent automatic perception and intelligent storage strategy through automatic identification of state signal waveform characteristics, automatic perception of changes in working conditions and environmental related state data.

[0066] Data cleaning includes cleaning of erroneous data, cleaning of polluted data and cleaning of redundant data. In order to prevent cleaning errors or transitions, the raw data that has not been cleaned can be cached for a period of time (such as 3 days).

[0067] 4) Data Fusion:

[0068] A hydropower unit is a large, complex system with strong coupling. A large number of examples show that when a fault occurs in one device, the cause of the fault may be in other related devices or environments. In order to achieve accurate fault diagnosis, the state big data must be associated and integrated.

[0069] (1) Time synchronization technology:

[0070] Time synchronization (i.e. aligning the time stamp of data) is the basis of data association. At present, hardware or software synchronization is adopted to ensure the consistency of the clocks of each system. In addition, it is also required that each system should not produce serious delays when performing signal filtering (or data filtering). Therefore, it is necessary to study intelligent filtering technology of various states to replace conventional bandpass filtering technology to ensure no delay in filtering and achieve better filtering effect.

[0071] (2) Working condition synchronization technology:

[0072] Operating condition synchronization is to automatically identify the operating conditions and operating environment of the units, main transformers, etc. in real time online on the basis of the time scale, add operating conditions, operating condition parameters, and environmental parameter labels to the state process, and at the same time, synchronously record the status data of each system of important operation processes and important operating conditions, which is an important basis for conducting correlation analysis.

[0073] (3) Event synchronization technology:

[0074] Event synchronization means that after real-time identification of abnormal events of the unit, the data of each access system can be collected and recorded synchronously, and the integrated data server can store data synchronously.

[0075] 2. Hydropower unit state recognition mechanism based on multi-agent reinforcement learning:

[0076] Combine Q-Learning with W-Learning. In the action selection and execution stage, actions are selected independently according to the learned strategy to effectively reduce the amount of calculation in the learning stage. First, build a reinforcement learning agent and use the hydropower unit as the learning environment of the reinforcement learning agent. The agent m perceives the abnormal state attribute s during the operation of the unit. The state attribute set includes the equipment delay time s. t , abnormal operating temperature c, cooling water pressure p and other related equipment status information during the operation of the hydropower unit. On this basis, the intelligent agent selects the maintenance action A according to the corresponding abnormal state attribute s. e , the agent performs self-learning based on the perceived abnormal state, where the prediction function w of the agent in each learning iteration process [σ,σ+1] can be defined as formula (1)

[0077]

[0078] Where: x m (t) is the iterative function of the abnormal state attribute s of the hydropower unit.

[0079] When the hydropower unit is operating normally m (t) = 1, when the hydropower unit fails or is idle, xm(t) = 0. Assuming that the normal operating parameters of the hydropower unit in the initial state are x0, the multi-agents also use it as a reference when judging the abnormal state of the unit. However, during the operation, due to the combined effects of factors such as its own wear and tear and the surrounding environment, the normal operating parameters will fluctuate to a certain extent. The calculation method of this fluctuation degree is shown in formula (2).

[0080]

[0081] Where: k is the fluctuation degree of the normal operating parameters of the hydropower unit; x is the actual parameter information of the hydropower unit after running for t time.

[0082] When t approaches 0 infinitely, k and t exist in a linear relationship. This feature is used to select the decision-making behaviors of the intelligent agent that can achieve the maximum return under various abnormal conditions of the hydropower unit. In order to balance the discovery and utilization of the intelligent agent in the learning process, the balanced intelligent agent decision strategy local search and global search are adopted to improve the learning effect of the intelligent agent in the virtual environment. In addition, the fully trained intelligent agent is applied to the actual production process, which can independently make decision-making behaviors according to the current operating status of the hydropower equipment and optimize the uncertainty abnormal factors in the hydropower unit. Based on the above reinforcement learning mechanism content, with the help of the Speedy Q-learning update method, the problems of large computational complexity and slow convergence speed of the intensity learning algorithm are solved. The main idea of ​​Speedy Q-learning is to use the Q value updated in the previous iteration step as the maximum Q value of the next state, which can effectively improve the convergence speed of the algorithm. The standard value in the multi-agent is updated, and the corresponding update mechanism can be formula (3).

[0083]

[0084] Where x(t) is the standard parameter information for judging the abnormal state of the hydropower unit in the multi-agent at any time; Δ is the error range of the historical observation data; is the learning function, which can be used to calculate the continuous k change rate.

[0085] It should be noted that when the normal state judgment standard parameters of the hydropower unit calculated by formula (2) exceed the design value of the equipment itself, it indicates that the equipment has reached its maximum service life and needs to be updated and replaced.

[0086] The Speedy Q-learning algorithm is used to perform partial sampling on the subspace of the next state joint action space and obtain the maximum Q value. In order to reduce the amount of calculation in the learning stage and improve the efficiency of the algorithm, this paper introduces a reinforcement learning mechanism in the multi-agent so that the load can be accurately judged according to the actual operating status of the current hydropower unit during the standardized maintenance and control of the hydropower unit. If the actual operating status of the hydropower unit is abnormal, it can be discovered in time and the corresponding fault processing can be carried out.

[0087] 3. Process and method for identifying abnormal status of hydropower units:

[0088] The maintenance and control of hydropower units is carried out on the basis of constructing a multi-agent reinforcement learning mechanism. In the process of realizing the maintenance and control of hydropower units, the connection relationship between multi-agents and the hydropower unit control system is established. Considering that the original data of hydropower units mostly exist in the form of Wigner-Ville Distribution (WVD), the RDF strategy is used to set data conversion instructions for multi-agents. After receiving the status data file in the form of WVD, it is made into an RDF file, and then the corresponding parameter information is integrated into the form of SVG graphics using WVD-RDF data. The purpose of this is to simultaneously determine the corresponding fault location while determining the abnormality of the hydropower unit. In order to meet the reinforcement learning needs of multi-agents, a data storage mode is established for it. Considering that the data scale of the hydropower unit is large and direct storage requires a large space, EEMD decomposition is used to convert the data into a matrix form with elements as the basic unit. After receiving the real-time hydropower unit operation data, the multi-agent is processed as above, and the relationship between the fault elements in the data and the calculated real-time standard parameters is matched by the middleware infrastructure of the agent to achieve accurate identification of its abnormal situation. The specific implementation process is as follows. Figure 1 shown.

[0089] As shown in Figure 1, after the multi-agent establishes the operation data of the hydropower unit through the message passing interface (MPI), the real-time abnormal state judgment standard is calculated according to the development of historical data. On this basis, the data input into the multi-agent is converted into an RDF file, and the relationship between it and the state judgment standard is compared. When it is within the standard parameter range, it indicates that the unit is in a normal state at this time; when it exceeds the standard parameter range, it indicates that the unit is abnormal at this time. Combined with SVG graphics, the specific fault location is located, and it is transmitted to the hydropower unit control system as an execution instruction for maintenance work.

[0090] In this way, combined with the actual situation, the operating status of the hydropower unit can be judged, and the abnormal status of the equipment can be controlled with more reasonable standards. In addition, a single objective problem needs to be designed for each agent. In order to ensure the expected benefits, the agent must be rewarded at the optimal time, but this will also cause greedy behavior in a model. The solution is as follows: adjust the behavior of the agent in each period of time until it is recognized. In order to avoid repairing non-critical agents during high demand stages, the problem is converted into a multi-objective problem. Therefore, the second goal of hydropower unit optimization is to punish the improper behavior of the agent. If the agent decides to be repaired when the demand is high, a penalty is input to the agent model, which can be accomplished by the future environmental status information provided by the prediction component.

[0091] 4. Advanced applications of equipment intelligent operation and decision support

[0092] 1) Special analysis:

[0093] The system provides a rich intelligent report generation model, constructing a four-layer intelligent report configuration mode of "parameter selection-parameter conversion-comprehensive judgment-generation setting". You can arbitrarily select the parameters of the power station and equipment for statistical analysis according to your own needs. The parameter conversion includes multi-parameter state quantity conversion and single-parameter state quantity conversion mode to meet the needs of various parameter statistics involved in the report. The parameter conversion method includes data aggregation, exponential regression, linear regression, polynomial regression, logarithmic regression, FFT, data-based statistical methods and other analysis methods. For special reports such as special analysis reports that need to give comprehensive evaluation results, comprehensive evaluation can be performed through logical judgment rules between multiple parameters. The logical judgment rules include AND, OR, NOT, etc., so as to automatically give qualitative evaluation results. Finally, the data acquisition range and report generation frequency of the report are set to complete the overall configuration function of the intelligent report.

[0094] According to the requirements provided by the operating personnel, based on the system monitoring data and computer monitoring system data, statistical analysis is performed on the operating status of the units, switch stations, public equipment, auxiliary equipment, hydraulic facilities, etc., including continuous state statistical analysis (maximum value, minimum value, average value statistics, etc.), discrete state statistical analysis (such as equipment operating time, number of oil pump starts and stops, etc.), and corresponding data reports are generated.

[0095] 2) Analysis of key indicators of units and equipment:

[0096] The monitoring data is analyzed for correlation, and based on the special feature analysis of a single monitoring point and the analysis of the correlation feature quantity of multiple monitoring points, a behavioral relationship model is constructed, and an automatic alarm and early warning method for equipment failure based on status and performance is proposed. By introducing the influencing factors of the status, integrating dynamic threshold analysis, similar influencing factor trend analysis, and similar influencing factor analogy analysis, intelligent analysis of system anomalies is achieved; the performance indicators of the system execution tasks are automatically calculated according to the indicator analysis calculation model, and the performance quality is evaluated; then, alarm or early warning information is automatically generated according to the status anomaly detection and performance quality evaluation results; alarm or early warning information is automatically generated according to the status anomaly detection and performance quality evaluation results; and the relevant equipment status indicator quantity early warning information can be pushed to the corresponding equipment management personnel through the information push service, so as to realize the directional early warning of abnormal information, and the information sending strategy can be freely configured.

[0097] Intelligently identify the equipment operating conditions and historical data, and use comprehensive statistical analysis results to perform configurable automatic analysis and calculation of events such as normal operation threshold exceeding the limit, gradient exceeding the limit, and correlation coefficient exceeding the limit on the key state characteristic indicator values ​​and equipment fault indicator values. The alarm warning module provides functions such as alarm warning level setting, automatic recording of alarm events, and alarm event query. The warning methods provided by the early warning alarm system are as follows: vector target area map warning and spectrum distribution target area warning are used to monitor the distribution characteristic changes of each spectrum component of the unit vibration, swing, and pressure pulsation, and can detect early signs of faults, including vector target area map warning, spectrum distribution target area warning, trend warning, vibration swing measurement value exceeding the limit warning, etc. Users can customize almost all alarm conditions according to business needs. The alarm types are diversified, with limit alarm, amplitude alarm, missing number alarm, and custom alarm capabilities.

[0098] 3) Evaluation of unit and equipment status:

[0099] Formulate equipment condition evaluation standards and health status prediction standards, and develop a hydropower station intelligent equipment condition analysis module. By studying the equipment health status evaluation, specify the equipment operation status and equipment health evaluation standards, analyze and evaluate the data of each indicator item reflecting the equipment health status, and finally obtain the overall equipment health status level, namely normal, attention, abnormal and severe. After the relevant business review process, a water machine equipment status evaluation report is formed to guide the development of equipment maintenance work; and provide a decision-making basis for equipment maintenance fault prediction, including evaluation contents such as operating status characteristics, stability characteristics, absolute evaluation standards and relative evaluation standards.

[0100] The status evaluation module is based on relevant water machine equipment status evaluation guidelines, combined with pre-test procedures, monitoring analysis, etc., fully considering the differences in status evaluation parameters, focusing on the research of water machine equipment status evaluation algorithms, and realizing new status evaluation functions based on multi-monitoring information fusion technology, so that the results of status evaluation are more reasonable; research on software implementation solutions for status evaluation rule maintenance, automatic evaluation triggering and evaluation result review and confirmation mechanism; research on the implementation of manual evaluation functions, manual triggering of status evaluation, and improvement of the interactive performance of the status evaluation module; research on the implementation of status evaluation information query and statistical analysis functions, so that the statistical analysis function is more targeted and professional on the basis of convenience and speed.

[0101] 5. Hydropower unit status analysis microservice:

[0102] Based on the above device status and analysis and related software functions, a microservice layered architecture design method will be adopted, that is, a remote service publishing system based on the RESTful architecture to achieve comprehensive microservices in the application architecture.

[0103] RESTful Web service is a lightweight Web service architecture based on REST style. Each resource is assigned a unique identifier URI containing scope information. In the Panoramic Information Integration Platform, RESTful Web service is mainly used to implement access to decentralized data sources and other operations. The specific request and response process is as follows. Figure 2 shown.

[0104] When adding, reading, modifying, deleting, etc. data, the service requester places the requested data in the HTTP document body and sends the request to the server through URI① (first URI). After the server completes the processing according to the method parameters, it returns the result to the service requester through URI② (second URI) in the format of XML, Hypertext Markup Language (HTML), Lightweight Data Format (JSON), etc. RESTful Web services are simple, efficient, and low-coupled, and are particularly suitable for accessing data in the platform. Once the data source is changed or added, only a simple modification of the URI is required to achieve rapid response of the service.

[0105] The message-driven lightweight service bus is the core component of the platform. The platform bus is the link between the upper and lower levels and is the business processing center of the entire platform. The lightweight service bus provides a set of rich functions to enable management and monitoring of interactions between applications. It is integrated between the logic layer and service layer of the application system. It connects and integrates components on different systems through multiple communication protocols and maps them into services to complete interoperability between services. The message-driven lightweight service bus message routing and service management and monitoring are shown in the attached figure. Figure 3 shown.

[0106] The service container is one of the important components of the service bus, which can encapsulate user application software and its own basic services. Distributed processing of business is achieved through service registration, discovery and selection. In the platform, the service list is pre-registered in the service container according to the corresponding functional application. When the bus receives a request agent from the application layer undertaken by the logic layer, the service container discovers the request and selects the corresponding service according to the list. The service bus provides two message modes: publish / subscribe and request / reply. The message routing analyzes the steps of service transmission and establishes transmission lines and rules to finally realize the message transmission process. The bus performs data operations on the required data sources by calling RESTful Web services and returns the results to the service bus. The business logic, processes and advanced applications of the system are defined and decomposed and encapsulated as services. When implementing a specific application, the platform bus combines, reuses and orchestrates data services based on the process orchestration engine to achieve the final business integration and realize advanced applications at the application layer.

[0107] The present invention ensures the stability and accuracy of data through hybrid access mode and data completion and error correction technology;

[0108] The present invention adopts a mechanism based on multi-agent reinforcement learning to improve the accuracy and response speed of state recognition;

[0109] The present invention simplifies the identification process and improves the accuracy of fault location by optimizing the RDF strategy and data storage mode;

[0110] The present invention provides more comprehensive and in-depth decision support through special analysis, key indicator analysis and status evaluation;

[0111] The present invention adopts a service bus and message-driven mechanism based on RESTful architecture to simplify the interaction between services and improve the maintainability of the system.

[0112] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0114] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning, characterized in that: The following steps are involved: Collect relevant operation data of the hydropower station, perform data preprocessing, generate data sets, and obtain abnormal state attributes, where the state attribute set includes equipment delay time, abnormal operating temperature, cooling water pressure, and related equipment status information during the operation of the hydropower unit; An intelligent agent is set to perceive the abnormal state attributes, and according to the degree of fluctuation of normal operating parameters, the decision-making behavior of the intelligent agent that achieves the maximum return under various abnormal states of the hydropower unit is screened. A reinforcement learning mechanism is set to control the intelligent agent to determine whether there is an abnormality in the actual operating state of the current hydropower unit.

2. The method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning according to claim 1, characterized in that: In the process of obtaining relevant operating data of the hydropower station, a data access service is built by combining direct database interaction and standard industrial protocol communication to obtain relevant operating data of the hydropower station, which is used to ensure real-time access to online monitoring of original waveform data, characteristic spectrum structure and characteristic value parameters.

3. The method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning according to claim 2 is characterized by: In the process of data preprocessing, data preprocessing is carried out through data completion and error correction, data cleaning and data fusion, among which, The data completion and error correction include: Missing value processing: For analog quantities, linear interpolation is used for processing, and for switch quantities, the previous value method is used to fill in the missing values; Outlier processing: The mean and standard deviation method of the upper and lower limit methods are used to process outliers; The data cleaning includes: cleaning of erroneous data, cleaning of contaminated data and cleaning of redundant data; The data fusion includes time synchronization technology, working condition synchronization technology and event synchronization technology.

4. The method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning according to claim 3 is characterized by: In the process of obtaining the degree of fluctuation, based on the parameter x0 of the normal operation of the hydropower unit in the initial state, according to the actual parameter information x of the hydropower unit after running for t time, the degree of fluctuation is obtained, which is expressed as:

5. The method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning according to claim 4 is characterized by: In the process of setting up the agent, the prediction function of the agent is expressed as: In the formula, m represents the agent, [σ,σ+1] represents each learning iteration process, and x m (t) is the iterative function of the abnormal state attributes of the hydropower unit.

6. The method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning according to claim 5 is characterized by: In the process of setting up the mandatory learning mechanism, the mandatory learning mechanism includes: The agent decision strategy of balancing local search and global search is adopted to improve the learning effect of the agent in the virtual environment; The fully trained intelligent agent is applied to the actual production process to make autonomous decisions based on the current operating conditions of the hydropower equipment, and to optimize the uncertain abnormal factors in the hydropower units through intelligent decision-making.

7. The method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning according to claim 6 is characterized by: In the process of setting up the forced learning mechanism, the forced learning mechanism is optimized through the Speedy Q-learning update method, wherein the Speedy Q-learning algorithm performs partial sampling on the subspace of the next state joint action space and obtains the maximum Q value.

8. The method for assisting decision-making in intelligent operation of hydropower plant equipment based on reinforcement learning according to claim 7 is characterized by: In the process of setting up the intelligent agent, the RDF strategy is used to set data conversion instructions for the intelligent agent. The status data file received in WVD format is made into an RDF file. Then, the WVD-RDF data is used to integrate the corresponding parameter information into the form of SVG graphics, and the data is converted into a matrix form with elements as the basic units after EEMD decomposition.

9. A hydropower plant equipment intelligent operation auxiliary decision-making system based on reinforcement learning, the system is used to implement the hydropower plant equipment intelligent operation auxiliary decision-making method based on reinforcement learning as claimed in any one of claims 1 to 8, characterized in that: The system includes: The data processing module is used to collect the relevant operation data of the hydropower station, perform data preprocessing, generate data sets, and obtain abnormal state attributes, where the state attribute set includes equipment delay time, abnormal operation temperature, cooling water pressure, and related equipment status information during the operation of the hydropower unit; The prediction module is set in the intelligent agent to assist the intelligent agent in perceiving the abnormal state attributes, and screen the decision-making behavior of the intelligent agent to achieve the maximum return under various abnormal states of the hydropower unit according to the degree of fluctuation of the normal operating parameters. By setting a reinforcement learning mechanism, the intelligent agent is controlled to judge whether there is an abnormality in the actual operating state of the current hydropower unit.

10. The intelligent operation decision support system for hydropower plant equipment based on reinforcement learning according to claim 9 is characterized by: The system adopts the microservice layered architecture design method to build a remote service publishing system based on the restful architecture. When adding, reading, modifying, and deleting data, the service requester places the requested data in the HTTP document body and sends the request to the server through the first URI. After the server completes the processing according to the method parameters, it returns the result to the service requester in the format of XML, hypertext markup language, and lightweight data through the second URI.

Images