Method and monitoring system for monitoring the performance of decision logic of a controller

Abstract

During live operation of the controller (31), performance of the decision logic (35) of the controller (31) of the industrial automation control system is monitored. The monitoring system (20) receives operational data collected during live operation of the controller (31). The monitoring system analyzes the operational data using pre-operation data generated prior to live operation of the controller (31) and / or performance evaluation logic (27) generated prior to live operation of the controller (31) to evaluate performance of the decision logic (35). The monitoring system (20) generates an analysis output based on the analysis results.

Description

Technical Field

[0001] This invention relates to methods, apparatus, and systems for monitoring the performance of decision logic in controllers of industrial automation control systems (IACS), particularly power distribution, power transmission, or power generation systems. More specifically, this invention relates to a technique that allows for the evaluation of decision logic during controller operation in the field. Background Technology

[0002] Modern industrial automation control systems (IACS), such as power generation or transmission systems, power grids or substations, and modern industrial systems, comprise a large number of components. The decision logic of protective devices in such systems, such as protective relays, determines which of various actions to take under what circumstances.

[0003] For example, in the real-time operation of the power industry, transmission and distribution system equipment (including transformers, overhead lines, underground cables, series / parallel components, etc.) is protected through measurement systems (voltage, current), digital relays, and circuit breakers. Control logic deployed in digital relays uses the measured signals to determine if there are critical faults that should be cleared to avoid damaging system equipment and ultimately sends a disconnect signal to the circuit breaker. Rapid fault identification and clearance are crucial for the reliability and safety of the entire system.

[0004] The decision logic for each relay (i.e., the protection logic), as well as the coordination schemes among multiple relays, are designed and tested under the anticipated power grid scenario. This is typically done by human expert engineers. During the design phase, engineers simulate faults and other disturbances (such as switching events in the power grid) to evaluate and improve the performance of the protection control logic. Performance metrics are usually set by prevailing practices for a given power grid. Once deployed, the protection logic remains unchanged until performance errors are observed.

[0005] As converter-interface generators have become widespread and replaced traditional generators, they have introduced more randomness into power supply in both time and space. This, coupled with the surge in electric transportation adding further randomness to demand, has made the design of protection systems increasingly complex. Furthermore, the lack of short-circuit current capacity provided by converter-interface generators, the varying properties of short-circuit currents (e.g., delayed zero-crossing, distorted signals), and the multidirectional currents resulting from changes in the spatial patterns of power generation necessitate adjustments to protection systems to adapt to the ever-changing environment.

[0006] Therefore, the development of protection logic has become an increasingly complex task. Furthermore, due to changes in grid infrastructure and generation / demand patterns, the applicability of protection systems needs to be reassessed more frequently, thus requiring this task to be performed more and more often.

[0007] The traditional approach to protection logic design involves expert engineers selecting from multiple protection functions (e.g., overcurrent, directional, distance, differential protection) or combinations thereof for each specific design case and determining the settings (i.e., functional parameters) associated with the selected functions (or multiple selected functions). The increasingly complex operation of transmission and / or distribution systems, approaching their limits, is a task requiring the combination of different functionalities. The interaction of multiple controllers that can be deployed simultaneously in the field further complicates this task.

[0008] To assist expert engineers in the process of establishing highly complex decision-making logic, computer-based technologies can be used.

[0009] Not only is the generation of the decision logic complex, but monitoring it during field operation is also a challenging task. This is especially true when the decision logic is not (only) generated by expert engineers, but rather generated using computer-based implementations, such as machine learning (ML) techniques. For example, when decision logic is generated using ML, evaluating its performance during field operation by human experts can become challenging.

[0010] Typical machine learning (ML) systems are incredibly powerful at learning various functions, but they often appear as black boxes to human engineers. This can have various drawbacks. For example, human engineers typically cannot understand how the decision logic obtained by training a decision ML model maps input signals to various feasible control actions. This presents a challenging task for human engineers when the decision logic may need to be modified in response to changes in the power system topology or other updates to the power system. Summary of the Invention

[0011] There is a need for improved techniques for monitoring the performance of decision logic executed by controllers in industrial automation control systems (IACS), particularly power distribution, power transmission, or power generation systems. In particular, there is a need for an improved technique that allows for the automatic monitoring and evaluation of the decision logic during the field operation of the controller. An improved technique is also needed that allows for the automatic monitoring and evaluation of the decision logic, even when, for example, the decision logic has been created using machine learning (ML) techniques without explicit understanding of its internal operation.

[0012] According to embodiments of the present invention, a method and system for implementing a monitoring system are provided, which continuously and autonomously evaluates whether decision logic (e.g., control logic) is performed in accordance with quality standards during the field operation of a controller.

[0013] The monitoring system can be run to determine whether the decision logic needs to be updated.

[0014] The monitoring system can operate to autonomously assess the performance and / or applicability of the decision logic in changing environments.

[0015] The output generated by the monitoring system can be output through a human-machine interface (HMI) and / or fed back to a system that automatically generates decision logic for the controller of the IACS.

[0016] The monitoring system used to perform decision logic evaluation can be a component of the control system, or the monitoring system can reside in an external system, such as as part of a cloud service.

[0017] The monitoring system can monitor controller actions and the environment, can run to evaluate the controller's performance with respect to (a set of) performance metrics, can run to identify changes in the environment, and can evaluate whether the decision logic meets its original design objectives. The monitoring system can automatically notify operators that the decision logic needs to be redesigned or updated, thereby optionally providing insights and guidance on the root causes of the need for updates, and / or can automatically trigger modifications to the design process of the computer-implemented decision logic.

[0018] According to an embodiment, a method is provided for monitoring the performance of the decision logic of a controller in an Industrial Automation Control System (IACS), particularly a controller for a power system, during field operation of the controller. The method may include receiving operational data collected during field operation of the controller. The method may include performing analysis of the operational data to evaluate the performance of the decision logic using pre-operation data generated prior to field operation of the controller and / or performance evaluation logic generated using data generated during the computer implementation process of creating the decision logic. The method may include generating analysis output based on the analysis results.

[0019] This method can be executed automatically by the monitoring system.

[0020] Operational data can include information about the decision outputs of the decision logic during the field operation of the controller.

[0021] Execution analysis may include calculating one or more key performance indicators (KPIs) based on the decision output according to at least one metric.

[0022] At least one metric may depend on user input.

[0023] The method may also include receiving user input specifying at least one metric in the user interface.

[0024] Operational data may also include power system data that affects the operation of the controller.

[0025] Power system data may include at least one, several, or all of the following: data on a bus to which the controller can be connected, data from a bus different from the bus to which the controller can be connected, data from system-wide measurements, data from an energy management system (EMS), and data from a distribution management system (DMS).

[0026] Analysis can be performed using pre-run data.

[0027] The method may also include retrieving pre-run data from a database during the field operation of the controller.

[0028] The pre-run data may include at least one, several, or all of the following: a model used to perform simulations when generating decision logic; scenarios, test cases, and / or events simulated when generating decision logic; and the performance of the decision logic in the simulations performed when generating decision logic.

[0029] Execution analysis may include evaluating the accuracy of the models used to perform simulations when generating decision logic.

[0030] Evaluating the accuracy of the model used to perform the simulation when generating decision logic may include model validation, calibration and / or determination and / or comparison of signals observed during controller field operation with signals expected based on the model used to perform the simulation when generating decision logic.

[0031] Alternatively or additionally, performing analysis may include evaluating scenarios, test cases, and / or events simulated when generating decision logic.

[0032] Evaluating the scenarios, test cases, and / or events simulated during the generation of decision logic may include comparing run points observed during the field operation of the controller with run points generated by the scenarios, test cases, and / or events simulated during the generation of decision logic.

[0033] Alternatively or additionally, performing analysis may include comparing control actions taken by the controller during field operation with control actions simulated when generating decision logic.

[0034] Comparing the control actions taken by the controller during field operation with the control actions simulated when generating decision logic may include: creating a dataset that includes system operating conditions and control actions taken by the controller during field operation; and comparing the dataset with pre-operation data that includes system operating conditions and the simulated control actions that occur when generating decision logic.

[0035] Alternatively or additionally, performing analysis may include comparing the values ​​of at least one, more, or all KPIs output by the decisions made by the controller during field operation with the values ​​of at least one, more, or all KPIs stored in pre-operation data.

[0036] During the field operation of the controller, the monitoring system can retrieve information from the database about one or more of the following: the model used to perform the simulation when generating the decision logic, the scenario simulated when generating the decision logic, test cases and / or events, control actions simulated when generating the decision logic, and KPIs.

[0037] It can retrieve and process not only the decision logic used in field operation, but also one or more alternative decision logics considered in the decision logic creation process, regarding one or more of the following: the model used to perform the simulation when generating the decision logic, the scenario simulated when generating the decision logic, test cases and / or events, control actions simulated when generating the decision logic, and KPI information.

[0038] The decision logic used in field operation, as well as one or more alternative decision logics considered during the creation of the decision logic, can be a decision logic ML model, particularly an artificial neural network (ANN). The topology (e.g., in terms of the number of layers) and / or parameters (e.g., the weights of the forward function chain of the ANN) of the decision logic ML model can distinguish between the decision logic used in field operation and the one or more alternative decision logics considered during the creation of the decision logic.

[0039] Alternatively or additionally, the monitoring system can use performance evaluation logic to perform analysis.

[0040] The performance evaluation logic can be used to perform analysis without accessing a database containing information about the logic that generates the decision during the controller's field operation.

[0041] The performance evaluation logic can run autonomously during the controller's field operation.

[0042] The performance evaluation logic can perform classification tasks.

[0043] The performance evaluation logic can receive runtime data as input and can generate analytical output with multiple discrete values.

[0044] Multiple discrete values ​​may include indicators of whether the decision logic is considered to operate according to a standard based on performance metrics.

[0045] Multiple discrete values ​​may include at least one value that indicates that the decision logic is considered to be operating according to a performance metric-based standard, but gradually approaches an unacceptable operating mechanism.

[0046] The performance evaluation logic can accept inputs that may include voltage, current, phasor and / or quadrature measurements.

[0047] Alternatively or additionally, the performance evaluation logic may receive inputs that may include the power system topology.

[0048] Alternatively or additionally, the performance evaluation logic may receive input that may include weather information.

[0049] Alternatively or additionally, the performance evaluation logic may receive inputs that may include electricity prices.

[0050] The performance evaluation logic can generate outputs that can be indicators of the applicability of the decision logic.

[0051] Alternatively or additionally, the performance evaluation logic may generate output that may include predictive timing information indicating when the decision logic may need to be corrected or otherwise revisited.

[0052] Alternatively or additionally, the performance evaluation logic may generate output that can include information about the possible root causes of poor performance of the decision logic.

[0053] Alternatively or additionally, the performance evaluation logic may generate output that may include information about root cause solutions, which may indicate how the decision-making logic can be improved.

[0054] Information about the possible root causes of poor performance and / or about solutions to the root causes may include scenario parameters, particularly scenario signatures (i.e., parameters or combinations of parameters that are inputs to the decision logic).

[0055] The performance evaluation logic can be a performance evaluation ML model.

[0056] The performance evaluation logic can be an artificial neural network.

[0057] Data generated during the decision-making process can be used to train a performance evaluation ML model that implements the performance evaluation logic.

[0058] A performance evaluation ML model can be trained to implement the performance evaluation logic before the controller is run in the field and after the decision logic has been generated.

[0059] This method may include training a machine learning model using a second set of scenarios.

[0060] The second set of scenarios may include scenarios that may be outside the controller's operating specifications.

[0061] The second set of scenarios can be different from the first set of scenarios used to train the decision logic.

[0062] The method may further include generating at least one subset of a second set of scenarios using an additional scenario-generating ML model that challenges decision-making logic.

[0063] Scene creation logic can be used to generate a first set of scenes and / or a second set of scenes.

[0064] The first set of scenes and / or the second set of scenes can be generated by another machine learning model. The first set of scenes and / or the second set of scenes can be generated using a generative adversarial network.

[0065] The method may also include monitoring or predicting the time-related evolution of the analysis results and generating analysis output based on the time-related evolution.

[0066] Predicting time-related evolution can include predicting future run points and the output of predictive decision logic for those future run points.

[0067] If time-related evolution indicates that future controller performance does not meet performance metric-based standards, alarms, warnings, or other analytical outputs can be automatically generated.

[0068] Monitoring systems can be used to predict the time-dependent future evolution of the performance of decision logic.

[0069] To allow the performance evaluation logic to perform predictions, the method may include training the performance evaluation logic using scenario batches, where each batch represents a time series.

[0070] This method may include computer-implemented generation of scene batches, where evolution is driven by one or more latent variables.

[0071] Latent variables (multiple latent variables) can be associated with actual variables that define the scenario in a defined way.

[0072] Latent variables (multiple latent variables) can be associated with actual variables that define the scenario in a random manner.

[0073] To allow the performance evaluation logic to perform predictions, the monitoring system may alternatively or additionally execute at least one predictor, wherein the output of the predictor is supplied as input to the performance evaluation logic.

[0074] The monitoring system can execute multiple predictors and multiple instances of performance evaluation logic, with each of the multiple predictors generating the input to the performance evaluation logic.

[0075] Predictors (multiple predictors) can receive running data as input and can generate predictions for future running points based on, for example, historical data.

[0076] When multiple predictors are deployed, they can generate predictions for multiple different time points and / or multiple different environmental conditions.

[0077] The predictor (multiple predictors) can be a recurrent neural network.

[0078] The method may also include controlling a human-machine interface (HMI) to output analytical output, wherein the analytical output includes information about the past, present and / or predicted future performance of the controller during field operation.

[0079] A method for operating power system assets of a power system may include executing decision logic by at least one integrated circuit of the controller during field operation of the controller, which includes generating and outputting decision outputs to control the power system assets.

[0080] The method may further include a method for monitoring the decision logic of the controller during field operation of the controller, performed by at least one integrated circuit of the controller or by at least one, several or all other integrated circuits.

[0081] The method may also include analyzing outputs via a human-machine interface (HMI) and / or automatically executing control actions based on the analyzed outputs.

[0082] Control actions can trigger the regeneration or updating of decision logic.

[0083] The method may also include performing root cause determination for poor performance of decision logic during field operation and / or providing instructions for improving decision logic during field operation.

[0084] The method may also include implementing root cause solutions. Root cause solutions may include determining which scenario parameters (which scenario parameters) of the actual operating scenario need to be modified so that the decision logic meets performance metric-based standards.

[0085] Root cause solutions may include offsets in the scenario parameter space determined by the monitoring system that cause the performance of the decision logic to meet the criteria based on performance metrics.

[0086] Determining offsets in the scene parameter space can include performing constrained optimizations.

[0087] Constrained optimization can use an objective function as the offset standard.

[0088] Constrained optimization can use performance evaluation logic to determine whether the constraints are met.

[0089] The performance analysis may include the performance of decision logic determined by the monitoring system as being insufficient due to exogenous factors of the power system, such as spatial and / or temporal demand profiles and / or weather conditions.

[0090] Alternatively or additionally, the analysis may include the decision logic performance determined by the monitoring system to be insufficient due to changes in the power system infrastructure (e.g., new transmission and / or distribution lines, new generation or storage capacity, installation of new controllers, or possibly changes in the operating topology caused by switching).

[0091] Alternatively or additionally, the analysis may include determining, by the monitoring system, that the performance of decision logic is inadequate due to a change in another decision logic executed by another controller of the power system.

[0092] Alternatively or additionally, performance analysis may include decision logic performance that is deemed insufficient due to changes in performance metrics, as determined by the monitoring system.

[0093] The decision logic can generate a decision output by processing controller inputs, which include one, several, or all of locally available measurements, remotely captured measurements, and system-wide observations.

[0094] The controller can be a controller for distributed energy resources (DER).

[0095] The controller can be a local controller.

[0096] The local controller can be selected from the following groups: protection relays, generator control systems (e.g., governors and exciters / automatic voltage regulators), control systems for high-voltage direct current (HVDC) equipment, control systems for flexible alternating current transmission systems (FACTS) equipment, decision logic for switched capacitors and / or reactors, low-frequency unloading relays, and undervoltage unloading relays.

[0097] The controller can be a central controller.

[0098] The central controller can be selected from the following groups: EMS / DMS, such as decision logic for secondary frequency control, decision logic for generation redistribution, decision logic for demand flexibility, decision logic for reclosing operations, decision logic for secondary voltage control through reactive power redistribution, and decision logic for coordinating the actions of the entire power grid (e.g., in the case of system protection schemes or remedial action schemes, which are typically activated after a disturbance).

[0099] The controller can control controllable parameters, which can be selected from a group of the following: active (P) and reactive (Q) power injection from one or more generators and / or energy storage systems; P / Q injection from DC links or back-to-back DC; Q injection from FACTS devices (such as SVC or STATCOM); power flow control devices; tap positions of one or more capacitor banks, reactors, and OLTCs; requirements; P / Q injection from electronic traffic platforms; and the status of switches and / or reclosers.

[0100] A monitoring system for the decision logic of an industrial automation control system (IACS), particularly a power system controller, used during field operation includes at least one integrated circuit (ICC) for receiving operational data collected during field operation. The IC can operate to perform analysis of the operational data to evaluate the performance of the decision logic using pre-operation data generated prior to field operation of the controller and / or performance evaluation logic generated prior to field operation of the controller, and / or generated during the computer implementation process of creating the decision logic. The IC can also operate to generate analytical output based on the analysis results.

[0101] The monitoring system can be operational so that the operational data can include information about the decision outputs of the decision logic during the field operation of the controller.

[0102] The monitoring system can be operational, enabling the execution analysis to include calculating one or more key performance indicators (KPIs) from the decision output based on at least one metric.

[0103] The monitoring system can operate such that at least one metric can depend on user input.

[0104] The monitoring system can be operational to receive user input specifying at least one metric at the user interface.

[0105] The monitoring system is operational, allowing the operational data to further include power system data that affects the controller's operation.

[0106] The monitoring system can operate such that power system data may include at least one, several, or all of the following: data on a bus to which the controller can be connected, data from a bus different from the bus to which the controller is connected, data from system-wide measurements, data from an energy management system (EMS), and data from a distribution management system (DMS).

[0107] The monitoring system is operational, allowing for analysis using data collected before operation.

[0108] The monitoring system can be operational to retrieve pre-operation data from the database during the controller's field operation.

[0109] The monitoring system can be operational such that the pre-run data may include at least one, several, or all of the following: the model used to perform the simulation when generating the decision logic; the scenarios, test cases, and / or events simulated when generating the decision logic; and the performance of the decision logic in the simulation executed when generating the decision logic.

[0110] The monitoring system can be operational, allowing execution analysis to include evaluating the accuracy of the model used to perform the simulation when generating decision logic.

[0111] The monitoring system can be operational such that assessing the accuracy of the model used to perform the simulation when generating decision logic may include model validation, calibration, and / or determining and / or comparing signals observed during controller field operation with signals expected based on the model used to perform the simulation when generating decision logic.

[0112] Alternatively or additionally, the monitoring system may operate such that the performance analysis may include evaluating scenarios, test cases, and / or events simulated during the generation of decision logic.

[0113] The monitoring system can be operational such that evaluating the scenarios, test cases, and / or events simulated during the generation of decision logic can include comparing the run points observed during the controller's field operation with the run points generated by the scenarios, test cases, and events simulated during the generation of decision logic.

[0114] Alternatively or additionally, the monitoring system may operate such that performing analysis may include comparing control actions taken by the controller during field operation with control actions simulated when generating decision logic.

[0115] The monitoring system can be operated such that comparing the control actions taken by the controller during field operation with the control actions simulated when generating decision logic may include: creating a dataset that includes system operating conditions and control actions taken by the controller during field operation; and comparing the dataset with pre-operation data that includes simulated system operating conditions and control actions implemented as a result when generating decision logic.

[0116] Alternatively or additionally, the monitoring system may be configured such that the analysis performed may include comparing the output of one, several, or all KPIs of decisions made by the controller during field operation with the values ​​of at least one, several, or all KPIs stored in pre-operation data.

[0117] The monitoring system can operate to receive information during the field operation of the controller regarding one or more of the following: the scenarios simulated when generating decision logic, test cases and / or events simulated when generating decision logic, control actions simulated when generating decision logic, and KPIs from the database.

[0118] The monitoring system can operate to receive and process information about one or more of the following: models used to perform simulations when generating decision logic; scenarios simulated when generating decision logic; test cases and / or events; control actions simulated when generating decision logic; and KPIs for one or more alternative decision logics considered during the creation of decision logic, in addition to decision logic used not only during field operation.

[0119] The decision logic used in field operation, as well as one or more alternative decision logics considered during the creation of the decision logic, can be a decision logic ML model, particularly an artificial neural network (ANN). The decision logic ML model topology (e.g., in terms of the number of layers) and / or parameters (e.g., the chain weights of the ANN's positive function) can be distinguished between the decision logic used in field operation and the one or more alternative decision logics considered during the creation of the decision logic.

[0120] Alternatively or additionally, the monitoring system can be run to perform analysis using performance evaluation logic.

[0121] The performance evaluation logic can run to perform analysis without requiring access to a database containing information about the decision-making logic during the controller's field operation.

[0122] The performance evaluation logic can run autonomously during the controller's field operation.

[0123] The performance evaluation logic can perform classification tasks.

[0124] The performance evaluation logic can receive runtime data as input and can generate analytical output with multiple discrete values.

[0125] The performance evaluation logic can run such that multiple discrete values ​​can include values ​​that indicate whether the decision logic is considered to be running according to a metric based on performance metrics.

[0126] The performance evaluation logic can operate such that multiple discrete values ​​can include at least one value that indicates the decision logic is considered to be operating according to a performance metric-based standard, but gradually approaches an unacceptable operating mechanism.

[0127] The performance evaluation logic can run to receive inputs that may include voltage, current, phasor, and / or quadrature measurement results.

[0128] Alternatively or additionally, performance evaluation logic may run to receive inputs that may include the power system topology.

[0129] Alternatively or additionally, the performance evaluation logic can run to receive input that may include weather information.

[0130] Alternatively or additionally, performance evaluation logic can be run to receive inputs that may include electricity prices.

[0131] The performance evaluation logic can run to generate output, which can be a marker indicating the suitability of the decision logic.

[0132] Alternatively or additionally, the performance evaluation logic may run to generate output that may include predictive timing information indicating when the decision logic may need to be corrected or otherwise revisited.

[0133] Alternatively or additionally, the performance evaluation logic can be run to generate output that may include information about the possible root causes of poor performance of the decision logic.

[0134] Alternatively or additionally, the performance evaluation logic can run to generate output that may include information about root cause solutions. This information about root cause solutions can guide how the decision-making logic can be improved.

[0135] The performance evaluation logic can run, allowing information about the root cause solution to include scenario parameters, particularly the scenario signature.

[0136] The performance evaluation logic can be a performance evaluation ML model.

[0137] The performance evaluation logic can be an artificial neural network.

[0138] Data generated during the decision-making process can be used to train a performance evaluation ML model that implements the performance evaluation logic.

[0139] A performance evaluation ML model can be trained to implement the performance evaluation logic before the controller is run in the field and after the decision logic has been generated.

[0140] A performance evaluation ML model that implements performance evaluation logic can be trained using a second set of scenarios.

[0141] The second set of scenarios may include scenarios that may be outside the controller's operating specifications.

[0142] The second set of scenarios can be different from the first set of scenarios used to train the decision logic.

[0143] The monitoring system can be operational to monitor or predict the time-related evolution of analysis results and generate analysis output based on that time-related evolution.

[0144] The monitoring system can operate such that predicting time-related evolution can include predicting future run points and the decision outputs of the decision logic used to predict future run points.

[0145] The monitoring system can operate so that if time-related evolution indicators do not meet future controller performance criteria based on performance metrics, alarms, or other analytical outputs can be automatically generated.

[0146] The monitoring system is operational, enabling the use of the monitoring system to perform time-related future evolution of predictive decision-making logic.

[0147] The performance evaluation logic can be trained using scenario batches, with each batch representing a time series.

[0148] Scene batches may evolve, with the evolution driven by one or more latent variables.

[0149] Latent variables (multiple latent variables) can be associated with actual variables that define the scenario in a defined way.

[0150] Latent variables (multiple latent variables) can be associated with actual variables that define the scenario in a random manner.

[0151] The monitoring system may operate alternatively or additionally to execute at least one predictor, wherein the output of the predictor is supplied as input to the performance evaluation logic.

[0152] The monitoring system can run multiple predictors and multiple instances to execute performance evaluation logic, each of which generates inputs for the performance evaluation logic.

[0153] Predictors (multiple predictors) can run to receive running data as input and can generate predictions for future running points based on, for example, historical data.

[0154] When multiple predictors are deployed, they can run to generate predictions for multiple different time points and / or for multiple different environmental conditions.

[0155] The predictor (multiple predictors) can be a recurrent neural network.

[0156] The monitoring system can be run to control the output analysis of the human-machine interface (HMI), which includes information about the past, present and / or predicted future performance of the controller during field operation.

[0157] The monitoring system can operate to generate commands for power system assets used to operate the power system.

[0158] The monitoring system may include a controller that operates to execute decision logic during the controller's field operation, including generating and outputting decision outputs to control power system assets.

[0159] The monitoring system can operate to automatically execute control actions based on the analysis output via the human-machine interface (HMI) and / or based on the analysis output.

[0160] The monitoring system can be operational to perform root cause determination of poor performance of decision logic during field operation and / or provide instructions for improving decision logic during field operation.

[0161] The monitoring system is operational and can be used to execute root cause solutions. The monitoring system is operational so that root cause solutions can include determining which scenario parameters in the actual operating scenario need to be modified so that the decision logic meets performance metric-based standards.

[0162] The monitoring system can be made operational to determine the performance of the decision logic in accordance with the offset in the scenario parameter space based on performance metrics, in order to execute the root cause solution.

[0163] The monitoring system can be operational so that determining the offset in the scene parameter space can include performing constrained optimizations.

[0164] The monitoring system can be operational, allowing constrained optimization to use an objective function as the offset standard.

[0165] The monitoring system is operational, allowing constrained optimizations to use performance evaluation logic to determine whether the constraints are met.

[0166] The monitoring system can be operational, enabling the execution analysis to include, as determined by the monitoring system, inadequate performance of decision logic due to exogenous factors of the power system, such as spatial and / or temporal demand profiles and / or weather conditions.

[0167] Alternatively or additionally, the monitoring system may operate such that the analysis performed may include the decision logic performance being deemed insufficient by the monitoring system due to changes in the power system infrastructure, such as new transmission and / or distribution lines, new generation or storage capacity, installation of new controllers, or topology changes that may be caused by switching during operation.

[0168] Alternatively or additionally, the monitoring system may operate such that the analysis performed may include the monitoring system determining that the decision logic performance is inadequate due to a change in another decision logic executed by another controller of the power system.

[0169] Alternatively or additionally, the monitoring system may operate such that the execution analysis may include the monitoring system determining that the decision logic performance is inadequate due to changes in performance metrics.

[0170] According to another aspect of the present invention, a method is provided for monitoring the performance of decision logic in a controller of an industrial automation control system (IACS), particularly a controller of a power system, during field operation of the controller. This method is executed by at least one integrated circuit of the monitoring system and includes:

[0171] Receive operational data collected during controller field operation;

[0172] Perform analysis of operational data to evaluate the performance of the decision logic, wherein the analysis includes predictive analysis to identify future system operating conditions that could lead to performance degradation of the decision logic; and

[0173] At least the analytical output should be generated based on the results of the predictive analysis.

[0174] Operational data can include information about the decision outputs of the decision logic during the field operation of the controller.

[0175] Predictive analytics can include predicting future run points and predicting the decision outputs of the decision logic used to predict future run points.

[0176] Predictive analytics can be performed by performance evaluation logic.

[0177] Performance evaluation logic can be generated before the controller is run in the field and / or can be generated using data generated during the computer implementation process of creating the decision logic.

[0178] To allow the performance evaluation logic to perform predictions, the method may include training the performance evaluation logic using scenario batches, where each batch represents a time series.

[0179] This method may include computer-implemented generation of scene batches, where evolution is driven by one or more latent variables.

[0180] Latent variables (multiple latent variables) can be associated with actual variables that define the scenario in a defined way.

[0181] Latent variables (multiple latent variables) can be associated with actual variables that define the scenario in a random manner.

[0182] To allow the performance evaluation logic to perform predictions, the monitoring system may alternatively or additionally execute at least one predictor, wherein the output of the predictor is supplied as input to the performance evaluation logic.

[0183] The monitoring system can execute multiple predictors and multiple instances of performance evaluation logic, each of which generates the input to the performance evaluation logic.

[0184] Predictors (multiple predictors) can receive running data as input and can generate predictions for future running points based on, for example, historical data.

[0185] When multiple predictors are deployed, they can generate predictions for multiple different time points and / or multiple different environmental conditions.

[0186] The predictor (multiple predictors) can be a recurrent neural network.

[0187] The method may also include controlling a human-machine interface (HMI) to output analytical output, wherein the analytical output includes information about the past, present and / or predicted future performance of the controller during field operation.

[0188] If time-related evolution indicates that future controller performance does not meet performance metric-based standards, alarms, warnings, or other analytical outputs can be automatically generated.

[0189] According to another aspect of the invention, a monitoring system is provided for the decision logic of a controller in an industrial automation control system (IACS), particularly a controller for a power system, during field operation of the controller. This monitoring system includes at least one integrated circuit capable of operating to:

[0190] Receive operational data collected during controller field operation;

[0191] Perform analysis of operational data to evaluate the performance of the decision logic, wherein the analysis includes predictive analysis to identify future system operating conditions that could lead to a decline in the performance of the decision logic; and

[0192] At least the analytical output should be generated based on the results of the predictive analysis.

[0193] The monitoring system can be operational so that the operational data can include information about the decision outputs of the decision logic during the field operation of the controller.

[0194] The monitoring system can operate such that predictive analytics can include predicting future run points and predicting the decision outputs of the decision logic used to predict future run points.

[0195] The monitoring system is operational, enabling predictive analytics to be performed by the performance evaluation logic.

[0196] The monitoring system can operate to generate performance evaluation logic by generating and / or using data generated during the computer implementation process of creating decision logic prior to the field operation of the controller.

[0197] The performance evaluation logic can be trained using scenario batches, with each batch representing a time series.

[0198] The evolution of scene batches can be driven by one or more latent variables.

[0199] Latent variables (multiple latent variables) can be associated with actual variables that define the scenario in a defined way.

[0200] Latent variables (multiple latent variables) can be associated with actual variables that define the scenario in a random manner.

[0201] The monitoring system may operate alternatively or additionally to execute at least one predictor, wherein the output of the predictor is provided as input to the performance evaluation logic.

[0202] The monitoring system can run multiple predictors and multiple instances that execute performance evaluation logic, each of which generates the input to the performance evaluation logic.

[0203] The monitoring system can operate so that the predictors (multiple predictors) can receive running data as input and can generate predictions for future running points based on, for example, historical data.

[0204] The monitoring system can operate, enabling multiple predictors to generate predictions for multiple different time points and / or for multiple different environmental conditions.

[0205] The predictor (multiple predictors) can be a recurrent neural network.

[0206] The monitoring system can operate to control the human-machine interface (HMI) to output analytical outputs, which include information about the controller's past, current, and / or predicted future performance during field operation.

[0207] The monitoring system can operate so that if time-related evolution indicators do not meet future controller performance criteria based on performance metrics, alarms, or other analytical outputs can be automatically generated.

[0208] According to another aspect of the invention, a method is provided for monitoring the performance of the decision logic of a controller of an industrial automation control system (IACS), particularly a controller of a power system, during field operation of the controller. This method is executed by at least one integrated circuit of the monitoring system and includes:

[0209] Receive operational data collected during controller field operation;

[0210] Analyzing operational data to evaluate the performance of the decision logic includes identifying potential root causes of poor decision logic performance and / or implementing root cause solutions to improve the decision logic during field operation; and

[0211] The analysis output is generated based on the analysis results.

[0212] Operational data can include information about the decision outputs of the decision logic during the field operation of the controller.

[0213] The analysis output can include information about the possible root causes of poor performance of the decision logic.

[0214] The analysis output can include information about root cause solutions.

[0215] The monitoring system can execute performance evaluation logic.

[0216] Instructions for identifying the root cause of poor performance of decision logic during field operation and / or generating root cause solutions during field operation may include identifying which scenario(s) parameters in the actual operating scenario need to be modified to make the decision logic meet performance metric-based criteria.

[0217] Indications for identifying the root cause of poor performance of decision logic during field operation and / or generating root cause solutions during field operation may include determining the offset in the scenario parameter space by the monitoring system so that the performance of the decision logic meets the performance metric-based criteria.

[0218] Determining offsets in the scene parameter space can include performing constrained optimizations.

[0219] Constrained optimization can use an objective function as the offset standard.

[0220] Constrained optimization can use performance evaluation logic to determine whether the constraints are met.

[0221] According to another aspect of the invention, a monitoring system is provided for the decision logic of a controller in an industrial automation control system (IACS), particularly a controller for a power system, during field operation of the controller. This monitoring system includes at least one integrated circuit capable of:

[0222] Receive operational data collected during controller field operation;

[0223] Analyzing operational data to evaluate the performance of the decision logic includes identifying potential root causes of poor decision logic performance and / or implementing root cause solutions to improve the decision logic during field operation; and

[0224] The analysis output is generated based on the analysis results.

[0225] The monitoring system can be operational so that the operational data can include information about the decision outputs of the decision logic during the field operation of the controller.

[0226] The monitoring system can be operational so that the analysis output can include information about the possible root causes of poor performance of the decision-making logic.

[0227] The monitoring system can be operational, allowing the analysis output to include information on how decision-making logic can be improved.

[0228] The monitoring system can be run to execute performance evaluation logic.

[0229] The monitoring system can be operational to identify the root cause of poor performance of decision logic during field operation and / or implement root cause solutions. Improving decision logic during field operation may include identifying which scenario(s) parameters in the actual operating scenario need to be modified so that the decision logic meets performance metric-based standards.

[0230] The monitoring system can be operational to identify the root cause of poor performance of the decision logic during field operation and / or generate instructions for improving the decision logic during field operation. This may include the monitoring system determining the offset of the scenario parameter space so that the performance of the decision logic meets the performance metric-based criteria.

[0231] The monitoring system can operate to determine the offset in the scene parameter space, including performing constrained optimizations.

[0232] The monitoring system can be operational, allowing constrained optimization to use an objective function as the offset standard.

[0233] The monitoring system is operational, allowing constrained optimizations to use performance evaluation logic to determine whether the constraints are met.

[0234] Industrial automation control systems, particularly power systems, include: a controller that runs decision logic to determine which control actions must be taken; and a monitoring system, according to an embodiment, for monitoring the decision logic during its operation.

[0235] The decision logic can run to generate a decision output by processing controller inputs, which include one, several, or all of locally available measurements, remotely captured measurements, and system-wide observations.

[0236] The controller can be a controller for distributed energy resources (DER).

[0237] The controller can be a local controller.

[0238] The local controller can be selected from the following groups: protection relays, generator control systems (e.g., governors and exciters / automatic voltage regulators), control systems for high-voltage direct current (HVDC) equipment, control systems for flexible alternating current transmission systems (FACTS) equipment, decision logic for switched capacitors and / or reactors, low-frequency unloading relays, and undervoltage unloading relays.

[0239] The controller can be a central controller.

[0240] The central controller can be selected from the following groups: EMS / DMS, such as decision logic for secondary frequency control, decision logic for generation redistribution, decision logic for utilizing demand flexibility, decision logic for reclosing operations, decision logic for secondary voltage control through reactive power redistribution, and decision logic for coordinating the actions of the entire power grid, such as when activating system protection schemes or remedial measures after an interference.

[0241] The controller can control controllable parameters, which can be selected from a group consisting of: active (P) and reactive (Q) power injection from one or more generators and / or energy storage systems; P / Q injection from DC links or back-to-back DC; Q injection from FACTS devices (such as SVC or STATCOM); power flow control devices; tap positions of one or more capacitor banks, reactors, and OLTCs; requirements; P / Q injection from electronic traffic platforms; and the status of switches and / or reclosers.

[0242] Various effects and advantages are achieved through the methods, apparatus and systems according to the embodiments.

[0243] The methods, apparatus, and systems according to the present invention allow for monitoring of decision logic executed by the controller of an Industrial Automation Control System (IACS) during field operation. The decision logic can be automatically monitored and evaluated during field operation of the controller. These methods, apparatus, and systems can be used even when the internal workings of the decision logic are not explicitly understood, for example, when the decision logic has been created using machine learning (ML) techniques.

[0244] The methods, apparatus, and systems according to the invention enable the evaluation of the effectiveness of the decision logic of controllers in various power grid devices and operating levels in an autonomous and continuous manner during operation (i.e., without interrupting operation).

[0245] The methods, apparatus, and systems according to the invention enable decision logic to remain compatible with changing power grid environments without compromising its efficiency / effectiveness. To this end, the methods, apparatus, and systems can automatically identify deficiencies in the decision logic. Typically, this occurs because the controller is no longer operating in the environment expected during the generation of decision logic 35.

[0246] The methods, apparatus, and systems according to the invention are applicable to control problems of a single local or central controller and to the evaluation of coordinated control logic that takes into account the interactions between various decision blocks of various power system components. Attached Figure Description

[0247] The subject matter of the invention will be explained in more detail with reference to the preferred exemplary embodiments shown in the accompanying drawings, wherein:

[0248] Figure 1 This is a schematic diagram of a system that includes a monitoring system.

[0249] Figure 2 This is a schematic diagram of a system including a monitoring system according to an embodiment.

[0250] Figure 3 This is a schematic diagram of the system, in which the monitoring system and the controller are provided separately.

[0251] Figure 4 This is a schematic diagram of the system, in which the monitoring system is integrated into the controller.

[0252] Figure 5 This is a flowchart of the method.

[0253] Figure 6 This is a flowchart of the method.

[0254] Figure 7 It is a signal flow diagram.

[0255] Figure 8 This is a flowchart of the method.

[0256] Figure 9 This is a block diagram of the system.

[0257] Figure 10 It is a diagram illustrating the operation of the monitoring system.

[0258] Figure 11 This is a block diagram of the monitoring system.

[0259] Figure 12 This is a flowchart of the method.

[0260] Figure 13 This is a signal flow diagram illustrating the training of the performance evaluation logic.

[0261] Figure 14 This is a schematic diagram showing the training scene space.

[0262] Figure 15 It is a diagram that illustrates the scenario generated for training decision logic or for training performance evaluation logic.

[0263] Figure 16AThis is a diagram illustrating the scenario generated for training decision logic.

[0264] Figure 16B This is a diagram illustrating the scenario generated for training performance evaluation logic.

[0265] Figure 17 This is a flowchart of the method.

[0266] Figure 18 This is a diagram illustrating the implementation of the machine learning model for performance evaluation logic.

[0267] Figure 19 This is a flowchart of the method.

[0268] Figure 20 It is a diagram illustrating the operation of the monitoring system.

[0269] Figure 21 It is a diagram illustrating the operation of the monitoring system.

[0270] Figure 22 It is a diagram illustrating the operation of the monitoring system.

[0271] Figure 23 This is a flowchart of the method.

[0272] Figure 24 This is a flowchart of the method.

[0273] Figure 25 This is a schematic diagram of the scene parameter space.

[0274] Figure 26 It is a diagram illustrating the operation of the monitoring system.

[0275] Figure 27 It is a diagram illustrating the operation of the monitoring system. Detailed Implementation

[0276] Exemplary embodiments of the invention will be described with reference to the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements. Although some embodiments will be described in the context of power generation systems, power transmission systems, or power distribution systems, the methods and apparatus described in detail below can be used in a variety of systems.

[0277] Unless otherwise specified, the features of the embodiments may be combined with each other.

[0278] According to embodiments of the present invention, a monitoring system is deployed to monitor and evaluate the performance of the decision logic of controllers in industrial automation control systems (IACS) such as power systems.

[0279] While the disclosed technology is applicable to various controllers in industrial or power systems, it is described in this application in the context of the operation of transmission and distribution networks, as well as smart sites or smart cities, but is not limited thereto.

[0280] As used herein, the term "decision logic" broadly encompasses logic executed by a controller. Decision logic can be, but is not limited to, control logic that produces control outputs (e.g., control commands) as the result of its execution. Protection functions in power systems (e.g., generation or transmission systems) are examples of such decision logic, but are not limited to them. For illustration, the techniques disclosed herein can be used to generate decision logic for digital protection relays, but are not limited to them.

[0281] As will be explained in detail below, during the performance evaluation performed during the field operation of the decision logic, the monitoring system can use information generated or otherwise used during the computer implementation of the decision logic of the generator controller.

[0282] The monitoring system enables the evaluation of the effectiveness of the decision logic of controllers across various industrial equipment (such as power system equipment) and / or at various operating levels. Performance evaluation can be performed autonomously by the monitoring system. The monitoring system can perform performance evaluations continuously during the field operation of the controller, without interrupting the operation of the power system or industrial system.

[0283] Figure 1 and Figure 2 This is a schematic diagram of a system including a monitoring system according to an embodiment.

[0284] The system includes one or more controllers 31, 32, and 33, collectively referred to as controllers. Controllers 31, 32, and 33 may each operate to perform functions such as protection functions in response to signals from sensors, merging units, intelligent electronic devices (IEDs), or other devices providing data related to the operation of the IACS, power generation system, transmission system, or distribution system. For illustration, one or more of controllers 31, 32, and 33 may be digital protection relays that determine whether a circuit breaker (CB) trips and whether the trip is immediate or delayed.

[0285] The decision logic executed by one or more of controllers 31, 32, and 33 can be computer-generated. The decision logic executed by one or more of controllers 31, 32, and 33 can each be a trained machine learning (ML) model. Decision logic 35 can be automatically generated by a decision logic generator and can be deployed to controller 31 for execution during its field use.

[0286] As used herein, the term "decision logic" can specifically refer to the controller's determination of which control action to take in response to signals provided by one or more data sources (sensors, merging units, etc.) when executed by controllers 31, 32, 33 of the power system (or another IACS).

[0287] The decision logic is specifically designed for the controller in which it is deployed and the functions (multiple functions) performed by that controller. If a controller performs multiple different functions, multiple different decision logics can be deployed within that controller.

[0288] Typically, the first controller 31 may have first decision logic deployed therein. The first monitoring system can monitor the first decision logic.

[0289] The second controller 32 may have a second decision logic deployed therein, which may operate differently from the first decision logic. For example, the decision logic inputs and / or outputs of the second decision logic may differ from those of the first decision logic. Alternatively or additionally, even when the first and second decision logics receive the same decision logic inputs and / or generate the same type of decision logic output signals, the decision boundaries of the second decision logic may differ from those of the first decision logic. Alternatively or additionally, the architecture of the second decision logic (e.g., the number of layers, nodes, and / or weights of the artificial neural network (ANN) used to operate as the second decision logic) may differ from the architecture of the first decision logic.

[0290] The third controller 33 may have a third decision logic deployed therein, which may operate differently from the first and second decision logics. For example, the decision logic inputs and / or outputs of the third decision logic may differ from those of the first and second decision logics. Alternatively or additionally, even if the first, second, and third decision logics receive the same decision logic inputs and / or produce the same type of decision logic output signals, the decision boundaries of the third decision logic may differ from those of the first and second decision logics. Alternatively or additionally, the architecture of the third decision logic (e.g., the number of layers, nodes, and / or link weights of the artificial neural network (ANN) used as the third decision logic) may differ from the architectures of the first and second decision logics.

[0291] To ensure appropriate response, various controllers31, 32, 33 can be deployed at all hierarchical levels of the power system. Some controllers can be local, meaning they are physically located close to the power system components. Other controllers can be central controllers, where the decision-making logic is computed at (sometimes a regional) control center, substation, or cloud. The techniques disclosed herein (where monitoring systems are deployed as described more fully herein) are applicable to both local and central controllers.

[0292] The controller 31, in which the decision logic 35 is deployed, may include at least one integrated circuit. The at least one integrated circuit may include a processor, microprocessor, controller, microcontroller, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or any combination thereof, for executing the decision logic 35.

[0293] The controller in which the decision logic 35 is deployed has an input interface 36. The inputs to the decision logic 35 can be received at and / or derived from input data and / or input signals received at the input interface 35.

[0294] The controller that deploys decision logic 35 has an output interface 37. The output of decision logic 35, or output commands generated based on it, can be output through output interface 37. Output interface 37 can be activated to trigger control actions on relevant primary or secondary power system components. Output interface 37 can be coupled to a central control system or central control center. For illustration, output interface 37 can be coupled to a substation control center or regional or national control center to provide it with operational data related to controller 31.

[0295] Controllers 31, 32, and 33 may include one or more local controllers in which decision logic 35 is deployed. The local controllers monitored by the monitoring system according to the invention may be selected from the group consisting of: protective relays, generator control systems (e.g., governors and exciters / automatic voltage regulators), control systems for high-voltage direct current (HVDC) equipment (multiple HVDCs), direct current (HVDC) equipment (multiple direct current (HVDC) devices), control systems for flexible alternating current transmission system (FACTS) equipment (multiple FACTS devices), decision logic for switched capacitors and / or reactors, underfrequency unloading relays, and undervoltage unloading relays, but are not limited thereto.

[0296] Controllers 31, 32, and 33 may include one or more central controllers, in which decision logic 35 is deployed. The central controller monitored by the monitoring system may be selected from the group consisting of: energy management systems (EMS) / distribution management systems (DMS), such as decision logic for secondary frequency control, decision logic for generation redistribution, decision logic for the use of demand flexibility, decision logic for reclosing operations, decision logic for secondary voltage control through reactive power redistribution, and decision logic for cross-grid coordinated actions, such as in cases where system protection schemes or remedial measures typically need to be activated after an disturbance, but not limited to these.

[0297] Controllable quantities can be selected from the following groups: active (P) and reactive (Q) power injections from all generators and energy storage systems; P / Q injections from DC links or back-to-back DC; Q injections from FACTS devices such as static VAR compensators (SVCs) or static synchronous compensators (STATCOMs); power flow control devices; tap positions of one or more capacitor banks, reactors, and OLTCs; requirements; P / Q injections from electronic traffic platforms; and the status of switches and / or reclosers, but not limited to these.

[0298] Components such as energy storage devices, electric vehicles, and various distributed energy resources (DERs) can be controlled locally and / or centrally (e.g., aggregated use of DERs or control of electric vehicle fleets, providing various grid controllability services). The technologies described herein are applicable to, but not limited to, monitoring controllers for DERs and / or energy storage devices.

[0299] Decision logic 35 can be any of the following, but is not limited to: (i) logic in the control system of FACTS or HVDC equipment or generator; (ii) protection logic in relays; (iii) decision logic for switching capacitors / reactors or for unloading; (iv) decision logic at the SCADA / EMS / DMS level, such as for frequency / voltage control, reclosing operations, generation rescheduleing or demand flexibility; (v) decision logic of EMS / BMS at the site (building, plant, microgrid, etc.); (vi) control logic of converters connected to power sources (such as PV or energy storage, etc.).

[0300] Decision logic 35 adopts a decision in response to inputs. Inputs are obtained through system observables. Depending on the context, observables can be local (e.g., voltage and current on a bus connecting devices in a Flexible AC Transmission System (FACTS), regional, or system-wide (e.g., synchronous phasor measurements from a set of network buses). In addition to electrical measurements, observables received as inputs by the decision logic can include device states (e.g., switch states or various control setpoints), or more generally, dynamic network topology. Observables received as inputs by the decision logic can include inputs from outside the power system, such as observed or predicted weather, electricity prices, date, time, traffic conditions, etc. Controllers 31, 32, and 33 can be equipped with more than one set of settings, allowing the controller to automatically switch between these settings in response to changes in observed environmental conditions.

[0301] Typically, local controllers make decisions based on locally available measurements, while central controllers make decisions based on system-wide observability and control actuators throughout the power system. However, in some cases, local controllers operate based on input signals from remote locations (multiple remote locations) within the system, or initiate system-wide control actions based on local measurements. An example of the first case is the actuation of a FACTS controller for electromechanical oscillation damping (located where it has the most effective effect) based on remote measurements (located where they provide the best observability). An example of the second case is mitigating line / transformer congestion through generator reschedules. The techniques for monitoring the decision logic disclosed herein are applicable to all these cases.

[0302] The decision logic 35 for controllers 31, 32, and 33 is developed during the commissioning of the new controller. During the design phase, various conditions in the power grid (e.g., faults, switching events, load changes, etc.) can be simulated to evaluate and improve the performance of the control logic. When implemented in a computer-based manner, such as by training a machine learning (ML) model to generate the decision logic 35, techniques for monitoring the decision logic are suitable for the application.

[0303] Once operational, the performance of controllers 31, 32, and 33 may deteriorate. This can be due to a variety of reasons. For example, performance may decline when the environment in which the controllers were designed changes significantly. This can occur when there are changes in power system topology, new power system components, and / or new grid specifications. Controllers 31, 32, and 33 may face different conditions than those they were designed for, leading to inefficient or irrelevant decisions or actuations in response to events. Furthermore, understanding and coordinating the interoperability of controllers 31, 32, and 33 at different locations in the network (e.g., traditional generator and converter interface components) is not straightforward.

[0304] The monitoring system 20 can monitor the decision logic 35 of controllers 31, 32, and 33 during the field operation of the controllers. Deterioration of the decision logic 35, which can be determined based on one or more key performance indicators or another performance metric, can be detected by the monitoring system 20. The monitoring system 20 may also be equipped with mechanisms for predicting future changes in system operating conditions and / or identifying changes in the power system or industrial system that may lead to poor performance of the decision logic 20, as will be explained more fully herein.

[0305] Different monitoring systems can be deployed for different decision logics (e.g., decision logics in controllers 31, 32, and 33). Each monitoring system can be associated with a different decision logic deployed in controllers 31, 32, and 33 of the system. Each monitoring system can be specifically designed to monitor the performance of its associated decision logic during field operation and / or perform root cause identification and / or root cause solutions for poor performance during field operation.

[0306] Different monitoring systems (especially the different performance evaluation logics deployed within them) used for different decision logics can receive different inputs (also known here as scene signatures) captured online during field operation, and can process the inputs differently according to the decision logic associated with the monitoring system and / or generate different outputs according to the decision logic associated with the monitoring system.

[0307] The decision logic 35 monitored by the monitoring system 20 may be decision logic generated in a computer-implemented manner prior to its deployment for field use. Data generated or otherwise used during the generation of the decision logic 35 may be stored in one or more data storage devices 40. This data is generated or otherwise used when the decision logic 35 is created, i.e., before the decision logic 35 is run in the field. Therefore, for simplicity, the data generated or otherwise used during the generation of the decision logic 35 will also be referred to herein as "pre-run data".

[0308] As will be explained more fully herein, monitoring system 20 can utilize "pre-run data" to monitor the performance of decision logic 35. This can be accomplished in various ways. Monitoring system 20 can retrieve pre-run data during field operation of controller 31 and can use the pre-run data in combination with operational data collected during field operation of controller 31 to evaluate the performance of decision logic 35, such as... Figure 1 As shown.

[0309] Alternative or additional land, such as Figure 2As shown, the monitoring system 20 can execute performance evaluation logic 27 to evaluate the performance of the decision logic 35. The performance evaluation logic 27 may be logic generated using pre-run data in the data storage devices (multiple data storage devices) 40, but this logic can perform the performance evaluation autonomously without accessing the pre-run data in the data storage devices (multiple data storage devices) 40 during the field operation of the controller 31.

[0310] The monitoring system 20 can receive one or more performance metrics to measure the performance of the decision logic 35. Performance metrics can define a set of key performance indicators (KPIs). This performance metric, or a set of performance metrics, can be used to evaluate the process.

[0311] The monitoring system 20 may optionally receive models of the power system topology, power system components, and / or various primary and / or secondary devices in the portion of the power system associated with the monitored controller. This information, which may specifically include topology information, may be received via a user interface (UI) 28. Alternatively or additionally, the monitoring system 20 or a network topology analysis system may process construction descriptions (e.g., substation construction description (SCD) files) to determine the power system topology, identify power system components, and / or various primary and / or secondary devices in the portion of the power system associated with the monitored controller.

[0312] The monitoring system 20 can collect and store power system data related to the controller 31 it monitors. Examples of such data include: (i) local data, such as voltage and current signals on the bus where the controller is located or from adjacent buses; (iii) data from system-wide measurements; and (iv) data from the EMS / DMS, such as unit combination results, demand and generation forecasts for the next time step.

[0313] The monitoring system 20 can perform model determination / calibration based on the collected data. A communication framework can be used with other controllers 32, 33 within a contiguous range of controller 31, allowing multiple monitoring systems 20 to exchange models and data.

[0314] Monitoring system 20 performs analysis to evaluate decision-making logic using simulation or data-driven methods and key performance indicators (KPIs). The implementation of this analysis is described in more detail here.

[0315] If the decision logic 35 of controller 31 does not perform well enough, as determined by performance metrics, the monitoring system 20 may take appropriate action. This may involve signaling to human experts. If the performance of the decision logic 35 of controller 31 does not meet predefined criteria, optionally based on performance metrics, the monitoring system 20 may operate to generate alarms, warnings, or other outputs via a human-machine interface (HMI).

[0316] Alternatively or additionally, monitoring system 20 may operate to take control actions. For example, if the performance of decision logic 35 of controller 31 does not meet predefined criteria that may optionally be based on performance metrics, monitoring system 20 may schedule downtime, trigger maintenance personnel deployment, trigger preventative control actions, or take other actions.

[0317] Control actions may include redesigning and re-deploying decision logic 35 based on behavior observed during field operation of controller 31. Monitoring system 20 may operate to interact with the decision logic generator system, enabling the autonomous generation of decision logic for deployment.

[0318] The monitoring system 20 can operate to determine and optionally output information about possible root causes or explanations of the reasons for performance degradation of the decision logic 35, as explained in more detail below.

[0319] Monitoring system 20 can be an artificial intelligence (AI) monitoring system.

[0320] To evaluate the operation of decision logic 31, monitoring system 20 can use the following items:

[0321] - Data generated during the creation phase of decision logic 35 of controller 31, which is monitored by monitoring system 20. This data can be used in various ways. For illustration, the data can be accessed continuously, i.e., on a sustained basis, during the operation of monitoring system 20 and the field operation of controller 31. Alternatively or additionally, monitoring system 20 can execute performance evaluation logic based on the data generated during the creation phase of decision logic, without necessarily accessing the data generated during the creation phase of field operation of both monitoring system 20 and controller 31.

[0322] - Data collected through field measurements after controller deployment (i.e., online, operational data). This data may include inputs received by controller 31, which executes decision logic 35, at input interface 36. This data may also include controller outputs output by controller 31, which executes decision logic 35, at output interface 37.

[0323] The monitoring system 20 can operate during the field operation of the controller 31, thereby utilizing information generated during the decision generation phase of the decision generation logic 35 to evaluate the controller's performance during field operation. This can be done in real time.

[0324] The monitoring system 20 can continuously evaluate whether the operating conditions observed during field operation are part of the scenario in the generation phase of the controller and can take control actions based on this.

[0325] The monitoring system 20 can be generated as part of the combined generation process of the decision generation logic 35 and the performance evaluation logic of the evaluation decision logic, prior to the field operation of the controller 31, as shown in reference. Figure 2 This will be explained in detail below.

[0326] The monitoring system 20 can not only infer the quality of performance, but also infer the potential causes of performance degradation in the event of observed performance degradation.

[0327] The monitoring system 20 can be implemented by a computing device or a distributed computing system. The monitoring system 20 includes at least one integrated circuit 21. The at least one integrated circuit 21 may include a processor, microprocessor, controller, microcontroller, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or any combination thereof.

[0328] The monitoring system 20 may have a user interface (UI) 28. The user interface 28 may include an optical output device. The monitoring system 20 can be used to receive information about performance metrics through the user interface 28. The information about performance metrics may specify one or more Key Performance Indicators (KPIs). The information about performance metrics may specify several KPIs and their relative weights, which are used to quantitatively determine the performance of the decision logic 35.

[0329] The monitoring system 20 can generate output based on the analysis of the performance of the decision logic 35. The output may include one or more of the following:

[0330] - Information regarding one or more KPIs of the decision logic 35 determined during the field operation of controller 31,

[0331] - Alarms, warnings, or other information derived therefrom, and

[0332] - Control commands or other control signals that trigger actions, such as the automatic regeneration or update of decision logic 35.

[0333] The monitoring system 20 may have at least one interface 26 for receiving data during field operation of the controller 31, wherein the decision logic 35 is deployed in the controller 31. The data received through at least one interface 26 may include:

[0334] - Information about the input processed by the decision logic 35; this information can be received by the monitoring system 20 from the controller 31 on which the decision logic 35 is deployed and / or from other system devices via a push or pull mechanism.

[0335] - Optionally, information about the decision adopted by decision logic 35 can be received by monitoring system 20 from controller 31 on which decision logic 35 is deployed and / or from other system devices via push or pull mechanisms.

[0336] It should be noted that evaluation of decision logic 35 can be performed without requiring knowledge of the decisions made by decision logic 35. For illustration, monitoring system 20 can compare the operating scenarios encountered by decision logic 35 during field operation of controller 31 with those scenarios it was trained on. Alternatively or additionally, monitoring system 20 can compare the grid topology encountered by decision logic 35 during field operation of controller 31 with the grid topology for which decision logic 35 was trained. Alternatively or additionally, monitoring system 20 can compare the specifications of secondary or primary devices that controller 31 interacts with during field operation with the specifications used when training decision logic 35. This allows for the identification of potential deterioration without using the decisions output by decision logic 35 or the control commands output by controller 31 based on those decisions.

[0337] The monitoring system 20 can also retrieve and use information about decisions made by the decision logic 35 during the field operation of the controller 31, and can use this information for performance evaluation.

[0338] Controllers 31, 32, and 33 can provide operational data to the monitoring system 20 via the communication network 19. The operational data is used to perform performance evaluations and optionally stored in a data storage device. The operational data may include information about scenarios encountered by the decision logic 35 during field operation of controller 31. The operational data may also include information about inputs received and / or generated by controllers 32 and 33 other than controller 31, where the decision logic 35 is deployed. This information about the inputs received by controllers 32 and 33 and / or the outputs generated by controllers 32 and 33 allows the monitoring system 20 to draw conclusions about the power system topology, the scenarios encountered by the decision logic 35 during field operation of controller 31, and / or the functionality of primary and secondary devices associated with controllers 32 and 33 other than controller 31, where the decision logic 35 is deployed.

[0339] The monitoring system 20 can utilize data used during the computer implementation of the decision logic 35 (referred to herein as "pre-run data") to analyze the performance of the decision logic 35 during operation.

[0340] Decision logic 35 can be generated during a process that includes one or more machine learning (ML) models. Monitored ML can be used to train the decision logic ML model. The decision logic ML model can include artificial neural networks (ANNs) or multiple ANNs.

[0341] As used in this article, “training an ML model” refers to the process of determining parameter values ​​for the parameters of an ML model. For example, training an ML model may include determining the weights of an ANN.

[0342] The process of generating decision logic 35 may involve selecting from multiple different candidate decision logic ML models. These different decision logic ML models may differ in their ML construction (e.g., ANNs with different numbers of hidden layers, etc.). The process of generating decision logic 35 may involve selecting a trained decision logic ML model from multiple trained candidate decision logic ML models based on their performance in the face of a set of test scenarios.

[0343] Information about the decision logic ML model 41 evaluated during the generation of decision logic 35 can be stored in data storage device 40. When analyzing decision logic 35 during the field operation of controller 31, monitoring system 20 can operate based on information about decision logic ML model 41. This can be accomplished by retrieving information about decision logic ML model 41 while monitoring system 20 monitors decision logic 35 during the field operation of controller and / or by using information about decision logic ML model 41 when establishing performance evaluation logic for monitoring system 20.

[0344] The process of generating decision logic 35 may involve subjecting the decision logic ML model to multiple test scenarios and determining their performance. Multiple test scenarios may include definitional inputs to candidate decision logic ML models during the process of generating decision logic 35. Test scenarios may be set according to the specifications of controller 31 (e.g., based on the power system topology, the functions controller 31 must be able to perform, and other controllers 32, 33 of the power system with which controller 31 interacts in field operation). Test scenarios may differ from each other regarding simulated power system parameters (e.g., voltage, current, phasor, or impedance measurements that may be for three-phase simulation). Alternatively or additionally, test scenarios may differ from each other regarding changes in dynamic power system topology (e.g., changes caused by circuit breaker (CB) tripping or reclosing). Alternatively or additionally, test scenarios may differ from each other in terms of power demand and / or generation curves (e.g., this may depend on different simulated weather conditions). Alternatively or additionally, test scenarios may differ from each other regarding simulated events, such as failures of DER generator units, transmission or distribution line failures, or other possible events.

[0345] Test scenarios may include a set of exogenous test scenarios that can be based on historical data and / or defined by human experts.

[0346] The test scenario can be a computer-generated scenario. For example, when Generative Adversarial Network (GAN) technology is used to generate decision logic 35 and scenario generation ANN during the generation of decision logic 35, the test scenario generated by the discriminator ANN can be included in test scenario 42 stored in data storage device 40.

[0347] Information about scenario 42 evaluated during the generation of decision logic 35 can be stored in data storage devices (multiple data storage devices) 40. Monitoring system 20 can analyze decision logic 35 based on information about scenario 42 during the field operation of controller 31. This can be accomplished by retrieving information about scenario 42 while monitoring decision logic 35 during the field operation of controller 31 and / or by using information about the scenario at the time of establishing performance evaluation logic for monitoring system 20.

[0348] The process of generating decision logic 35 may involve monitoring the decisions adopted by the candidate decision logic ML models. The decisions adopted by the candidate decision logic ML models may be control actions. This can be done for each recorded decision in various candidate decision logic ML models, each associated with the scenario faced by the candidate decision logic ML model.

[0349] To illustrate, the candidate decision logic ML model used to generate decision logic 35 can be a classifier that responds to a set of inputs and outputs one of a set of predefined possible outputs. Different outputs can correspond to different control actions.

[0350] Information regarding the decisions 43 made by the candidate decision logic ML model during the generation of decision logic 35 can be stored in data storage device 40. Monitoring system 20 can operate based on information about decisions 43 when analyzing decision logic 35 during the field operation of controller 31. This can be accomplished by retrieving information about decisions 43 while monitoring decision logic 35 is running in the field and / or by using information about decisions 43 when establishing the performance evaluation logic of monitoring system 20.

[0351] The process of generating decision logic 35 may involve determining the performance of candidate decision logic ML models (multiple candidate decision logic ML models). This may involve calculating one or more Key Performance Indicators (KPIs). One or more KPIs can quantify performance. One or more KPIs may be selected manually or otherwise specified before starting the generation of decision logic 35. The one or more KPIs used in the process of generating decision logic 35 may, but do not have to, be the same as one or more KPIs determined by monitoring system 20 during the field operation of controller 31.

[0352] For example, the candidate ML model used to generate decision logic 35 could be a classifier that responds to a set of inputs and outputs, representing one of a predefined set of possible outputs. Different outputs can correspond to different control actions. KPIs can compare the control actions taken by the candidate ML model during the generation of decision logic 35 with historical labeled data or the decision boundaries derived from it.

[0353] Information regarding the KPIs 44 used by the candidate ML model in generating decision logic 35 can be stored in data storage devices (multiple data storage devices) 40. Monitoring system 20 can analyze the KPIs 44 information of decision logic 35 during the field operation of controller 31. This can be accomplished by retrieving information about KPIs 44 while monitoring decision logic 35 is running on the controller and / or by using the information about KPIs 44 when establishing the performance evaluation logic of monitoring system 20.

[0354] Regardless of whether the monitoring system 20 accesses the data storage device 40 during the field operation of the controller 31, the monitoring system 20 can achieve this in various ways.

[0355] The monitoring system 20 can run on the computing infrastructure of the equipment / actuators in the field (e.g., STATCOM), on a computer or server in a central / regional control center or substation, or on a cloud-based computer or server infrastructure, such as... Figure 3 and Figure 4 As shown.

[0356] Figure 3 This is a schematic diagram illustrating a monitoring system 20 and controllers 31, 32 used in conjunction with components of a power distribution or generation system, in which decision logic is deployed. For illustration, a substation compartment may include switches QB1, QB2, QC1, AQ1, QB9, and QC9, transformers BC1 and BC2 for current sensing, and transformer BA1 for voltage sensing. Sensor devices are deployed to generate raw signals, which may optionally be provided to controllers 31 and 32 via a merging unit or other units that may optionally perform preprocessing.

[0357] like Figure 3 As shown, the monitoring system 20 can be provided separately from the controllers 31 and 32 and can be communicatively coupled to the controllers 31 and 32. For example, the monitoring system 20 can be deployed on a computer or server in a central / regional control center or substation, or on a cloud-based computer or server infrastructure.

[0358] The monitoring system 20 can execute different performance evaluation logic and / or access different data elements in the data storage devices (multiple data storage devices) 40, depending on whether the monitoring system is evaluating the performance of the first decision logic of the first controller 31 or the performance of the second decision logic of the second controller 32. Different performance metrics can be defined for different first controllers 31 and second controllers 32 and / or different decision logics.

[0359] Figure 4 This is a schematic diagram illustrating controllers 31 and 32 used in conjunction with components of a power distribution or generation system, in which decision logic is deployed. For illustration, a substation compartment may include switches QB1, QB2, QC1, AQ1, QB9, and QC9, transformers BC1 and BC2 for current sensing, and transformer BA1 for voltage sensing. Sensor devices are deployed to generate raw signals, which may optionally be provided to controllers 31 and 32 via a merging unit or other units that may optionally perform preprocessing.

[0360] like Figure 4 As shown, both controllers 31 and 32 include a dedicated monitoring system 20.

[0361] The first monitoring system 20 can be deployed on the first controller 31. The first monitoring system 20 can evaluate the performance of the decision logic of the first controller 31 during its field operation.

[0362] The second monitoring system 20' can be deployed on the second controller 32. The second monitoring system 20' can evaluate the performance of the decision logic of the second controller 32 during its field operation.

[0363] The first monitoring system 20 and the second monitoring system 20' can be different from each other and can be customized according to the functions performed by the controllers 31, 32 deployed therein.

[0364] Although Figure 3 and Figure 4 Only two controllers 31 and 32 are shown in this document, but the techniques disclosed herein are applicable to any number of N controllers and / or any number of M decision logics executed therefrom.

[0365] like Figure 4 As shown, the monitoring system 20 can be implemented on the computing infrastructure (e.g., STATCOM) of the field devices / actuators.

[0366] Figure 5 This is a flowchart of method 50. Method 50 can be derived from... Figures 1 to 4 The monitoring system of any one of them will execute automatically.

[0367] In step 51, operational data is collected. Operational data may be, or may include, data from an industrial or power system that influences the inputs to decision logic 35. Optionally, operational data may include information about the outputs of decision logic 35, which is related to the data from the industrial or power system influencing the inputs to decision logic 35. Operational data may include data from the industrial or power system that defines the topology and / or capabilities of components of the industrial or power system (e.g., in the form of a construction description).

[0368] In step 52, an analysis is performed to evaluate the performance of decision logic 35. This may include assessing whether decision logic 35 faces a scenario significantly different from the scenario used to train the decision logic (e.g., determined based on criteria applied to different input vectors). Step 52 may include determining whether the KPIs of decision logic 35 determined during field operation of controller 31 match the KPIs determined during the generation of decision logic 35. Step 53 may include determining whether the model used when generating decision logic 35 (e.g., a topology model or load flow model) matches the configuration and behavior of the power system.

[0369] The analysis in step 52 may include identifying possible root causes and / or predictive analysis of why decision logic 35 is not performing well, as described more fully herein.

[0370] In step 53, the monitoring system 20 can take action automatically. This can be done selectively, based on whether the analysis in step 52 indicates that the decision logic is no longer suitable, either currently or in the future, for performing the functions it was designed to perform.

[0371] The action in step 53 can take various forms: if the performance of the decision logic 35 of controller 31 does not meet a predefined criterion, the action may include generating an alarm, warning or other output for output via a human-machine interface (HMI), which may optionally be based on a received performance metric.

[0372] Alternatively or additionally, the action in step 53 may include a control action. The control action may be selected from a group consisting of scheduling downtime, triggering maintenance personnel deployment, triggering preventative control actions, or taking other actions if the performance of the decision logic 35 of controller 31 does not meet a pre-set standard, which may optionally be based on a received performance metric.

[0373] Alternatively or additionally, step 53 may include redesigning and re-deploying decision logic 35 based on behavior observed during field operation of controller 31. Monitoring system 20 may operate to interact with a decision logic generator system that autonomously generates decision logic for deployment.

[0374] Figure 6This is a flowchart of method 60. Method 60 can be derived from... Figures 1 to 4 The monitoring system of any one of them will execute automatically.

[0375] In step 61, UI 27 can be controlled to allow the user to specify a performance metric. This may include controlling UI 27 to allow input of a metric to measure the performance of decision logic 35. The metric can define a single KPI or a set of KPIs. When multiple KPIs are used, the received input may include weighting factors that define the relative importance of the KPIs. For example, performance can be quantified as a weighted sum of KPIs.

[0376] p =Σ i=1,…,n w i × KPI i (1)

[0377] KPIs are different key performance indicators. i These are the relevant weights. KPIs can be metrics. For example, when a KPI aims to quantify the similarity between the scenarios encountered by decision logic 35 and the scenarios used to train decision logic 35, the KPI can be defined as...

[0378] KPI =min J (2)

[0379] in This represents the input vector of the decision logic corresponding to the current on-site operating conditions, and This represents the input corresponding to the test scenario used when generating decision logic 35, where J is the label of the test scenario.

[0380] Other examples of performance KPIs may include, but are not limited to, minimizing electricity costs, increasing grid power transmission limits, ensuring stability, maximizing safety and reliability protection objectives, keeping voltage and current within limits, and maximizing economic benefits.

[0381] In step 62, the monitoring system 20 may optionally retrieve information about the power system or industrial system. For illustration, the monitoring system 20 may receive a first primary or statistical model of the power grid topology, power grid components, and / or various participants in the power grid portion associated with the monitored controller 31. Alternatively or additionally, the monitoring system 20 may receive and analyze a construction description (such as an SCD file) to determine relevant information about the power system or industrial system.

[0382] In step 63, monitoring system 20 may continuously collect power system or industrial system data related to the controller 31 it monitors. Examples of such data include: (i) local data, such as voltage and current signals on the bus where the controller resides or from adjacent buses; (iii) data from system-wide measurements; and (iv) data from EMS / DMS, such as unit combination results, demand and generation forecasts for the next time step. In step 63, the collected data may be stored. This may be done locally on monitoring system 20 or remotely, such as in a data storage system within a cloud-based computing infrastructure.

[0383] In step 64, monitoring system 20 may optionally perform model determination and / or calibration based on the data collected in step 63. Monitoring system 20 may optionally communicate with controllers 32, 33 (or their associated monitoring systems) instead of controller 31, in which monitored decision logic 35 for model determination and / or calibration is deployed.

[0384] In step 65, the monitoring system 20 can perform analysis to evaluate the decision logic. The monitoring system 20 may perform simulations and / or use data-driven methods in the analysis. The monitoring system can quantitatively determine performance based on KPIs. The implementation of the analysis steps will be described in more detail herein.

[0385] In step 66, the monitoring system 20 can control the HMI and / or take another action based on the analysis results. For example, if the decision logic 35 in the controller 31 is not performing well, the monitoring system 20 can generate an output for a human expert to warn them of this situation. The monitoring system 20 can also provide root cause solutions, i.e., propose reasons or explanations for why the performance of the decision logic has deteriorated. Alternatively or additionally, the monitoring system 20 can initiate an autonomous redesign process by activating the decision logic generator to regenerate the decision logic 35.

[0386] Figure 7 This is a block diagram illustrating the monitoring system 20 that interacts with the logic generation and / or modification device 73. The monitoring system 20 may be... Figures 1 to 4 20. Any one of the monitoring systems in the middle.

[0387] Decision logic 35 generates decision 71 during field operation. Decision 71 may include or be a control action.

[0388] Decision 71, adopted during field operation, triggers system response 72. System response 72 may include decisions (multiple decisions) adopted by other decision logics (multiple decision logics) besides decision logic 35. These other decision logics may include one or more other decision logics deployed in the same controller 31 as decision logic 35 and / or one or more other decision logics deployed in controllers 32, 33 other than controller 31. System response 72 may include observables such as current, voltage, phasor, impedance, synchronization phasor, or other observables. System response 72 may include changes in dynamic system topology (e.g., when decision 71 is a disconnect or reconnect command).

[0389] The monitoring system 20 can receive decisions 71 and system responses 72. The monitoring system 20 can evaluate the performance of the decision logic 20. Performance metrics can be evaluated to assess performance, optionally using one or more KPIs.

[0390] If the performance determined by the monitoring system 20 does not meet the performance criteria (for example, if the performance calculated based on performance metrics does not meet the performance threshold criteria), the monitoring system 20 may trigger the logic modification and / or generation unit 73 to modify the decision logic 35 and / or generate the modified decision logic 35. The modified decision logic 35 can be deployed to the controller 31 without replacing the controller 31.

[0391] Figure 8 This is a flowchart of method 80. Method 80 can be derived from... Figures 1 to 4 The monitoring system of any one of them will execute automatically.

[0392] In step 51, operational data is collected. The operational data may be, or may include, data from an industrial or power system that influences the inputs to decision logic 35. The operational data may optionally include information about the outputs of decision logic 35, relating to the data from the industrial or power system influencing the inputs to the decision logic. The operational data may include data from the industrial or power system that defines the topology and / or capabilities of components of the industrial or power system (e.g., in the form of a construction description).

[0393] In step 52, an analysis is performed to evaluate the performance of decision logic 35. This may include assessing whether decision logic 35 faces a scenario significantly different from the scenario used to train the decision logic (e.g., determined based on criteria applied to different input vectors). Step 52 may include determining whether the KPIs of decision logic 35 determined by controller 31 during field operation match the KPIs determined during the generation of decision logic 35. Step 53 may include determining whether the model used in generating decision logic 35 (e.g., a topology model or load flow model) matches the configuration and behavior of the power system.

[0394] The analysis in step 52 may include identifying possible root causes of why decision logic 35 is not performing well and / or predictive analysis of its performance, as described more fully herein.

[0395] In step 81, if the performance determined by the monitoring system 20 does not meet the performance criteria (for example, if the performance calculated based on performance metrics does not meet the performance threshold criteria), the monitoring system 20 may trigger the logical modification decision logic 35 and / or trigger the generation and deployment of the modified decision logic 35. The modified decision logic 35 can be deployed to the controller 31 without requiring the controller 31 to be replaced.

[0396] exist Figure 7 Box 73 in or Figure 8 The generation and deployment of the modified decision logic at step 81 can be executed by a self-running decision logic generator.

[0397] Figure 9 This is a block diagram illustrating the interaction between the monitoring system 20 and the decision logic generator.

[0398] The decision logic generator uses a set of 82 candidate decision logics. Set 82 can be a group of different decision logic ML models. For illustration, set 82 can include support vector machines, artificial neural networks with different numbers of hidden layers, or other decision logic ML models.

[0399] A simulation engine 83 is provided to simulate the behavior of several candidate decision logics in the set 82 in response to various scenarios 84 and / or using different models 85.

[0400] Scenario 84 can represent different operating conditions in a power system or an industrial system.

[0401] In response to the decision adopted by decision logic 35, model 85 can represent the behavior of controller 31, other controllers 32, 33, and / or other primary or secondary devices in the power system or industrial system. Various processing techniques can be used to simulate the impact of the decision adopted by the candidate decision logic on system quantities. For illustration, model 85 may include power flow simulation, numerical simulation of system response in the time and / or frequency domains, transient calculations, optimal power flow calculations, eigenvalue analysis, or other processing techniques.

[0402] Based on the performance of candidate decision logic 82 in the scenario simulated by simulation engine 83, one of the candidate decision logic 82 is selected for deployment 86.

[0403] Although Figure 9Not shown, but simulation engine 83 and evaluation engine 86 can execute optimization routines to train candidate decision logic. This may involve monitored learning using labeled data or gradient descent techniques, but is not limited to this.

[0404] In the optimization routine, the performance of the decision logic is evaluated by simulating a set of test scenarios. Models 85 of various system components (including controller 31 itself, and other controllers 32, 33 operating in the same power grid environment) are utilized. The performance of the candidate decision logic is quantified by applying a set of KPIs to the simulation results. This process can be repeated until convergence is achieved. Examples of performance KPIs may include, but are not limited to, minimizing electricity costs, increasing grid power transmission limits, ensuring stability, maximizing protection objectives for safety and reliability, keeping voltage and current within limits, and maximizing economic benefits.

[0405] The performance of the deployed decision logic 35 is monitored by the monitoring system 20 during its field operation. The analysis results performed by the monitoring system 20 can be fed back to the decision logic generator.

[0406] To illustrate, monitoring system 20 can determine whether the scenario faced by decision logic 35 during field operation is sufficiently similar to at least one of the scenarios 84 used during the generation of decision logic 35. If it is determined during field operation that the scenario is not similar to any of the previously used scenarios 84, information about the scenario (e.g., the input vectors encountered during field operation for decision logic 35) can be fed back to the decision logic generator.

[0407] Alternatively or additionally, the monitoring system 20 may determine whether the model 85 used to simulate the system response when generating decision logic 35 effectively reflects the actual system response behavior observed during the field operation of decision logic 35. Information regarding mismatches may be fed back to improve model 85.

[0408] Figure 10 This means that it can be done by Figures 1 to 4 The output graph generated by any of the monitoring systems 20 may, additionally or alternatively, modify the decision logic 35.

[0409] The monitoring system 20 can analyze the decision logic 35 during the field operation of the controller 31 to generate and output an indicator 91. Indicator 91 indicates whether the decision logic 35 has performance that meets performance criteria (for illustration, whether the performance calculated based on the performance metric meets a threshold comparison). Indicator 91 can be selected from a set of discrete values. Indicator 91 can be binary, indicating whether the performance is considered sufficient or insufficient. Indicator 91 can be selected from a set of three or more discrete values, indicating whether the performance is safe, requires attention, or is insufficient. Multiple threshold comparisons can be performed against the performance calculated based on the performance metric to determine which of the multiple discrete values ​​to output. Indicator 91 can also be taken from a continuous range of values. The continuous range of values ​​can represent a "healthy" or "degraded" index within a predefined interval, such as from 0 to 1, and can be calculated based on the performance metric.

[0410] Alternatively or additionally, monitoring system 20 may perform analysis of decision logic 35 during field operation of controller 31 to generate and output predicted time 92, which indicates when the decision logic may need to be revisited. Predicted time 92 may be calculated based on the temporal evolution of performance observed by monitoring system 20. Alternatively or additionally, one or more predictors may be used, which can be implemented using recurrent neural networks, as explained in more detail herein.

[0411] Alternatively or additionally, the monitoring system 20 may perform analysis of the decision logic 35 during the field operation of the controller 31 to generate and output information 93 that identifies possible root causes of poor performance of the decision logic. Information 93 may be determined by the monitoring system 20 by combining performance evaluation with constrained optimization to identify the most likely root causes of non-compliance with performance metric-based criteria, as explained in more detail herein.

[0412] Operating conditions that cause a decline in the performance of decision logic, whether they are currently existing (as shown in indicator 91) or predicting future points in time (as shown in prediction time 92), may have a variety of causes.

[0413] For example, the performance of the decision logic may be considered inadequate for any of the following non-exhaustive reasons:

[0414] - Exogenous factors affecting power system or industrial system infrastructure, such as demand (spatial distribution and / or temporal variation profile), weather conditions and / or other external factors;

[0415] - Factors residing in the infrastructure of power systems or industrial systems; examples include: new transmission and / or distribution lines, new generation and / or storage capacity, new decision-making devices (e.g., FACTS), and topology changes due to switching operations;

[0416] - Modification decision logic for devices (such as FACTS, relays, batteries, etc.), rather than the controller 31 that executes the monitored decision logic 35;

[0417] - Changes in performance metrics (e.g., new grid specifications or new electricity market rules).

[0418] The following describes different techniques that allow the monitoring system 20 to evaluate the performance of the decision logic 35 during field operation of the controller 31 and optionally in real time.

[0419] Monitoring system 20 is used to compare online data with data used offline to generate decision logic.

[0420] The monitoring system 20 can operate to compare data captured during the field operation of the controller 31 (i.e., data captured online) with data used in generating decision logic 35 (specifically, the generated data). An exemplary configuration will be referenced. Figure 1 , 10 And 11 are described.

[0421] Information created or otherwise used during the generation of decision logic 35 can be stored in storage device 40, such as... Figure 1 As shown. The monitoring system 20 is connected to the storage device 40 during the field operation of the controller 31.

[0422] Therefore, the following dataset from the process of generating decision logic 35 can be used for monitoring system 20:

[0423] - All models used to perform various simulations 41

[0424] - All simulated scenarios, test cases (e.g., power grid environment, topology) and / or events 42;

[0425] - Controller performance across all simulated use cases: 44.

[0426] This data can be used not only for the deployed decision logic 35, but also for candidate versions of the decision logic to be tested during the generation of decision logic 35.

[0427] This data can be used by the monitoring system 20 during the field operation of the controller 31. The monitoring system 20 can operate to compare the data observed during the field operation of the controller 31 with the data from the process of generating the decision logic 35. The monitoring system 20 can operate to assess whether the observed reality was taken into account during the process of generating the decision logic 35, and / or whether the controller 31 operates as expected during the process of generating the decision logic 35. As mentioned above, due to factors external to the power or industrial system, and / or due to factors that may be inherent in the power or industrial system, there are various reasons why the decision logic may behave differently from its expected behavior. Inappropriate behavior of the decision logic may be due to incorrect assumptions and / or simplifications made during the process of generating the decision logic 35, or due to environmental changes that occur during the life cycle of the controller.

[0428] Using operational data captured during field operation of controller 31 and pre-operation data stored in storage devices (multiple storage devices) 40, monitoring system 20 can be run to observe and evaluate the following:

[0429] a) Model: Monitoring system 20 can operate to evaluate the accuracy of the model used in generating decision logic 35. Monitoring system 20 can operate to perform real-time model validation, calibration, or determination. Monitoring system 20 can operate to compare observed signals with signals expected based on the model used in generating decision logic 35. Monitoring system 20 can operate to determine the difference between real-world behavior and the model used in generating decision logic 35. Such difference, if deemed significant, may indicate that modification of decision logic 35 may be necessary.

[0430] b) Scene The monitoring system 20 can be run to evaluate the coverage of the scenarios used in the process of generating decision logic 35. The monitoring system 20 can also be run to compare the observed running points with the running points used in the offline simulation of the process of generating decision logic 35. The comparison can be performed using statistical metrics, such as anomaly detection (detecting “anomalies” in the observed running conditions relative to the simulated running conditions) and / or density estimation methods (e.g., k-nearest neighbors, Gaussian processes, principal component analysis, hidden Markov models, neural networks such as autoencoders, but not limited to these).

[0431] c) Control ActionMonitoring system 20 can be run to evaluate the consistency of control actions observed during operation and simulated during the generation of decision logic 35 under given operating conditions. Monitoring system 20 can run during the field operation of controller 31 to create datasets, where each sample consists of system operating conditions and controller actions. Monitoring system 20 can be run to check the similarity between this online dataset and the dataset created during the generation of decision logic 35 for control actions under similar operating conditions. Inconsistencies indicate that decision logic 35 needs to be revisited. Alternatively or additionally, monitoring system 20 can be run to check the consistency of decisions made by decision logic 35 for various similar observed operating conditions during field operation.

[0432] d) KPI(s) The monitoring system 20 can be run to evaluate the consistency between real-time observations and the values ​​of performance metrics simulated during the generation of decision logic 35 for given operating conditions and control actions. Inconsistencies indicate the need to revisit decision logic 35.

[0433] By executing (a) and (b), the monitoring system 20 can operate to improve its understanding of the environment compared to the assumptions made during the generation of decision logic 35. By executing (c) and (d), the monitoring system 20 can evaluate the ability of the controller 31's decision logic 35 to perform in a real-world environment according to its specifications and / or KPIs. Therefore, not all of these functions need to be performed by the monitoring system 20, although all or several of these functions can be combined.

[0434] Figure 11 This is a block diagram of the monitoring system 20 according to an embodiment.

[0435] One or more ICs of the monitoring system 20 may be used to execute the model module 101, which is used to verify, calibrate and / or determine the model of the controller 31 that executes the decision logic 35, the model of other controllers 32, 33, and / or other primary or secondary devices in the system.

[0436] Alternatively or additionally, one or more ICs of the monitoring system 20 may operate to execute scenario module 102, which may be operated to evaluate whether the scenarios encountered during field operation (e.g., run points) are significantly different from all scenarios used when generating decision logic 35.

[0437] Alternatively or additionally, one or more ICs of the monitoring system 20 may operate to execute a decision evaluation module 103, which operates to evaluate whether the decisions made by the decision logic 35 during the field operation of the controller are consistent with the decisions expected when the decision logic 35 was generated.

[0438] Alternatively or additionally, one or more ICs of the monitoring system 20 may operate to execute the KPI module 104, which may be operated to evaluate whether the KPIs obtained by the decision logic 35 during the field operation of the controller are consistent with the KPIs expected when the decision logic 35 was generated.

[0439] Monitoring system 20 can run to execute performance evaluation logic.

[0440] The monitoring system 20 can execute the performance evaluation logic 27, either alternatively or additionally, by accessing the data used in generating the decision logic for online performance evaluation. Figure 2 As shown.

[0441] The performance evaluation logic 27 can be trained offline using data from data storage devices (multiple data storage devices) 40. During the field operation of the controller 31, the resource evaluation logic 27 may not require the monitoring system 20 to access the data in the storage devices (multiple storage devices) 40, such as... Figure 2 As shown.

[0442] The performance evaluation logic can be a robustness evaluator for the operation of the evaluation decision logic 35.

[0443] During the field operation of controller 31, performance evaluation logic 27 can run to monitor the power system or industrial system environment of controller 31. Performance evaluation logic 27 can run to continuously evaluate whether the observed environment requires modification of decision logic 35. The need for modification may be immediate, corresponding to a situation where decision logic 35 is no longer suitable for the power grid environment in which it operates, or it may not be immediate but subsequent, corresponding to the determination of a persistent trend (gradual change in the environment) that will eventually make decision logic 35 unsuitable.

[0444] The environmental monitoring of performance evaluation logic 27 is application-specific. The relevant environment depends on the inputs to the decision logic and may, but is not necessarily, limited to a geographical environment. For illustration, local controllers make decisions based on locally available measurements, while the central controller makes decisions based on system-wide observability and controls actuators throughout the power system. However, in some cases, local controllers operate based on input signals from remote locations (multiple remote locations) within the system, or initiate system-wide control actions based on local measurements. An example of the first case is the actuation of a FACTS controller for electromechanical oscillation damping (located where it has the most effective influence) based on remote measurements (located where they provide the best observability). An example of the second case is congestion mitigation of lines and / or transformers through generator reschedules. The relevant environment considered by performance evaluation logic 27 depends on which quantities influence the operation of the monitored decision-making logic 35.

[0445] The performance evaluation logic 27 can be a performance evaluation machine learning (ML) model, such as an artificial neural network (ANN). The performance evaluation logic 27 can perform a classification task. For this purpose, the output of the performance evaluation logic 27 can be a discrete value indicating the suitability of the controller (e.g., corresponding to different signs of red, orange, and green), or it can be a probability value indicating how confident the performance evaluation logic 27 is in its decision logic 35 regarding the need to revisit the monitored controller 31.

[0446] In addition to assessing the suitability of the decision logic 35 of the controller 31 for the current observation environment, the performance evaluation logic 27 can optionally determine the expected trend of performance degradation in the near future (e.g., the predictive controller will need to be revisited at some prediction time, e.g., within the next year).

[0447] Alternatively or additionally, performance evaluation logic 27 can be trained to provide insights into how decision-making logic 35 should be improved / modified.

[0448] The performance evaluation logic 27 is trained using data from storage devices (multiple storage devices) 40, which is used during the computer implementation of the decision generation logic 35. This is done before the performance evaluation logic 27 is run in the field.

[0449] Figure 12 This is the flowchart for method 110.

[0450] In step 111, decision logic 35 is generated. This can be accomplished using machine learning during computer implementation. The data used in generating decision logic 35 is stored as "pre-run data" in data storage devices (multiple data storage devices) 40.

[0451] In step 112, the data used in generating decision logic 35 is used to generate performance evaluation logic 27. The data used in generating decision logic 35, particularly the decision logic 35 obtained in this process, can be used to generate training data for training the performance evaluation ML model, which is then deployed as performance evaluation logic 27.

[0452] In step 113, training will be deployed as a performance evaluation ML model for performance evaluation logic 27. Training data for performance evaluation logic 27 may include test scenarios used in the process of generating decision logic 35. Training data for performance evaluation logic 27 may include additional test scenarios, which may be based on historical data, expert input, and / or can be generated computationally.

[0453] In step 114, performance evaluation logic 27 is deployed to monitoring system 20. During field operation, performance evaluation logic 27 monitors decision logic 35.

[0454] The techniques used to train performance evaluation logic 27, which can monitor the performance of decision logic 35 during field operation, will be described in more detail. Techniques that allow the performance evaluation logic 27 to be trained so that it can be used to provide guidance on how to improve decision logic 35 and / or predict future operating conditions that may lead to poor performance of decision logic 35 will also be described.

[0455] The performance evaluation logic 27 can be trained using conventional training techniques for decision logic ML models. These techniques are known to technicians but require appropriate training data. For this ML to be reliable and therefore valuable, the training dataset should be sufficiently rich, i.e., it should contain sufficiently representative samples of all situations that the ML model might encounter during field operation.

[0456] For reference Figures 13 to 18 As explained in detail, the training dataset used to train the performance evaluation ML model that will be deployed as performance evaluation logic 27 can be generated through offline simulation. Here, the term "simulation" should be interpreted in a general sense, including any one or any combination of the following: (i) numerical simulations in the time and / or frequency domains (e.g., power flow, short-circuit calculations, electromagnetic transients), (ii) solutions to optimization problems (e.g., optimal power flow, unit combination), or (iii) other computations that allow the creation of scenarios that can challenge the decision logic.

[0457] The training data used to train the performance evaluation ML model that will be deployed as performance evaluation logic 27 may include a first training dataset used when generating decision logic and an additional training dataset including scenarios not included in the first training dataset.

[0458] Figure 13 This is a diagram illustrating the generation of training data for training an ML model, which will be deployed as performance evaluation logic 27.

[0459] The decision logic generator 120 can be used to generate decision logic 35 using a first training dataset. The decision logic generator 120 may include a first scene generator 121, which can generate at least a portion of the first training dataset. The first training dataset may include supplementary training data, such as historical scenes and / or scenes specified by human experts.

[0460] The first training dataset includes a first set of test scenarios, which can be stored in a first historical database 122.

[0461] For illustrative purposes only, the first scene generator 121 can be another ML model that challenges the ML model trained to create decision logic 35. The first scene generator 121 and the other ML model trained to create decision logic 35 can be two adversarial ML models. The first scene generator 121 and the other ML model trained to create decision logic 35 can be a generator and discriminator network of a GAN.

[0462] The result of the process executed by the logic generator 120 is decision logic 35.

[0463] The challenge scenario generation module 125 is responsible for generating additional training data for training the ML model that will be deployed as the performance evaluation logic 27. The challenge scenario generation module 125 may include a second scenario generator 126. The second scenario generator 126 may be an ML model (e.g., an ANN) that challenges the decision logic 35. Note that the decision logic 35 is no longer modified because the second scenario generator 126 generates scenarios that cause the decision logic 35 to perform poorly. As will be explained below, various techniques can be used to ensure that the second scenario generator 126 generates scenarios that cause the decision logic 35 to perform poorly.

[0464] For illustration, the second scene generator 126 can be specifically configured such that it generates scenes outside the specification of the decision logic 35, so as to allow the performance evaluation logic 27 to evaluate the performance of the decision logic 35 outside its specification when it is being trained.

[0465] The second scene generator 126 can generate additional training data including scenes outside the specifications of decision logic 35. This additional training data can be stored in the historical database 127. Most of the scenes included in the additional training data can be outside the specifications of decision logic 35.

[0466] The first training data used to generate decision logic 35 can be combined with additional training data stored in historical database 127 to generate a larger second training dataset 128.

[0467] The second training dataset 128 includes the first training dataset used when generating the decision logic and an additional training dataset that includes scenarios not included in the first training dataset. The second training dataset 128 is used to train the performance evaluation logic 129.

[0468] The decision logic generator 120, the challenge scenario generation module 125, and the performance evaluation logic training 129 can be executed by one or more ICs in a computer, server, or distributed computing infrastructure.

[0469] The data in the first and second training datasets are labeled data, where the labels are based on the performance evaluation performed by the decision logic generator 120 and the challenge scenario generation module 125.

[0470] Therefore, in order to train the performance evaluation logic 27 to perform well, the autonomous decision logic generator 120 is supplemented by the subsystem 125, which can run to generate additional data that is not necessary for generating the decision logic 35 itself, but is used to train the performance evaluation logic 27.

[0471] Therefore, as referenced Figures 14 to 18 To explain in more detail, the required second training dataset 128 is obtained from two sources:

[0472] a. A scenario created by a second scenario generator 126 that challenges the decision logic 35 before it runs on-site and in which the performance of the decision logic is evaluated.

[0473] b. The scene created by the first scene generator 121 during the generation of decision logic 35.

[0474] As detailed below, the data mentioned in project (a) ensures that the second training dataset 128 contains scenarios that cause the decision logic 35 to perform poorly, while the data mentioned in project (b) ensures that the second training dataset 128 contains scenarios that correspond to the appropriate performance of the decision logic 35.

[0475] Challenge Scenario Generation

[0476] Decision logic 35 can be generated by adversarial logic, where the first scenario generator 121 acts as an opponent challenging the decision logic being trained. The first scenario generator 121 can provide the simulation system with scenarios to evaluate the performance of decision logic 35 during its training. The role of the first scenario generator 121 is to challenge the decision logic 35, which can be autonomously created by the control logic generator subsystem. Thus, the process of generating decision logic 35 is driven towards a well-functioning decision logic 35.

[0477] The successful first scenario generator 121 generates two types of scenarios: a) scenarios that the decision logic 35 is expected to face regularly (e.g., scenarios corresponding to normal power system operation), and b) rarely occurring but challenging scenarios for which the decision logic 35 should be robust. The deployed decision logic 35 needs to function well for both scenario types.

[0478] Based on this process of automatically generating decision logic 35 using adversarial logic, performance evaluation logic 27 can be generated in a computer-implemented manner before the decision logic 35 is run in the field.

[0479] To create the data required to provide performance evaluation logic 27, a second scenario generation subsystem, called challenge scenario generation system 125, is used. The goal of challenge scenario generation system 125 is to determine scenarios that determine the performance of challenge decision logic 35. Scenario generated by the second scenario generator 126 of successful challenge decision logic 35 is used together with other data to train performance evaluation logic 27 before its field run.

[0480] The difference between the second scenario generator 126 and the first scenario generator 121 is that the first scenario generator 121 can run to generate only legal scenarios, scenarios in which the designed decision logic 35 should operate in an appropriate manner, because the decision logic 35 is expected to face such scenarios during operation. The first scenario generator 121 can run to generate only scenarios within the specification of the system in which the controller 31 is used.

[0481] The output of the second scene generator 126 includes scenes outside the legal scene space, i.e., scenes that do not need to be considered when designing well-performing decision logic 35. The second scene generator 126 can be run to generate (at least) scenes within the specifications of a system in which controller 31 is not used.

[0482] The role of the second scene generator 126 is to determine potential future scenes. If materialized, the potential future scene will cause the decision logic 35 to malfunction.

[0483] This potential future scenario could be caused by changes in system topology, system hardware, and / or functional changes in primary or secondary equipment in power or industrial systems.

[0484] For example, if the design decision logic 35 is a STATCOM controller, the second scenario generator 126 can create a scenario corresponding to another power electronics-based device (e.g., another STATCOM with an SVC) existing in the electrical vicinity of the STATCOM, but for which no such device is actually (or planned) installed. Such scenarios are not even created by the first scenario generator 121 because they are not part of the STATCOM controller specification.

[0485] Figure 14 The scenario space is illustrated schematically. A first scenario set 131 is used to train decision logic 35. The first scenario set 131 can be generated by a first scenario generator 121 or can include scenarios generated by the first scenario generator 121. When deploying decision logic 35, the first scenario set 131 can consist entirely of scenarios within the specification of the system deploying decision logic 35.

[0486] The second scenario set 132 is specifically designed to successfully challenge the decision logic 35. The second scenario set 132 may be generated by the second scenario generator 126 or may include scenarios generated by the second scenario generator 126. When deploying the decision logic 35, the second scenario set 132 may include or may consist entirely of scenarios outside the specifications of the system deploying the decision logic 35.

[0487] Sets 131 and 132 can be merged to form a merged set, which is then used to train performance evaluation logic 27.

[0488] Figure 15 The illustration shows the construction of a scene generation system that can be used in the embodiments. When referring to... Figure 16A and 16B When describing some adjustments, Figure 15 The general construction can be used to generate the first scene set 131 and the second scene set 132.

[0489] The training scene generation module 140 may include a repository of exogenous scenes 141. Exogenous scenes 141 may include historical scenes, may be based on historical scenes, or may include scenes specified by human experts.

[0490] The training scene generation module 140 may include a scene generator 142 for generating new scenes. The scene generator 142 may be an adversarial ML model that challenges the decision logic 35 during training (in the process of generating the decision logic) or after training of the decision logic 35 has been completed (to generate additional scenes for training the performance evaluation logic 27).

[0491] Scene generator 142 may be an ML model that is updated during its operation, with the aim of improving the ability of scene generator 142 to challenge decision logic.

[0492] The training scenario generation module 140 may include a repository of challenge scenarios 143. Challenge scenarios 143 may include scenarios identified as challenging for decision logic 35.

[0493] Scenes can be selected from three sources 141, 142, and 143 and can be fed into the simulation system. A batch of 144 active scenes can be maintained.

[0494] Initially, a scene will be selected from exogenous scene 141.

[0495] During the training of the decision logic (and scenario generator 142, which is an adversarial logic that improves the decision logic), more scenarios created by scenario generator 142 are selected to challenge the decision logic 35 that is currently being trained.

[0496] During the design process, challenge scenarios 143 that trigger updates to decision logic 35 can be recorded. Scenarios to be revisited can be selected intermittently to ensure that decision logic 35 can still handle previously identified challenge scenarios as it evolves.

[0497] Selecting to revisit 143 from the scene database can be done in various ways. To illustrate, the scene generation module 140 can be trained to understand which scenes are most likely to influence the decision logic 35.

[0498] The role of scenario generators 121, 126, and 142 is to challenge the proactive decision-making logic 35, that is, to identify scenarios that will cause the decision-making logic 35 to perform poorly. The output of scenario generators 121, 126, and 142 is the value of all parameters assigned to define the scenario, such as demand and generation levels / modes, contingencies, faults, topology, etc. Such a set of parameters can be called a scenario vector.

[0499] ML models (such as ANNs) can be used to generate scene vectors, such as... Figure 18 As shown. This model can be called a scene generation ML model. Its output (i.e., scene vectors) can be continuous, enabling it to generate useful challenge scenes. This can be done in various ways.

[0500] The scene generation ML model can operate as a (potentially stochastic) function mapping from an input containing at least a description of the current decision logic 35 to a scene vector. The scene generation ML model is learned during the generation of decision logic, such that the generated scenes will be challenging scenarios where decision logic 35 predictably exhibits low performance. As part of an autonomous design process, the scene generation model is trained using the corresponding performance of previously simulated scenarios and decision logics, described by its architecture and parameter values ​​(such as weights of inter-node links or other parameters of the activation function). One objective of the training process is to converge to a scene generation ML model that can create challenging scenarios for a given decision logic 35. The parameter values ​​defining the scene generation ML model are expected to change more rapidly at the beginning of the iteration process.

[0501] Alternatively, the scene-generating ML model can be continuously updated (e.g., the ANN weights change from one iteration to the next). This approach is not intended to converge to a single set of parameters. The parameters will continuously change so that in each iteration, the scene-generating ML model creates a useful and challenging scene.

[0502] A pre-selected, relatively small input vector can be used in a scene generation ML model. This input vector can be static (i.e., constant) or randomly varying (within a statistical distribution). For example, if the scene generation ML model is an ANN, the way this input vector is transformed into a scene vector at the ANN output layer through internal ANN operations and hidden layers will change as the ANN weights are updated.

[0503] The architecture (and input vectors, if needed) of the scene generation ML model can be provided by human engineers or determined automatically during the design process.

[0504] Figure 16A The training scene generation module 140a, which can be used in the process of generating decision logic 35, is shown, and Figure 16B A training scene generation module 140b is shown that can be used in the process of generating additional scenes for training performance evaluation logic 27. Scene generation modules 140a and 140b can have the same general construction as described above and can include a repository of exogenous scenes 141a and 141b, an ML model of automatically generated new scenes 142a and 142b, and a repository of scenes 143a and 143b to be revisited (this is optional).

[0505] To allow the training scene generation module 140b to generate scenes outside the specification of the decision logic (as opposed to the scene generation module 140a, which creates scenes within the specification), the following can be used:

[0506] The second set of exogenous scenarios, 141b, can be selected by the engineer to include potential future scenarios beyond the system specifications. Potential future scenarios can be based on possible future changes in the system topology (e.g., installation of new transmission and distribution lines, new transformers, etc.), possible future changes in equipment capabilities (e.g., possible upgrades to existing circuit breakers, protective relays, etc.), or other possible future changes.

[0507] Alternatively or additionally, when training the ML model as a second scene generator 142b, a penalty can be assigned to the output of the second scene generator 142b, which consists of "legitimate" scene vectors. In this way, as the second scene generator 142b becomes better trained during iterations, it will gradually produce scene vectors that correspond only to potential future scenes.

[0508] The second scene generator 142b can be used as Figure 13 The second scene generator 126 in the system.

[0509] Figure 17This is a flowchart of method 150. Method 150 can be used to generate a second training scenario set 132 that exceeds the first training scenario set 131 and can be used to train the performance evaluation logic. Method 150 is executed before the on-site execution of the performance evaluation logic 27. Method 150 can be executed before the on-site execution of the decision logic 35.

[0510] In step 151, a second scene generator 142b can be trained to challenge the decision logic 35. During the training of the second scene generator 142b, the second scene generator 142b is trained to encourage the generation of scenes that are not within the system specifications of the decision logic.

[0511] For illustration, the second scene generator 142b can be an ML model (e.g., an ANN). The following steps can be performed iteratively:

[0512] - A scenario for challenging decision logic 35 is generated by the second scenario generator 142b;

[0513] - The generated scenario is provided to the simulation decision logic 35 to determine the decision and response of the power system or industrial system to the scenario;

[0514] - Consider the decision logic 35 to evaluate the quality of the scenario and whether the scenario is within the system specification to assess the second scenario generator 142b; and

[0515] - Based on the evaluation results, modify the ML model to form the second scene generator 142b.

[0516] The evaluation process can assess the performance of decision logic 35 based on one or more KPIs. The overall evaluation result can be a combination of the performance of decision logic 35 determined based on one or more KPIs and selectively imposed penalties if the scenario is found to be within the system specifications.

[0517] Modifying an ML model can include, but is not limited to, changing the parameters of the activation function.

[0518] Figure 18 This is a schematic diagram of ML model 160, which can be implemented as an ANN and trained as a second scene generator 126, 142b. ML model 160 may include nodes 161 in multiple layers, where activation functions are iteratively modified to improve the performance of ML model 160. In response to input 162, ML model 160 generates output 163. As described above, ML model 160 is trained in such a way that it also generates scenes not within the specification of decision logic 35, such as those used in power systems or industrial systems. The input 162 can be static (i.e., constant) or randomly varying (within a statistical distribution). Output 163 can be the set of all parameter values ​​required to define the running scene.

[0519] Training performance evaluation logic

[0520] Return to Figure 13 and 14 The decision logic generation system 120 includes a first scene generator 121, which competes with another subsystem during an autonomous computer implementation process to generate decision logic 35. This adversarial logic simultaneously improves both the logic for generating decision logic 35 and the first scene generator 121 (e.g., it can act as a creator and discriminator of a GAN). This ensures the autonomous, computer-driven generation of robust and high-performance decision logic 35. The first scene generator 121 itself is trained as part of an autonomous design process.

[0521] The process of generating decision logic 35 converges to the final decision logic 35 of the design process, which is then deployed to controller 31.

[0522] When the design process of decision logic 35 has converged to the final decision logic 35 to be deployed, training of the second scene generator 126 can begin, and thus generate additional training data for performance evaluation logic 27.

[0523] The second scenario generator 126 is trained to iteratively evolve into a system capable of identifying new potential future scenarios that are increasingly challenging for the final decision logic 35. The second scenario generator 126 can be used to generate data that, in an iterative manner, competes with the final decision logic 35 to provide increasingly challenging scenarios outside the current system specifications of the power or industrial system.

[0524] This ultimately leads to a high-performance second scene generator 126, which is able to generate scenes that cause the final decision logic 35 to perform poorly.

[0525] like Figure 18 As shown, the architecture of the challenge scene generator (e.g., the topology of the ANN) and its related parameters (e.g., the weights between ANN nodes) may be continuously updated during the data generation process, i.e., driven by the second scene generator 126 itself.

[0526] The performance of the second scenario generator 126 can be automatically evaluated. A potential future scenario is considered good when the final decision logic 35 performs poorly according to appropriate metrics (multiple appropriate metrics). Furthermore, as mentioned above, the performance of the second scenario generator 126 is evaluated as poor when the output scenario is within the specifications of the decision logic 35, as is the case in a power system or industrial system.

[0527] In this way, the second scene generator 126 gradually learns to identify potential future scenarios that pose challenges.

[0528] Training or learning the second scene generator 126 is an iterative process. In each generation, the ANN weights can be updated. This is done by utilizing the gradient of some output criterion of the ANN relative to the ANN weights. The gradient is used for (stochastic) gradient descent, applied to batches of training data. For example, in a monitored learning setting, the output criterion is the output error, i.e., the difference between the output given by the ANN as output in the simulation and the output that should ideally be output, hereinafter referred to as the "correct ANN output" (e.g., "ground fact"). This "correct ANN output" serves as the monitored learning signal.

[0529] It may be impossible to provide monitored learning signals. In the absence of ANN output errors, methods based on performance metrics to guide ANN training (i.e., updating its weights) can be used. Various parameter search methods can be employed, including trial-and-error search methods such as evolutionary and genetic algorithms, hill climbing, simulated annealing, and other heuristics, and / or model-based search techniques.

[0530] In the case of ANN training, the updated parameters can correspond to the ANN weights.

[0531] The input to the ANN training process is the "learning rate," which corresponds to the impact of the last training sample on the ANN weight updates. The higher the learning rate, the more the gradients of the ANN weights calculated based on the latest (set of) samples are modified.

[0532] The learning rate can be dynamically modified during the training of the second scene generator 126, allowing the ANN to move away from poorly performing solutions more quickly by using a large learning rate, or allowing the ANN to train for more iterations to further improve seemingly well-performing solutions by using a small learning rate.

[0533] Furthermore, by fixing the decision logic 35 during the training of the second scene generator 126, scenes that can challenge the current effective control logic can be effectively identified.

[0534] The evolution of the performance of the active ANN can be automatically monitored, where the performance of the ANN of the second scene generator 126 is the performance of the challenge decision logic 35. It can automatically decide when to select an ANN with a different topology. For example, when the ANN performance no longer improves significantly, or if the active ANN architecture seems unable to reach the desired performance level, the training epochs in which the ANN with the given architecture is updated can be stopped, and different ANN architectures can be autonomously selected (e.g., from a set of predefined candidate ANN architectures).

[0535] At the end of the training cycle, the trained ANN can be stored in a historical database and a new architecture can be selected by using a new set of hyperparameter values.

[0536] The process terminates based on termination criteria (such as convergence to a sufficiently good performance metric, the maximum number of training epochs reached, etc.) and selects the best architecture from the historical database to be used as the second scene generator 126.

[0537] The process of training the second scene generator 126 does not modify the already deployed decision logic 35.

[0538] During the generation of decision logic 35, a first dataset 122, 131 of labeled data is generated. This data contains scenarios 131 where the deployed decision logic 35 performs well. Only cases corresponding to the final decision logic 35 should be used from repository 122.

[0539] The second dataset 127, 132 includes labeled data, where each sample consists of simulated potential future scenarios labeled with the corresponding performance of decision logic 35. Typically, this dataset contains scenarios corresponding to decision logic 35 performing well for it and other scenarios where it performs unacceptably or shockingly poorly.

[0540] These two datasets are merged into one. The merged dataset is used to train the performance evaluation logic 27. The result of this training is an ML model that takes raw or processed system observables as input and provides an estimated controller performance as output. This ML model is called the performance evaluation ML model.

[0541] To train such a performance evaluation ML model, the training dataset needs to be in an appropriate format. For example, if performance is chosen as a binary classification method, then all samples should be labeled with two flags: "1" if the corresponding performance metric is above a certain value (judged as "acceptably good"), and "0" otherwise.

[0542] Of all the attributes defining the scenario vector, only a subset can be used as input attributes for the performance evaluation ML model: those attributes that can be observed by the performance evaluation logic 27 in subsequent field use. These attributes are also referred to herein as “scenario signatures.” Relative to a given decision logic 35, the scenario signature corresponds to the achievable optimal observability of the power grid system. A byproduct of training the performance evaluation logic 27 is that the corresponding performance evaluation ANN (or other performance evaluation ML model) converges to the training, which is a model that extracts the most probable information about the controller performance from the scenario signatures.

[0543] For example, while the most relevant scenario attribute for decision logic evaluation might be the exact control decision of a newly installed power electronic interface device, performance evaluation logic 27 could be trained to evaluate decision logic 35 based solely on locally measured voltage and current, because performance evaluation logic 27 might not have access to the former attribute at runtime. However, these attributes can and should be part of the scenario space searched by the second scenario generator 126 during the data generation phase.

[0544] With labeled data available to train performance evaluation logic 27, monitored machine learning techniques can be used. Similar to what has been explained above, training performance evaluation logic 27 can be an iterative process that includes:

[0545] - Select an ANN architecture for performance evaluation (which may be defined by hyperparameters);

[0546] - Adjust the parameters of the selected performance evaluation ANN architecture to improve the selected performance evaluation ANN for the task of classifying the performance of decision logic;

[0547] - If further improvement cannot be achieved by continuing to train the performance evaluation ANN architecture, store the performance evaluation ANN architecture, the parameters that cause the performance evaluation ANN to have the best performance in the classification decision logic task, and the relevant ratio of correct classification in the history, select another performance evaluation ANN architecture (e.g., by modifying hyperparameters that may affect the number of layers and / or nodes), and repeat the previous steps for the new performance evaluation ANN architecture.

[0548] - If the termination criteria are met, the best-performing performance evaluation ANN is deployed as the performance evaluation logic 27.

[0549] On-site operation of performance evaluation logic 27:

[0550] Once the performance evaluation logic 27 is trained and the required observables (i.e., scene signatures) are defined, the monitoring system 20 can use the performance evaluation logic 27 to analyze the performance of the deployment decision logic 35 based on the scene signatures captured during field operation.

[0551] Output can be generated whenever the performance evaluation logic 27 estimates that the system topology and / or operating conditions have changed, causing the decision logic to fail to meet the expected performance criteria. Output may include alarms, warnings, or other outputs for use via the HMI. Alternatively or additionally, output may automatically trigger control actions, such as activating protection functions, scheduling downtime, triggering personnel deployment, and / or initiating modifications to the decision logic 35.

[0552] Deterioration trend in detection performance

[0553] Monitoring system 20 can be used to detect trends of performance degradation. Monitoring system 20 can operate to provide prognostic information. For illustration, monitoring system 20 can operate to determine and output the predicted time when decision logic 35 needs to be corrected. That is, in addition to operating as a classifier to label currently observed performance, performance evaluation logic 27 can operate to determine trends toward conditions where decision logic 35 is considered to be performing poorly. For example, a "trend" might be an increasing penetration rate of distributed energy through converter interfaces in a region. Various techniques can be used to allow monitoring system 20 to detect such a trend.

[0554] Training performance evaluation logic to determine trends

[0555] When using performance evaluation logic 27, it can be trained to determine trends toward scenarios where decision logic 35 performs poorly. As mentioned above, performance evaluation logic 27 can be generated by training a performance evaluation ML model.

[0556] To allow for trend detection, a second scene generator 126 can be run to create scene batches, each driven by the evolution of a latent variable representing the trend. Each such batch of scenes will be a time series corresponding to the simulated gradual evolution of the trend over time.

[0557] These latent variables can be explicitly defined by human engineers before the autonomous process of generating performance evaluation logic 27 begins. Each trend can be represented by a variable in the scenario vector, which is modified by the second scenario generator 126 to generate new scenarios. The latent variables (multiple latent variables) can be associated with actual variables that define scenarios in a deterministic or stochastic manner. For example, if the latent variable is the total electricity demand of a region, then increasing its value may correspond to a corresponding increase in the demand situation for each consumer.

[0558] By training the performance evaluation ML model, and using batch scenarios driven by the evolution of latent variables during the generation of performance evaluation logic 27, the performance evaluation logic 27 can be trained to implicitly determine the evolution of trends, which may lead to poor future controller performance.

[0559] The performance evaluation logic 27 can generate predictive outputs (e.g., alarms, warnings, or other outputs) that indicate anticipated future conditions in which the decision logic 35 may perform poorly.

[0560] Figure 19 This is a flowchart of method 180. Method 180 can be used before the field run of performance evaluation logic 27 and optionally before the field run of decision logic 35, in order to generate performance evaluation logic 27 so that it can detect trends in decision logic 35 that tend to perform poorly.

[0561] In step 181, scene batches are automatically generated as time series, where the evolution within a batch is driven by the evolution of a variable. This variable can be one of the parameters defining the scene. Not only can one batch be generated, but multiple such batches can be generated.

[0562] In step 182, training data including scene batches is used to train the performance evaluation logic.

[0563] Figure 20 This is a block diagram illustrating the operation of performance evaluation logic 27. Performance evaluation logic 27 receives parameters as input from the scenario defined during the field operation of controller 31.

[0564] The performance evaluation logic 27 operates as a classifier, thereby producing an output 101 indicating the suitability of the decision logic 35 for the current power system topology and operating conditions.

[0565] The performance evaluation logic 27 operates as prediction logic, thereby generating prediction output 102, which indicates that the power system or industrial system is expected to develop toward a state where the decision logic 35 is not performing well.

[0566] Performance evaluation logic 27 may optionally run to output information 103, which indicates the possible root causes of poor performance of decision logic 35, such as references. Figures 24 to 27 As explained.

[0567] Use predictors (multiple predictors)

[0568] To allow for the detection of performance degradation trends, monitoring system 20 can combine performance evaluation logic 27 with at least one predictor, i.e., a model that performs predictions about the evolution of the monitored system. Recurrent neural networks can be utilized to create a data-based predictor. This prediction can be provided as input to performance evaluation logic 27, which then evaluates the performance of the decision logic under the predicted future system conditions. Based on the predictor's forecasts (e.g., future electricity demand or the future penetration of distributed renewable energy), performance evaluation logic 27 is trained to receive and process the predicted quantities as input.

[0569] Figure 21 This is a block diagram of monitoring system 20. Monitoring system 20 includes one or more ICs (multiple ICs) of execution predictor 201 and performance evaluation logic 27. Monitoring system 20 receives scene signatures as input 200.

[0570] At least one of the scenario signatures is fed into predictor 201, which calculates predicted future system parameters (e.g., future electricity demand or future penetration of distributed renewable energy). Performance evaluation logic 27 processes the predicted future system parameters, which are typically combined with other scenario signatures in input 200, to determine whether decision logic 35 has acceptable performance at future points in time (multiple future points in time), with the predictions (multiple predictions) calculated by predictor 201 relating to this acceptable performance.

[0571] Figure 22 A monitoring system is shown, which includes one or more ICs that execute several predictors 201-209 and several instances of performance evaluation logic 27a-n.

[0572] Different predictors 201-209 can be used to predict system parameters at various future points in time (e.g., future electricity demand or the future penetration of distributed renewable energy) (e.g., a first predictor 201 can provide a prediction for a first future point in time, and a second predictor 202 can provide a prediction for a second future point in time different from the first future point in time). Alternatively or additionally, the same predictor can be used repeatedly until it reaches a specific future point in time, such as... Figure 21 As shown.

[0573] For any of these predictions, the predictor (multiple predictors) will generate a prediction scenario signature and analyze the predictive performance of the decision logic given the current trend (multiple given current trends).

[0574] Regardless of the technique used to predict and identify potential future system conditions that could lead to poor performance of decision logic 35, the resulting estimates can prompt human experts to issue early warnings about the potential future performance of decision logic 35 (e.g., given current trends, the performance of the control logic will become concerning in two years). Such early warnings can be updated periodically as new predictions are made.

[0575] Figure 23 This is a flowchart of method 210, which can be automatically executed by monitoring system 20 during field operation.

[0576] In step 211, predicted changes in the power system or industrial system can be identified, which may cause decision logic 35 to malfunction. This can be accomplished using any of the techniques described above, for example, by using the time-series evolution of the scenario and / or by training performance evaluation logic 27 using one or more predictors that provide inputs (multiple inputs) to performance evaluation logic 27.

[0577] In step 212, a warning, alarm, or other output may be generated. The warning, alarm, or other output may be output via an HMI. Alternatively or additionally, the monitoring system 20 may automatically trigger the decision logic generator 120 to generate modified decision logic 35.

[0578] Provide insights into the root causes.

[0579] The monitoring system 20 can operate to provide insights into any observed deterioration that may be the root cause (or reason) of the applicability of the monitored controller 31.

[0580] Determining the underlying cause of the observed performance degradation corresponds to identifying the scenario signature input (i.e., one of the system observables received as input by the monitoring system 20) that may be responsible for the system condition that causes decision logic 35 to perform poorly. This may be a single scenario signature or a combination of multiple scenario signatures.

[0581] Recall that the term "scenario signature" refers to parameters of a power system or industrial system that are received and processed by the monitoring system 20 during field operation.

[0582] Monitoring system 20 may operate to determine one of its inputs that needs to be modified so that decision logic 35 meets a performance metric-based standard (e.g., performance determined based on a performance metric exceeding a threshold). Monitoring system 20 may also operate to determine one of its inputs that requires minimal modification to make decision logic 35 meet a performance metric-based standard. Monitoring system 20 may operate to output information about determined system parameters (multiple system parameters), which leads to poor performance of decision logic 35.

[0583] Figure 24 This is a flowchart of method 220. Method 220 can be automatically executed by monitoring system 20 during field operation.

[0584] In step 221, the system state of the power system or industrial system is detected, which causes decision logic 35 to not meet the criteria based on performance metrics (e.g., having performance determined according to a performance metric that exceeds a threshold).

[0585] In step 222, one or more system parameters that must be changed to make decision logic 35 meet performance metric-based criteria (e.g., performance determined according to a performance metric exceeding a threshold) are automatically determined by the monitoring system. Information about which system parameters (multiple system parameters) must be changed to ensure compliance with the performance metric-based criteria is output.

[0586] Figure 25 This schematically illustrates the role of multiple system parameters sp1, ..., sp n ...spM The state space traversed. Parameters sp1, ..., sp n ...sp M It is an observable that can be received and processed by the performance evaluation logic 27 or more generally by the monitoring system 20.

[0587] The parameter space is subdivided into an acceptable scenario space 231 and an unacceptable scenario space 232. All combinations of scenario signatures 233 that lead to well-functioning decision logic 35 are arranged in the acceptable scenario space 231. All combinations of scenario signatures 234 that lead to poorly-functioning decision logic 35 are arranged in the unacceptable scenario space 232.

[0588] These spaces 231, 232 depend on the performance metric and threshold used, and the computational performance of decision logic 35 is compared with that threshold.

[0589] When an unacceptable scene signature 235 is identified in scene space 232, a procedure for determining the potential root cause of poor performance of decision logic 35 can be automatically triggered. Alternatively, the procedure can be selectively triggered only if the calculated performance is below a first threshold but above a second threshold.

[0590] The monitoring system 20 can operate to determine the minimum modification 236 to the current scene signature 235, which would result in a modified scene signature located within the acceptable scene space 231. That is, the monitoring system 20 can operate to determine the minimum modification to the scene signature that would qualify such a scene signature to be classified within the acceptable scene space.

[0591] This can be accomplished in various ways. For illustration, monitoring system 20 can run to solve a constrained optimization problem to determine the minimum modification 236 to the current scenario signature 235 that results in acceptable performance for the decision-making logic. When performance evaluation logic 27 is deployed, it can be used to evaluate the impact of offsets (multiple offsets) 236 in the scenario signature explored within the constrained optimization routine.

[0592] The monitoring system 20 can use the current scene signature as the objective function to be minimized. And the hypothetical modified scenario signature The standard distance between them

[0593] (3)

[0594] The monitoring system 20 can minimize the objective function (3) under constraints.

[0595] (4)

[0596] Enables signature in modified scenarios The determined performance value satisfies the standard based on performance metrics, i.e., the modified scene signature. Located within the acceptable scene space 231.

[0597] Figure 26 This is a schematic block diagram of the monitoring system 20, which combines performance evaluation logic with a constrained optimization engine 240. The constrained optimization engine 240 searches the parameter space for modification points that are as close as possible to the actual system state currently observed during field operation of the controller 31, while causing the performance of the decision logic 35 determined by the performance evaluation logic 27 to be considered acceptable.

[0598] The monitoring system 20 generates an output 241 that indicates which system parameters(s) must be changed to ensure that the decision logic 35 has the performance deemed acceptable by the performance evaluation logic 27. Output 241 may be provided to a human expert via an HMI. Alternatively or additionally, control actions corresponding to deviations in the system parameters determined by the monitoring system 20 may be automatically taken or suggested.

[0599] Some differences between the quantities in the scene signatures will be zero, meaning no impact; some differences will be small, meaning minimal impact; and some differences will be large, meaning significant impact. This can provide insights into which changes in the environment are affecting the suitability / performance of the controller being evaluated by the monitoring system.

[0600] The constrained optimization engine 240 can utilize various techniques. Essentially, solving constrained optimization problems is a parameter search problem. The constrained optimization engine 240 can use trial-and-error search, which is available without requiring any analytical information (such as gradients) to guide the search of the parameter space. Established trial-and-error search methods include evolutionary and genetic algorithms, hill climbing, simulated annealing, and other heuristic algorithms.

[0601] The constrained optimization engine 240 can perform a model-based search that estimates the expected performance of parameter selection, thereby bypassing the need to simulate (batch) scenarios for computing performance metrics. In this case, the model can be determined from simulations of multiple scenario batches used when generating the performance evaluation logic 27.

[0602] Another approach to determining the root cause of why the current system state of a power system or a problem in an industrial system leads to the poor performance of decision logic 35 can be achieved using sensitivity analysis. For illustration, monitoring system 20 can utilize the sensitivity of the output of performance evaluation logic 27 to its inputs. This is a conditional sensitivity, i.e., given an observation (indicating that the controller's representation is unsuitable). In this way, monitoring system 20 can report to human experts the actual determined output of performance evaluation logic 27 to the scenario signature to which it is most sensitive.

[0603] Figure 27 This is a schematic block diagram of monitoring system 20, which combines performance evaluation logic 27 with sensitivity analysis engine 250. Sensitivity analysis engine 250, in response to modifications to various parameters included in the input, can determine the sensitivity of the output of performance evaluation logic 27 to its input. The output 251 of monitoring system 20 can indicate the signature of the scenario to which the actual determined output of performance evaluation logic 27 is most sensitive.

[0604] The methods, apparatus, and systems according to the invention allow for the automatic monitoring of decision logic during field operation. The disclosed techniques remain effective even when the decision logic is a trained ML model, which appears as a black box to technical experts.

[0605] This method, device, and system can be used to simultaneously monitor individual decision logic deployed in industrial systems, such as power generation, transmission, or distribution systems. In this context, dedicated performance evaluation logic can be generated and deployed for any decision logic in the field.

[0606] The methods, apparatus, and systems according to the present invention can identify scenarios that would cause decision logic to fail to perform in accordance with quality standards based on performance metrics.

[0607] As used in this article, a “scenario” typically includes a set of operating conditions and events that the decision-making logic faces, such as power generation, demand and weather patterns, faults (i.e., short circuits), planned power outages, topology changes, the addition of new components, demand evolution, storage, and renewable energy generation.

[0608] Decision logic can be control logic (e.g., protection logic, such as distance protection logic), but is not limited to this.

[0609] The methods, apparatus, and systems according to the present invention can be used to evaluate whether any of the following logics can operate according to desired performance standards, but are not limited thereto:

[0610] - Protection system, and SIPS (System Integrity Protection Solution).

[0611] - Local FACTS / HVDC controller,

[0612] - Coordinate and control many devices across the power grid (e.g., FACTS, HVDC, generator active demand, storage).

[0613] - Control DER (directly or through aggregators or distribution system operators).

[0614] - Distributed control in power grids

[0615] - Control logic that allows for relaxation of grid operation restrictions within various lead time ranges (the control logic may include control actions before and after disturbances).

[0616] - This is used for the logic of arranging asset maintenance / replacement.

[0617] - Power system recovery logic,

[0618] - Propose the logic for upgrading and expanding the power system.

[0619] Although embodiments have been described in the context of power systems, the methods and computer systems are not limited to generating and / or using the decision logic of protective relays in power generation, distribution, or transmission systems. Rather, the disclosed methods and computer systems can be used to generate the decision logic of one or more controllers of an IACS.

[0620] Although the invention has been described in detail in the accompanying drawings and the foregoing description, such description is to be considered illustrative or exemplary rather than restrictive. Variations on the disclosed embodiments can be understood and implemented by those skilled in the art and by practice through study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plural. The fact that certain elements or steps are stated in different claims does not mean that combinations of such elements or steps cannot be used in advantageous, specific claims other than the actual dependent claims, but rather any other meaningful combination of claims should be considered disclosed.

Claims

1. A method for monitoring the performance of a decision logic (35) of a controller (31) during field operation of the controller (31) of a power system, the decision logic (35) generating control commands for protection functions in the power system, the method being executed by a monitoring system (20) and comprising: Receive operational data collected during field operation of the controller (31); The operational data is analyzed to evaluate the performance of the decision logic (35). The analysis is performed using pre-run data generated prior to the field operation of the controller (31), wherein the pre-run data includes at least one of the following: - The topology model or load flow model used to perform the simulation when generating the decision logic (35), - The scenario simulated during the generation of the decision logic (35) and represented as the input vector of the decision logic (35). - The performance of the decision logic (35) in the simulation performed when generating the decision logic (35), and / or The analysis is performed using performance evaluation logic (27), which is a machine learning model trained using data (122, 127, 128; 131, 132) generated during the generation of the decision logic (35) before the field operation of the controller (31) and after the generation of the decision logic (35). The performance evaluation logic is configured to perform the analysis without accessing a database containing information about the generation of the decision logic during the field operation of the controller. and Analysis output is generated based on the results of the analysis. Performing the analysis includes calculating one or more key performance indicators (KPIs) based on at least one metric.

2. The method according to claim 1, wherein the analysis is performed by comparing the control actions taken by the controller (31) during field operation with the control actions simulated when generating the decision logic (35), and / or comparing the operating points observed by the controller (31) during field operation with the operating points generated by the simulated scenarios, test cases and / or events when generating the decision logic (35).

3. The method according to claim 1, wherein the analysis uses performance evaluation logic (27), the performance evaluation logic (27) receiving the running data as input and / or receiving predictions of future running points from a predictor that receives the running data as input, the performance evaluation logic (27) being generated using data generated during the computer implementation of creating the decision logic (35), wherein the performance evaluation logic (27) is a machine learning model trained with a second set of scenarios (132), the second set of scenarios being different from the first set of scenarios (131) used to train the decision logic (35).

4. The method according to any one of claims 1 to 3, wherein the method further comprises controlling a human-machine interface (HMI) to output the analysis output and / or automatically performing control actions based on the analysis output, the control actions including feeding the analysis output back to a generation system for forming the decision logic.

5. The method according to any one of claims 1 to 3, wherein the at least one metric depends on user input.

6. The method according to any one of claims 1 to 3, wherein, The operating data also includes power system data that affects the operation of the controller (31).

7. The method according to any one of claims 1 to 3, wherein the analysis is performed using the pre-run data, and the method further comprises retrieving the pre-run data from the database (40) during field operation of the controller (31).

8. The method of claim 1, wherein performing the analysis comprises at least one of the following: - Evaluate the accuracy of the model used to perform the simulation when generating the decision logic (35); - Evaluate the scenarios, test cases, and / or events simulated when generating the decision logic (35); - Compare the control actions taken by the controller (31) during field operation with the control actions simulated when generating the decision logic (35); - The value of at least one key performance indicator (KPI) output for a decision made by the controller (31) during field operation is compared with the value of the at least one key performance indicator (KPI) stored in the pre-operation data.

9. The method according to any one of claims 1 to 3 and 8, wherein the performance evaluation logic (27) is an artificial neural network.

10. The method of claim 1, further comprising training the machine learning model with a second set of scenes (132), wherein The second scenario set (132) includes scenarios outside the operating specifications of the controller (31), and / or The second scenario set (132) is different from the first scenario set (131) used to train the decision logic (35).

11. The method according to any one of claims 1 to 3, 8 and 10, further comprising: Monitor or predict the time-related evolution of the results of the analysis and generate the analysis output based on the time-related evolution.

12. The method according to any one of claims 1 to 3, 8 and 10, further comprising: Identify the root cause of poor performance of the decision logic (35) during field operation and / or implement root cause solutions to improve the decision logic (35) during field operation.

13. The method according to any one of claims 1 to 3, 8 and 10, further comprising: The human-machine interface (HMI) is controlled to output the analysis output, wherein the analysis output includes information about the past, present and / or predicted future performance of the controller (31) during field operation.

14. A method for operating power system assets of a power system, the method comprising: At least one controller (31) executes decision logic (35) during the field operation of the controller (31), the decision logic (35) including generating and outputting decision outputs to control the power system assets thereby performing protection functions for the power system; The method according to any one of the preceding claims is executed by at least one integrated circuit of the controller (31) or by at least one other integrated circuit for monitoring the decision logic (35) of the controller (31) during field operation of the controller (31); and The analysis output is output through the human-machine interface (HMI), and / or control actions are automatically executed based on the analysis output, wherein the control actions trigger the regeneration or update of the decision logic (35).

15. The method of claim 14, wherein the method comprises: The analysis output is output via a human-machine interface (HMI) and / or control actions are automatically executed based on the analysis output.

16. A monitoring system (20) for the decision logic (35) of a controller (31) for a power system during field operation of the controller (31), the monitoring system (20) comprising: At least one integrated circuit, said at least one integrated circuit being operated to perform the method of any one of claims 1 to 15.

17. An electric power system, the electric power system comprising: A controller (31) operates to execute decision logic (35) to determine which control actions must be taken; and The monitoring system (20) according to claim 16 is used to monitor the decision logic (35) during its operation.

Images