Real-time optimization method and system for power supply quality of micro-grid information physical system

Abstract

The invention discloses a real-time optimization method and system for power supply quality of a micro-grid information physical system. The micro-grid information physical system power supply quality real-time optimization method comprises the following steps: acquiring electric quantity data and equipment state data at the current moment; obtaining a power supply quality index at the current moment according to the electrical quantity data at the current moment; constructing a state vector at the current moment based on the power supply quality index at the current moment and the equipment state data at the current moment; inputting the state vector at the current moment into a UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains a Q value function and an action access counting table; for each selectable action under the state vector at the current moment, the UCB-H reinforcement learning model calculates the confidence interval upper bound score of each selectable action according to the Q value of each selectable action and the action access count, and selects the selectable action with the highest confidence interval upper bound score as the optimal action at the current moment; and performing power supply quality real-time optimization based on the optimal action at the current moment.

Description

Technical Field

[0001] This invention belongs to the field of power control technology, and in particular relates to a method and system for real-time optimization of power supply quality in a microgrid cyber-physical system. Background Technology

[0002] Microgrids, as autonomous systems integrating distributed renewable energy sources, energy storage, and loads, are a key component in building new power systems. One of their core tasks is to ensure power quality, namely, maintaining voltage and frequency stability and suppressing harmonic pollution. However, power fluctuations from intermittent power sources such as photovoltaics and wind turbines, as well as the switching of nonlinear loads such as frequency converters, can cause rapid voltage changes and waveform distortions, seriously threatening the safety of sensitive equipment and the stable operation of the power grid.

[0003] To address these issues, current technologies primarily rely on power electronic devices such as energy storage converters and active power filters for rapid compensation. Traditional control methods, such as PID control or model predictive control, heavily depend on the precise mathematical model of the microgrid for their effectiveness. However, in actual operation, changes in system parameters, network topology adjustments, and strong random disturbances make it extremely difficult to establish an accurate and universal model. This results in slow response and weak adaptive capabilities of traditional methods, making it difficult to meet the requirements of millisecond-level real-time control.

[0004] To overcome model dependence, data-driven reinforcement learning technology has been introduced into the field of microgrid optimization. Its basic idea is to allow the agent to learn optimization strategies autonomously through trial and error interaction with the environment. However, when classic algorithms such as Q-Learning are directly applied to highly dynamic power quality control, obvious defects are exposed. The commonly used ε-greedy action selection strategy is inefficient in exploring high-dimensional complex state spaces, involves a large amount of blind trial and error, has slow convergence, and may produce unsafe actions. At the same time, existing methods lack an effective evaluation mechanism for the learning process and cannot intelligently determine when the strategy converges and when it should switch between "exploration" and "stable execution" modes. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for real-time optimization of power supply quality in a microgrid cyber-physical system, in order to solve the problems mentioned in the background art, such as the lack of real-time performance and adaptability of traditional model-dependent methods, as well as the low efficiency and slow convergence of the original exploration mechanism of existing reinforcement learning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for real-time optimization of power supply quality in a microgrid cyber-physical system, comprising: Collect current electrical quantity data and equipment status data; Based on the electrical quantity data at the current moment, the power supply quality index at the current moment is calculated; based on the power supply quality index at the current moment and the equipment status data at the current moment, the state vector at the current moment is constructed. The current state vector is input into the UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains a Q-value function and an action access count table; for each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval of each selectable action under the current state vector based on the Q-value and action access count of each selectable action under the current state vector, and selects the selectable action with the highest upper bound score of the confidence interval as the optimal action at the current time; Power quality is optimized in real time based on the optimal action at the current moment.

[0007] Optionally, the power supply quality indicators include voltage deviation rate, frequency deviation value, total harmonic distortion rate and interharmonic components, and the equipment status data includes distributed power output fluctuation, energy storage SOC status and critical load sensitivity level.

[0008] Optionally, for each selectable action under the state vector at the current time, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval for each selectable action under the state vector at the current time based on the Q-value and action access count of each selectable action under the state vector at the current time, specifically including the following steps: Based on the action access count table, determine the total number of times the state corresponding to the state vector at the current moment is accessed, and the historical number of times each optional action under the state vector at the current moment is selected; For each selectable action under the state vector at the current time, the upper bound score of the confidence interval for each selectable action under the state vector at the current time is calculated using a preset upper bound formula based on the Q value of each selectable action under the state vector at the current time, the total number of times the state has been accessed, and the historical number of times each selectable action under the state vector at the current time.

[0009] Optionally, the formula for the preset upper bound of the confidence interval is: ; Where s is the state vector at the current moment, a is the available action under the state vector at the current moment, UCB(s,a) is the upper bound score of the confidence interval of the available action under the state vector at the current moment, Q(s,a) is the Q value of the available action under the state vector at the current moment, c is the exploration coefficient, N(s,a) is the number of times the available action under the state vector at the current moment has been selected in history, and N(s) is the total number of times the state has been visited.

[0010] Optionally, after the step of performing real-time power quality optimization based on the optimal action at the current moment, the method further includes: Collect electrical quantity data and equipment status data for the next moment; Based on the electrical quantity data at the next moment, the power supply quality index for the next moment is calculated; based on the power supply quality index and the equipment status data at the next moment, the state vector for the next moment is constructed. Based on the state vector at the next time step, the Q-value function of the UCB-H reinforcement learning model is queried to obtain the Q-value of each selectable action under the state vector at the next time step; based on the state vector at the next time step and the Q-value of each selectable action under the state vector at the next time step, the policy evaluation loss value is calculated; the changing trend of the policy evaluation loss value in multiple consecutive control cycles is monitored to determine whether the policy of the UCB-H reinforcement learning model has converged. If the policy of the UCB-H reinforcement learning model fails to converge, the Q-value function and action access count table of the UCB-H reinforcement learning model are updated based on the immediate reward obtained after executing the optimal action at the current moment, the state vector at the current moment, the optimal action at the current moment, and the state vector at the next moment. The process of inputting the state vector at the current moment into the UCB-H reinforcement learning model is then returned to make a decision for the next control cycle. If the determination result is that the policy has converged, the Q-value function of the UCB-H reinforcement learning model is locked, and the optimal action is output using the Q-value function of the UCB-H reinforcement learning model in subsequent control cycles.

[0011] Optionally, the step of calculating the policy evaluation loss value based on the state vector at the next time step and the Q-values ​​of each selectable action under the state vector at the next time step specifically includes the following steps: Based on the state vector at the next moment and the Q-values ​​of each selectable action under the state vector at the next moment, the policy probability distribution of the UCB-H reinforcement learning model at the current moment is generated; Based on the probability distribution of the policy, calculate the cross-entropy loss value, and use the cross-entropy loss value as the policy evaluation loss value.

[0012] Optionally, the monitoring strategy evaluates the trend of the loss value over multiple consecutive control periods to determine whether the strategy of the UCB-H reinforcement learning model has converged, specifically including the following steps: Monitor the numerical change of the cross-entropy loss value over multiple consecutive control cycles; If the cross-entropy loss value shows a monotonically decreasing trend within a consecutive first preset number of control cycles, and the fluctuation range of the cross-entropy loss value within a consecutive first preset number of control cycles is greater than a first preset threshold, then it is determined that the strategy has not converged. If the fluctuation range of the cross-entropy loss value within a second preset number of consecutive control cycles is less than a second preset threshold, then the strategy is determined to have converged.

[0013] In a second aspect, the present invention provides a device for real-time optimization of power quality in a microgrid cyber-physical system, comprising: The quality monitoring module is used to collect electrical quantity data and equipment status data at the current moment; The quality monitoring module is also used to calculate the power supply quality index at the current moment based on the electrical quantity data at the current moment; and to construct the state vector at the current moment based on the power supply quality index at the current moment and the equipment status data at the current moment. The decision generation module is used to input the current state vector into the UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains a Q-value function and an action access count table; for each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval of each selectable action under the current state vector based on the Q-value and action access count of each selectable action under the current state vector, and selects the selectable action with the highest upper bound score of the confidence interval as the optimal action at the current time; The execution control module is used to optimize the power supply quality in real time based on the optimal action at the current moment.

[0014] In a third aspect, the present invention provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement a real-time power quality optimization method for a microgrid cyber-physical system as provided in any embodiment.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the real-time power quality optimization method for a microgrid cyber-physical system as provided in any of the preceding embodiments.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a method and system for real-time optimization of power supply quality in microgrid cyber-physical systems. This method addresses the shortcomings of traditional model-dependent methods in terms of real-time performance and adaptability, as well as the low efficiency and slow convergence of existing reinforcement learning's primitive exploration mechanisms. The invention achieves the following beneficial effects: by fusing power supply quality indicators with equipment status data to construct a state vector, it provides comprehensive and real-time condition awareness for intelligent decision-making. Furthermore, by introducing a UCB-H reinforcement learning model, it avoids blind random exploration or reliance on fixed rules during each decision. Instead, it intelligently calculates the upper bound score of the confidence interval based on the historical Q-value and access count of each action. This achieves a dynamic balance between "utilizing known optimality" and "exploring potential better," enabling the system to quickly and accurately identify the most effective control action under complex and changing conditions. This eliminates the reliance on precise offline mathematical models, enhances the system's adaptability to new energy fluctuations and random disturbances, and improves the exploration efficiency and convergence speed of optimization strategies. Ultimately, it achieves synchronous, rapid, and online optimization of multi-dimensional power supply quality indicators.

[0017] Furthermore, by combining core power quality parameters such as voltage, frequency, harmonics, and interharmonics with key operating states such as power output fluctuations, energy storage status, and load sensitivity, the constructed state vector provides the UCB-H model with a high-fidelity, multi-dimensional real-time "portrait" of the system. This enables the model to not only perceive actual deviations in power quality but also simultaneously grasp the sources of these deviations and the system's regulatory capacity boundaries. Based on this comprehensive information, the UCB-H's confidence interval upper bound exploration mechanism operates within a precise decision space, avoiding blind attempts at irrelevant or inefficient actions. This improves the targeting and convergence speed of finding superior control strategies under complex operating conditions, achieving deep collaborative optimization from perception to decision-making.

[0018] Furthermore, by quantifying the historical exploration experience of each control action to guide decision-making, optimization efficiency is directly improved. By statistically analyzing the total number of occurrences of each operating state and the previous number of times each adjustment action has been executed in that state, the uncertainty of actions with insufficient attempts is quantified and transformed into higher exploration value. For actions that have been fully validated, the system relies more on their stable Q value. This allows the UCB-H model to proactively and purposefully try control strategies with high potential value but limited application, while consolidating mature strategies. This fundamentally avoids the blindness of traditional random exploration, enabling the system to quickly lock in the most effective optimization action with fewer and safer interactions in the rapidly changing operating conditions of the microgrid.

[0019] Furthermore, by calculating the loss value of the strategy evaluation, an objective and quantitative basis is provided for judging whether the current control strategy has become stable. When the strategy has not converged, the system can use the immediate reward after the action is executed and the new system state to update the model in real time, so that it can continuously learn and improve from actual interactions. Once the strategy converges, the system locks the current efficient Q-value function and directly outputs the optimal action, which greatly improves the response efficiency and operational stability. This allows the system to not only quickly find the best in the early stage, but also to have the intelligence to learn and use immediately and learn again when changes occur. It realizes a smooth and adaptive switch from continuous exploration to reliable operation, which enhances the applicability and reliability of long-term operation.

[0020] Furthermore, by introducing cross-entropy loss as the core indicator for policy evaluation, a precise "measuring stick" is provided for the optimization process. The model's preference for each action in the next moment is transformed into a calculable scalar loss. The smaller the loss value, the clearer and more confident the model's judgment on the optimal action in the current state. At the same time, by continuously monitoring the changes in this loss value, the system can clearly and quantitatively perceive the dynamic improvement of the policy, thereby transforming the convergence judgment from subjective experience to objective data-driven judgment. This enables the system to keenly identify learning plateaus, avoid ineffective continuous exploration, or promptly capture policy degradation, providing a reliable and direct decision-making basis for the intelligent switching between learning and locking modes, and ensuring the efficiency and stability of the entire optimization process.

[0021] Furthermore, by setting clear quantitative criteria, the judgment of policy convergence becomes objective, automatic, and reliable. When the cross-entropy loss value continues to decrease significantly, it indicates that the model is in an active optimization learning phase, and the system will continue to explore and update. When the loss value enters and remains within a very small fluctuation range, it signifies that the policy has stabilized, and the system can switch to an efficient locked execution mode. This transforms policy management from relying on subjective experience to making precise decisions based on data, effectively avoiding ineffective exploration or premature convergence caused by the inability to accurately judge the learning state in traditional methods. It not only ensures the sufficiency of the learning process but also ensures that the system can run stably and reliably after the policy matures, thus achieving a balance between optimization efficiency and operational robustness in a dynamic environment.

[0022] The present invention provides a real-time power quality optimization device, electronic device and computer-readable storage medium for a microgrid cyber-physical system, which also solves the problems of insufficient real-time performance and adaptability of traditional model-dependent methods proposed in the background section, as well as the low efficiency and slow convergence of the original exploration mechanism of existing reinforcement learning. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 A flowchart of a method for real-time optimization of power supply quality in a microgrid cyber-physical system provided in an embodiment of the present invention; Figure 2 A structural block diagram of a real-time power quality optimization device for a microgrid cyber-physical system provided in an embodiment of the present invention; Figure 3 A structural block diagram of an electronic device provided in an embodiment of the present invention; Among them, 100 is electronic equipment; 101 is memory; 102 is processor; 103 is computer program; and 104 is communication bus. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0025] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0026] Example 1 like Figure 1 As shown, in a first aspect, the present invention provides a method for real-time optimization of power supply quality in a microgrid cyber-physical system, comprising: S1: Collect electrical quantity data and equipment status data at the current moment; S2: Calculate the power supply quality index at the current moment based on the electrical quantity data at the current moment; construct the state vector at the current moment based on the power supply quality index at the current moment and the equipment status data at the current moment; S3: Input the current state vector into the UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains the Q-value function and the action access count table; for each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval of each selectable action under the current state vector based on the Q-value and action access count of each selectable action under the current state vector, and selects the selectable action with the highest upper bound score of the confidence interval as the optimal action at the current time; S4: Optimize power supply quality in real time based on the optimal action at the current moment.

[0027] It should be noted that the real-time power quality optimization method of the microgrid cyber-physical system is implemented through a closed-loop cyber-physical system of "perception-decision-execution-evaluation". Its core lies in an innovative reinforcement learning agent - the UCB-H model, which constitutes a continuously running online optimization loop in terms of process.

[0028] Therefore, the real-time power quality optimization method and system for microgrid cyber-physical systems provided by this invention solves the problems of insufficient real-time performance and adaptability of traditional model-dependent methods in the prior art, as well as the low efficiency and slow convergence of the original exploration mechanism of existing reinforcement learning. It achieves the following beneficial effects: by fusing power quality indicators with equipment status data to construct a state vector, it provides comprehensive and real-time operating condition perception for intelligent decision-making. It also introduces the UCB-H reinforcement learning model, which, in each decision, does not blindly explore randomly or rely on fixed rules, but intelligently calculates the upper bound score of its confidence interval based on the historical Q value and access count of each action. This achieves a dynamic balance between "utilizing known optimality" and "exploring potential better," enabling the system to quickly and accurately identify the most effective control action under the current complex and changing state. It gets rid of the dependence on precise offline mathematical models, enhances the system's adaptive ability to cope with new energy fluctuations and random disturbances, and improves the exploration efficiency and convergence speed of optimization strategies. Finally, it realizes synchronous, rapid, and online optimization of multi-dimensional power quality indicators.

[0029] In step S1: collect electrical quantity data and equipment status data at the current moment.

[0030] It should be noted that this step is the starting point of each optimization cycle, and aims to transform continuous signals from the physical world into state information that the reinforcement learning agent can understand. The electrical quantity data is collected in real time through synchronous phasor measurement units (PMUs), power quality analyzers, and high-frequency sensor networks deployed at the microgrid's point of common connection (PCC), key bus, and important distributed power access points. The electrical quantity data includes synchronous phasors of three-phase voltage and current, real-time transient values ​​and rates of change of voltage and frequency, and harmonic spectra of current and voltage.

[0031] Therefore, by collecting real-time data from the system's operation, the reliance on fixed models is eliminated, and the dynamic changes of the microgrid are directly captured. This provides accurate and timely on-site data for subsequent intelligent decision-making, enabling the optimization process to start from actual operating conditions and quickly respond to fluctuations in new energy sources and load changes, thus laying a reliable data foundation for building an adaptive optimization system.

[0032] In step S2: the power supply quality index at the current moment is calculated based on the electrical quantity data at the current moment; the state vector at the current moment is constructed based on the power supply quality index at the current moment and the equipment status data at the current moment.

[0033] Therefore, the raw data is transformed into a structured state vector that represents the system's operational status. This not only reflects the instantaneous status of core quality indicators such as voltage and frequency in real time, but also integrates key equipment information such as power output and energy storage level. This provides a comprehensive and accurate situation profile for subsequent intelligent decision-making, enabling the model to make optimization decisions based on an accurate assessment of the system's current actual capabilities.

[0034] In one embodiment, the step of constructing the state vector at the current moment based on the power quality index at the current moment and the device status data at the current moment specifically includes: The power quality index and the device status data at the current moment are normalized respectively to obtain the normalized power quality index and device status data at the current moment. The normalized power quality index and the device status data at the current moment are combined in a predetermined order to construct the state vector at the current moment.

[0035] In one embodiment, the power quality indicators include voltage deviation rate, frequency deviation value, total harmonic distortion rate and interharmonic components, and the equipment status data includes distributed power output fluctuation, energy storage SOC status and critical load sensitivity level.

[0036] It should be noted that the voltage deviation rate reflects the degree to which the voltage amplitude deviates from the rated value, the frequency deviation value reflects the stability of the system frequency, and the total harmonic distortion rate (THD) is the ratio of the effective value of all harmonic components in the voltage or current to the effective value of the fundamental component, reflecting the overall level of waveform distortion.

[0037] Therefore, by combining core power quality parameters such as voltage, frequency, harmonics, and interharmonics with key operating states such as power output fluctuations, energy storage status, and load sensitivity, the constructed state vector provides the UCB-H model with a high-fidelity, multi-dimensional real-time "portrait" of the system. This enables the model to not only perceive actual deviations in power quality but also simultaneously grasp the sources of these deviations and the system's regulatory capacity boundaries. Based on this comprehensive information, the UCB-H's confidence interval upper bound exploration mechanism operates within a precise decision space, avoiding blind attempts at irrelevant or inefficient actions. This improves the targeting and convergence speed of finding superior control strategies under complex operating conditions, achieving deep collaborative optimization from perception to decision-making.

[0038] In step S3: the current state vector is input into the UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains a Q-value function and an action access count table; for each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval of each selectable action under the current state vector based on the Q-value and action access count of each selectable action under the current state vector, and selects the selectable action with the highest upper bound score of the confidence interval as the optimal action at the current time.

[0039] It should be noted that this step is the core of this method, executed by the UCB-H reinforcement learning agent, whose internal mechanism replaces the complex solution process of traditional optimization. The UCB-H reinforcement learning model is obtained by replacing the greedy policy in the Q-Learning algorithm with the confidence upper bound policy UCB. At each decision time, when the UCB-H reinforcement learning model is input with the current state vector, it does not adopt the ε-greedy policy of random exploration in the traditional Q-Learning algorithm, but uses UCB to intelligently select the optimal action at the current time. This ensures that the system can actively explore control strategies with high potential value but insufficient attempts, while making full use of proven effective strategies, thereby finding the global optimum more efficiently in an uncertain environment. The optional actions include the reactive power compensation of the energy storage converter, the harmonic injection command of the active filter, and the dynamic compensation coefficient of the static reactive power generator.

[0040] Therefore, by quantifying the historical performance and number of explorations of each action, the system intelligently balances "utilizing mature strategies" and "exploring new solutions," enabling it to quickly identify the most likely effective control action when facing complex situations. This avoids the blindness of traditional random exploration and significantly improves the accuracy of decision-making and learning efficiency.

[0041] In one embodiment, the definition of the UCB-H model includes defining the action space, state space, and reward function.

[0042] It should be noted that the action space is defined as a series of discrete or continuous power quality adjustment commands. In this embodiment, preferably, the action space includes a reactive power compensation setting value issued to the energy storage converter PCS, a specific harmonic current injection command issued to the active power filter APF, and a dynamic reactive power compensation coefficient adjustment command issued to the static var generator SVG. The state space is a set of all possible values ​​of the state vector. The reward function is used to guide the agent to optimize power supply quality. The immediate reward is obtained after the action is performed in the state. The value of the immediate reward is calculated by the reward function and is usually a combination of negative and positive rewards.

[0043] In one embodiment, for each selectable action in the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval for each selectable action in the current state vector based on the Q-value and action access count of each selectable action, specifically including the following steps: Based on the action access count table, determine the total number of times the state corresponding to the state vector at the current moment is accessed, and the historical number of times each optional action under the state vector at the current moment is selected; For each selectable action under the state vector at the current time, the upper bound score of the confidence interval for each selectable action under the state vector at the current time is calculated using a preset upper bound formula based on the Q value of each selectable action under the state vector at the current time, the total number of times the state has been accessed, and the historical number of times each selectable action under the state vector at the current time.

[0044] Therefore, by quantifying the historical exploration experience of each control action to guide decision-making, optimization efficiency is directly improved. By statistically analyzing the total number of occurrences of each operating state and the previous number of times each adjustment action has been executed in that state, the uncertainty of actions with insufficient attempts is quantified and transformed into higher exploration value. For actions that have been fully verified, the system relies more on their stable Q value. This allows the UCB-H model to actively and purposefully try control strategies with high potential value but few applications, while consolidating mature strategies. This fundamentally avoids the blindness of traditional random exploration, enabling the system to quickly lock in the most effective optimization action with fewer and safer interactions in the rapidly changing operating conditions of the microgrid.

[0045] In one embodiment, the formula for the preset upper bound of the confidence interval is: ; Where s is the state vector at the current moment, a is the available action under the state vector at the current moment, UCB(s,a) is the upper bound score of the confidence interval of the available action under the state vector at the current moment, Q(s,a) is the Q value of the available action under the state vector at the current moment, c is the exploration coefficient, N(s,a) is the number of times the available action under the state vector at the current moment has been selected in history, and N(s) is the total number of times the state has been visited.

[0046] Thus, through rigorous mathematical expression, the trade-off mechanism between "utilizing experience" and "exploring the unknown" is solidified. The Q-value represents the known performance of an action, while the additional term quantifies its uncertainty. The more state visits and the fewer attempts at a specific action, the larger this term becomes, incentivizing the model to explore it. This allows the system to automatically and purposefully test potentially effective control strategies that are not yet fully validated, rather than blindly and randomly trying new operating conditions in microgrids, such as sudden changes in renewable energy sources. Simultaneously, the exploration coefficient provides a safety adjustment valve for this process, ensuring that the exploration behavior remains within a controllable range. This allows the model to quickly adapt to system changes with minimal and safe interactions, significantly accelerating the discovery and convergence of superior strategies.

[0047] In step S4: Power supply quality is optimized in real time based on the optimal action at the current moment.

[0048] It should be noted that this step converts the abstract action instructions output by the UCB-H learning model into modulated signals that can be executed by physical layer power electronic devices.

[0049] This transforms intelligent decision-making into actual control, driving devices such as energy storage converters to quickly perform operations such as reactive power compensation and harmonic suppression. It enables millisecond-level real-time adjustment of issues such as voltage fluctuations and waveform distortion, allowing optimization strategies to directly act on the physical system from algorithm output, forming a complete "decision-execution" closed loop, and improving the immediacy and effectiveness of power supply quality control.

[0050] In one embodiment, the real-time optimization of power quality based on the optimal action at the current moment includes the following steps: The optimal action at the current moment is parsed into the corresponding low-level control instructions of the power electronic equipment; The corresponding pulse width modulation signal is generated based on the underlying control instructions of the power electronic device; The electrical quantities of the microgrid are adjusted based on the pulse width modulation signal.

[0051] It should be noted that a decision generation module can be configured, which parses the optimal action into a set of setpoint instructions for specific devices; an execution control module can be configured, which receives the instruction set and uses mature power electronic control algorithms to generate underlying drive signals; in this embodiment, optionally, for the energy storage converter PCS, a current inner loop control based on dq-axis decoupling is used to convert the reactive power compensation command into a q-axis current reference value, and a drive signal for the switching transistor is generated through a PI regulator and PWM modulation to achieve fast and accurate output of reactive power; in this embodiment, optionally, for the active power filter APF, a harmonic detection algorithm combined with the action command is used to generate a reference waveform of the harmonic current to be compensated, and a PWM signal for the APF converter is generated through current tracking control to dynamically inject reverse harmonic current to offset load harmonics; in this embodiment, optionally, for the static var generator SVG, the required capacitive or inductive reactive current is calculated and output in real time according to the dynamic reactive power compensation coefficient, and the generated PWM drive signal is applied to the power switching transistor of the converter, thereby actually changing the current / voltage injected into the microgrid and completing the physical adjustment of power supply quality.

[0052] In one embodiment, after the step of performing real-time power quality optimization based on the optimal action at the current moment, the method further includes: Collect electrical quantity data and equipment status data for the next moment; Based on the electrical quantity data at the next moment, the power supply quality index for the next moment is calculated; based on the power supply quality index and the equipment status data at the next moment, the state vector for the next moment is constructed. Based on the state vector at the next time step, the Q-value function of the UCB-H reinforcement learning model is queried to obtain the Q-value of each selectable action under the state vector at the next time step; based on the state vector at the next time step and the Q-value of each selectable action under the state vector at the next time step, the policy evaluation loss value is calculated; the changing trend of the policy evaluation loss value in multiple consecutive control cycles is monitored to determine whether the policy of the UCB-H reinforcement learning model has converged. If the policy of the UCB-H reinforcement learning model fails to converge, the Q-value function and action access count table of the UCB-H reinforcement learning model are updated based on the immediate reward obtained after executing the optimal action at the current moment, the state vector at the current moment, the optimal action at the current moment, and the state vector at the next moment. The process of inputting the state vector at the current moment into the UCB-H reinforcement learning model is then returned to make a decision for the next control cycle. If the determination result is that the policy has converged, the Q-value function of the UCB-H reinforcement learning model is locked, and the optimal action is output using the Q-value function of the UCB-H reinforcement learning model in subsequent control cycles.

[0053] It should be noted that this step constitutes the feedback loop of the optimization closed loop, used to evaluate the performance of the current strategy and determine whether the learning process tends to stabilize. This is the key to deciding whether to deploy a new strategy. The agent executes actions in the real microgrid environment. The actions act on physical devices through the execution layer, and the microgrid operating state changes. After one control cycle, the new state and immediate reward are collected. The Q-Learning update rule is used, combined with the Q-value experience of the new state, to correct the estimate of the Q-value function and update the access count. The next moment is the moment after a delay of one control cycle.

[0054] Therefore, by calculating the loss value of the strategy evaluation, an objective and quantitative basis is provided for judging whether the current control strategy has become stable. When the strategy has not converged, the system can use the immediate reward after the action is executed and the new system state to update the model in real time, so that it can continuously learn and improve from actual interaction. Once the strategy converges, the current efficient Q-value function is locked and the optimal action is directly output, which greatly improves the response efficiency and operational stability. This allows the system to not only quickly find the best in the early stage, but also to have the intelligence of learning and using immediately and learning again when changes occur. It realizes a smooth and adaptive switch from continuous exploration to reliable operation, and enhances the applicability and reliability of long-term operation.

[0055] In one embodiment, the formula used to update the Q-value function of the UCB-H reinforcement learning model is: ; Where s_t is the state vector at time t, a_t is the optional action under the state vector at time t, Q(s_t,a_t) is the Q value of the optional action under the state vector at time t, r_t is the immediate reward obtained by performing the optional action under the state vector at time t, Q(s_{t+1},a) is the Q value of the optional action under the state vector at time t+1, α is the learning rate and 0<α≤1, and γ is the discount factor and 0≤γ<1.

[0056] It should be noted that the learning rate is used to control the strength of new information covering old information, and an adaptive learning rate can be used, which is fast in the early stage and fine-tuned in the later stage; the discount factor is used to measure the current value of future rewards.

[0057] In one embodiment, the formula used to update the action access count table is: N(s_t)+=1; N(s_t,a_t)+=1; Where N(s_t) is the total number of times the state vector at time t is accessed, and N(s_t,a_t) is the total number of times the available actions under the state vector at time t are historically selected.

[0058] In one embodiment, calculating the policy evaluation loss value based on the state vector at the next time step and the Q-values ​​of each selectable action under the state vector at the next time step specifically includes the following steps: Based on the state vector at the next moment and the Q-values ​​of each selectable action under the state vector at the next moment, the policy probability distribution of the UCB-H reinforcement learning model at the current moment is generated; Based on the probability distribution of the policy, calculate the cross-entropy loss value, and use the cross-entropy loss value as the policy evaluation loss value.

[0059] It should be noted that the cross-entropy loss is introduced as a quantitative tool for judging policy convergence, rather than for backpropagation during model training. Given the state vector at time t, the UCB-H model, based on the Q-value table and UCB formula at time t, calculates a "preference" or "inclination" for each possible action. This inclination can be transformed into a policy probability distribution using the Softmax function. The optimal policy should tend towards determinism upon convergence; that is, for the state vector at time t, the probability of the optimal action is close to 1, and the probabilities of other actions are close to 0. The cross-entropy loss measures the difference between the current policy distribution and the ideal deterministic distribution. In practice, the cross-entropy loss is simplified to the negative logarithm of the sum of probabilities of non-optimal actions under the current policy, or more practically, monitoring the logarithm of the optimal action probability. The smaller the loss value, the closer the current policy is to the deterministic optimal policy.

[0060] Therefore, by introducing cross-entropy loss as the core indicator for policy evaluation, a precise "measuring stick" is provided for the optimization process. The model's preference for each action in the next moment is transformed into a calculable scalar loss. The smaller the loss value, the clearer and more confident the model's judgment on the optimal action in the current state. At the same time, by continuously monitoring the change of this loss value, the system can clearly and quantitatively perceive the dynamic improvement of the policy, thereby transforming the convergence judgment from subjective experience to objective data-driven. This enables the system to keenly identify learning plateaus, avoid ineffective continuous exploration, or promptly capture policy degradation, providing a reliable and direct decision-making basis for the intelligent switching between learning and locking modes, and ensuring the efficiency and stability of the entire optimization process.

[0061] In one embodiment, the monitoring strategy evaluates the trend of the loss value over multiple consecutive control periods to determine whether the policy of the UCB-H reinforcement learning model has converged, specifically including the following steps: Monitor the numerical change of the cross-entropy loss value over multiple consecutive control cycles; If the cross-entropy loss value shows a monotonically decreasing trend within a consecutive first preset number of control cycles, and the fluctuation range of the cross-entropy loss value within a consecutive first preset number of control cycles is greater than a first preset threshold, then it is determined that the strategy has not converged. If the fluctuation range of the cross-entropy loss value within a second preset number of consecutive control cycles is less than a second preset threshold, then the strategy is determined to have converged.

[0062] It should be noted that, in this embodiment, preferably, if the moving average of the cross-entropy loss value shows a monotonically decreasing trend over at least 5 consecutive control cycles, and the fluctuation range of adjacent control cycles is greater than 10... -3 If the strategy fails to converge, the system will continue running steps S2-S4 and update the Q-value function and action access count table of the UCB-H reinforcement learning model online. In this embodiment, preferably, if the cross-entropy loss value plateaus within 5 consecutive periods, the fluctuation range of the cross-entropy loss value is less than 5 × 10⁻⁶. -4 If the corresponding power quality indicators have stabilized within the target range, then the UCB-H model is considered to have learned the optimal strategy for the current operating conditions, and the learning process has converged. Once convergence is confirmed, the system can solidify the optimal action selection strategy under the current state and deploy it to the fast execution channel. For a period of time, the system can directly select the optimal action based on the Q table, skipping the UCB exploration calculation, to further improve the response speed. At the same time, the system continuously monitors the power quality indicators, the state vector, and the cross-entropy loss value at time t+1. If the power quality indicators exceed the limit again or the cross-entropy loss value increases significantly due to significant changes in load, renewable energy output, etc., the system will immediately unlock and reactivate the complete real-time optimization of the microgrid cyber-physical system power quality to adapt to the new operating conditions.

[0063] Therefore, by setting clear quantitative criteria, the judgment of policy convergence becomes objective, automatic, and reliable. When the cross-entropy loss value continues to decrease significantly, it indicates that the model is in an active optimization learning phase, and the system will continue to explore and update. When the loss value enters and remains within a very small fluctuation range, it indicates that the policy has become stable, and the system can switch to an efficient locked execution mode. This transforms policy management from relying on subjective experience to making accurate decisions based on data, effectively avoiding ineffective exploration or premature convergence caused by the inability to accurately judge the learning state in traditional methods. It not only ensures the sufficiency of the learning process but also ensures that the system can run stably and reliably after the policy matures, thus achieving a balance between optimization efficiency and operational robustness in a dynamic environment.

[0064] Example 2 like Figure 2As shown, based on the same inventive concept as the above embodiments, a second aspect of the present invention also provides a real-time power quality optimization device for a microgrid cyber-physical system, comprising: The quality monitoring module is used to collect electrical quantity data and equipment status data at the current moment; The quality monitoring module is also used to calculate the power supply quality index at the current moment based on the electrical quantity data at the current moment; and to construct the state vector at the current moment based on the power supply quality index at the current moment and the equipment status data at the current moment. The decision generation module is used to input the current state vector into the UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains a Q-value function and an action access count table; for each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval of each selectable action under the current state vector based on the Q-value and action access count of each selectable action under the current state vector, and selects the selectable action with the highest upper bound score of the confidence interval as the optimal action at the current time; The execution control module is used to optimize the power supply quality in real time based on the optimal action at the current moment.

[0065] Example 3 like Figure 3 As shown, in a third aspect, the present invention also provides an electronic device 100 for implementing the real-time power quality optimization method for a microgrid cyber-physical system as provided in any of the above embodiments; The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0066] The memory 101 can be used to store the computer program 103. The processor 102 implements the real-time power quality optimization method steps of the microgrid cyber-physical system provided in any of the above embodiments by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0067] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0068] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0069] The memory 101 in the electronic device 100 stores multiple instructions to implement a real-time power quality optimization method for a microgrid cyber-physical system, and the processor 102 can execute multiple instructions to achieve the following: Collect current electrical quantity data and equipment status data; Based on the electrical quantity data at the current moment, the power supply quality index at the current moment is calculated; based on the power supply quality index at the current moment and the equipment status data at the current moment, the state vector at the current moment is constructed. The current state vector is input into the UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains a Q-value function and an action access count table; for each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval of each selectable action under the current state vector based on the Q-value and action access count of each selectable action under the current state vector, and selects the selectable action with the highest upper bound score of the confidence interval as the optimal action at the current time; Power quality is optimized in real time based on the optimal action at the current moment.

[0070] Example 4 In a fourth aspect, the present invention provides a computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the real-time power quality optimization method for a microgrid cyber-physical system as provided in any of the preceding embodiments.

[0071] It should be noted that if the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0072] Therefore, the real-time power quality optimization device, electronic device and computer-readable storage medium of the microgrid cyber-physical system provided by the present invention also solves the problems of insufficient real-time performance and adaptability of traditional model-dependent methods proposed in the background section, as well as the low efficiency and slow convergence of the original exploration mechanism of existing reinforcement learning.

[0073] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for real-time optimization of power supply quality in a microgrid cyber-physical system, characterized in that, include: Collect current electrical quantity data and equipment status data; Based on the electrical quantity data at the current moment, the power supply quality index at the current moment is calculated; The current state vector is constructed based on the power quality index and the device status data at the current moment. The current state vector is input into the UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains a Q-value function and an action access count table; for each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval of each selectable action under the current state vector based on the Q-value and action access count of each selectable action under the current state vector, and selects the selectable action with the highest upper bound score of the confidence interval as the optimal action at the current time; Power quality is optimized in real time based on the optimal action at the current moment.

2. The method for real-time optimization of power supply quality in a microgrid cyber-physical system according to claim 1, characterized in that, The power supply quality indicators include voltage deviation rate, frequency deviation value, total harmonic distortion rate and interharmonic components, and the equipment status data includes distributed power output fluctuation, energy storage SOC status and critical load sensitivity level.

3. The method for real-time optimization of power supply quality in a microgrid cyber-physical system according to claim 2, characterized in that, For each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval for each selectable action under the current state vector based on the Q-value and action access count, specifically including the following steps: Based on the action access count table, determine the total number of times the state corresponding to the state vector at the current moment is accessed, and the historical number of times each optional action under the state vector at the current moment is selected; For each selectable action under the state vector at the current time, the upper bound score of the confidence interval for each selectable action under the state vector at the current time is calculated using a preset upper bound formula based on the Q value of each selectable action under the state vector at the current time, the total number of times the state has been accessed, and the historical number of times each selectable action under the state vector at the current time.

4. The method for real-time optimization of power supply quality in a microgrid cyber-physical system according to claim 3, characterized in that, The formula for the upper bound of the preset confidence interval is: ； Where s is the state vector at the current moment, a is the available action under the state vector at the current moment, UCB(s,a) is the upper bound score of the confidence interval of the available action under the state vector at the current moment, Q(s,a) is the Q value of the available action under the state vector at the current moment, c is the exploration coefficient, N(s,a) is the number of times the available action under the state vector at the current moment has been selected in history, and N(s) is the total number of times the state has been visited.

5. The method for real-time optimization of power supply quality in a microgrid cyber-physical system according to claim 1, characterized in that, After the step of performing real-time power quality optimization based on the optimal action at the current moment, the method further includes: Collect electrical quantity data and equipment status data for the next moment; Based on the electrical quantity data at the next moment, the power supply quality index for the next moment is calculated; based on the power supply quality index and the equipment status data at the next moment, the state vector for the next moment is constructed. Based on the state vector at the next time step, the Q-value function of the UCB-H reinforcement learning model is queried to obtain the Q-value of each selectable action under the state vector at the next time step; based on the state vector at the next time step and the Q-value of each selectable action under the state vector at the next time step, the policy evaluation loss value is calculated; the changing trend of the policy evaluation loss value in multiple consecutive control cycles is monitored to determine whether the policy of the UCB-H reinforcement learning model has converged. If the policy of the UCB-H reinforcement learning model fails to converge, the Q-value function and action access count table of the UCB-H reinforcement learning model are updated based on the immediate reward obtained after executing the optimal action at the current moment, the state vector at the current moment, the optimal action at the current moment, and the state vector at the next moment. The process of inputting the state vector at the current moment into the UCB-H reinforcement learning model is then returned to make a decision for the next control cycle. If the determination result is that the policy has converged, the Q-value function of the UCB-H reinforcement learning model is locked, and the optimal action is output using the Q-value function of the UCB-H reinforcement learning model in subsequent control cycles.

6. The method for real-time optimization of power supply quality in a microgrid cyber-physical system according to claim 5, characterized in that, The step of calculating the policy evaluation loss value based on the state vector at the next time step and the Q-values ​​of each selectable action under the state vector at the next time step specifically includes the following steps: Based on the state vector at the next moment and the Q-values ​​of each selectable action under the state vector at the next moment, the policy probability distribution of the UCB-H reinforcement learning model at the current moment is generated; Based on the probability distribution of the policy, calculate the cross-entropy loss value, and use the cross-entropy loss value as the policy evaluation loss value.

7. The method for real-time optimization of power supply quality in a microgrid cyber-physical system according to claim 6, characterized in that, The monitoring strategy evaluates the trend of the loss value over multiple consecutive control periods to determine whether the strategy of the UCB-H reinforcement learning model has converged, specifically including the following steps: Monitor the numerical change of the cross-entropy loss value over multiple consecutive control cycles; If the cross-entropy loss value shows a monotonically decreasing trend within a consecutive first preset number of control cycles, and the fluctuation range of the cross-entropy loss value within a consecutive first preset number of control cycles is greater than a first preset threshold, then it is determined that the strategy has not converged. If the fluctuation range of the cross-entropy loss value within a second preset number of consecutive control cycles is less than a second preset threshold, then the strategy is determined to have converged.

8. A real-time power quality optimization device for a microgrid cyber-physical system, characterized in that, include: The quality monitoring module is used to collect electrical quantity data and equipment status data at the current moment; The quality monitoring module is also used to calculate the power supply quality index at the current moment based on the electrical quantity data at the current moment; and to construct the state vector at the current moment based on the power supply quality index at the current moment and the equipment status data at the current moment. The decision generation module is used to input the current state vector into the UCB-H reinforcement learning model; the UCB-H reinforcement learning model maintains a Q-value function and an action access count table; for each selectable action under the current state vector, the UCB-H reinforcement learning model calculates the upper bound score of the confidence interval of each selectable action under the current state vector based on the Q-value and action access count of each selectable action under the current state vector, and selects the selectable action with the highest upper bound score of the confidence interval as the optimal action at the current time; The execution control module is used to optimize the power supply quality in real time based on the optimal action at the current moment.

9. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the real-time power quality optimization method for a microgrid cyber-physical system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the real-time power quality optimization method for a microgrid cyber-physical system as described in any one of claims 1 to 7.

Images