Method for strategy optimization for agent cooperative game

Abstract

The application relates to the technical field of power supply and demand balance control, and discloses a strategy optimization method for agent collaborative game, which comprises the following steps: collecting electrical data of a distributed energy control node to construct a power loss function; maintaining a sensitivity vector representing local curvature of a parameter space; calculating the dot product of a strategy gradient and a virtual momentum anchor vector to determine an adjustment conflict; applying non-uniform spatial stiffness damping to a conflict state based on the sensitivity; and reconstructing a power strategy update instruction to adjust active and reactive set points. The application can inhibit excessive update of high-sensitivity parameter sites, eliminate tail oscillation when approaching a power balance point, and ensure voltage and frequency stability of a local power grid in a non-stationary environment.

Description

Technical Field

[0001] This invention relates to a strategy optimization method for intelligent agent cooperative game, belonging to the field of power supply and demand balance control technology. Background Technology

[0002] Currently, in complex engineering systems such as power dispatch networks, automated logistics clusters, and industrial process control, multi-node collaborative control is the core technical architecture for maximizing overall system efficiency. These systems typically contain multiple control units (agents) with independent computing capabilities. Each agent needs to adjust its control strategy in a dynamic environment with incomplete information by collecting local observation data and interacting with other nodes to maintain the system's supply-demand balance or achieve collaborative operation goals. Currently, gradient-based decision-making algorithms are widely used in such scenarios. Their basic working principle is to use the policy gradient method to calculate the parameter update direction based on the reward signal from environmental feedback, guiding the policy parameters of each control node to gradually evolve to a steady-state operating point that satisfies global constraints. Under ideal conditions where the system state is relatively stable and communication latency is negligible, this iterative mechanism based on the negative gradient direction can effectively drive system convergence. Although the existing underlying physical infrastructure of distributed systems... While possessing high data acquisition accuracy and communication bandwidth, simply relying on hardware stacking cannot solve collaborative game problems in complex dynamic environments. The key lies in strategy optimization at the control algorithm level. For example, Chinese invention patent application CN117114136A discloses a multi-agent distributed deep deterministic policy gradient tracking method. This method reduces computational costs through a decentralized execution architecture and distributed evaluators, and optimizes training using an experience replay buffer and a loss-minimizing update function. However, although this type of scheme has certain advantages in general target tracking tasks, its core logic is still based on an ideal isotropic gradient update mechanism. It does not fully consider the non-uniform curvature characteristics exhibited when the high-dimensional parameter space approaches steady state, and lacks proactive compensation methods for the physical phase of long-link communication delays. Faced with typical non-stationary dynamic environments such as drastic fluctuations in grid photovoltaic power, such conventional gradient methods are unable to suppress system-level tail oscillations caused by perturbations of local highly sensitive parameters, and cannot meet the stringent frequency stability requirements of the power grid.

[0003] However, in actual physical control systems, while each control node updates its own strategy, the strategies of other related nodes also change concurrently. This concurrency means that the controlled environment faced by each node is actually a non-stationary dynamic process. Due to the inherent timing deviations in the computation cycle and communication links between nodes, the gradient update direction calculated by the node based on the current observations often lags behind the actual evolution direction of the system's global state. Especially in the critical stage when the system approaches the target equilibrium point, this misalignment of the gradient vector in the spatiotemporal dimension can easily lead to excessive compensatory adjustments in the positive and negative directions of the control command. At this time, the system is difficult to stably stop at the Nash equilibrium point, but instead exhibits high-frequency oscillations in the control output between extreme values. This not only reduces the control accuracy of the system, but may also cause mechanical fatigue of physical equipment or power grid fluctuations due to frequent actions of actuators. In addition, when dealing with parameter updates of control models, existing technologies usually assume that the parameter space has isotropic geometric characteristics, that is, they assume that the sensitivity of all control parameters to the system output is uniform. However, in high-dimensional nonlinear control models, the parameter space often exhibits... Complex heterogeneous manifold structures, with parameter sites in specific high-curvature regions, are extremely sensitive to gradient responses. Even minute numerical updates can be amplified through nonlinear processes, leading to drastic changes in the output strategy. Existing unified update step-size mechanisms struggle to identify and adapt to these stiffness differences in the parameter space, causing the system to be prone to strategy collapse in locally sensitive regions. Furthermore, the prevalent communication latency in distributed networks inevitably results in phase lag in the collaborative information acquired by nodes. Traditional optimization methods based on historical momentum are essentially lag compensations and cannot fundamentally eliminate the phase difference caused by physical transmission delays, resulting in the control system's response consistently lagging behind real-time environmental changes. To address these issues, conventional industry solutions typically involve introducing centralized evaluation mechanisms or simply increasing the communication frequency between nodes. While centralized evaluation can mitigate non-stationarity, it places extremely high demands on the computational load and communication bandwidth of the central node, making it unsuitable for the real-time requirements of large-scale distributed systems. Simply increasing the communication frequency not only significantly increases network load but also fails to eliminate the rigid latency determined by physical distance and processing time.

[0004] Therefore, the technical problem to be solved by this invention is how to construct a cooperative control strategy that can smooth gradient perturbations in non-stationary environments, adaptively adjust and update stiffness based on the geometric characteristics of the parameter space, and effectively offset the impact of communication delay. Summary of the Invention

[0005] To address the problems mentioned in the background art, the technical solution of the present invention is as follows: A strategy optimization method for agent cooperative game, comprising the following steps: By deploying electrical measurement units at the output end of distributed energy control nodes, real-time voltage, current data and power flow status at the grid connection point are acquired. Based on the supply and demand deviation of the entire network and the thermal stability constraints of the lines, a power loss function under non-stationary dynamic environment is constructed, and the original strategy gradient vector of the current power dispatch cycle is calculated accordingly. A parameter sensitivity vector consistent with the power control strategy parameter dimension is maintained synchronously. The parameter sensitivity vector, as a physical field quantity characterizing the local curvature characteristics of the control parameter space under random disturbances of grid load, is obtained by updating the exponentially weighted moving average based on the second-order moment estimation of the historical power error gradient. Calculate the dot product of the original strategy gradient vector and the virtual momentum anchor vector that represents the long-term steady-state power flow direction of the power grid. When the value of the dot product is less than 0, it is identified that the adjustment direction of the current control node is in reverse opposition to the long-term evolution trend of the power grid, and it is determined to be a gradient-trend conflict state of power regulation. In response to the gradient-trend conflict state, non-uniform stiffness damping is applied to the control parameter update space under this state based on the parameter sensitivity vector. By suppressing the update step size of high-sensitivity parameter sites, the power strategy update command that can smooth out grid disturbances is reconstructed. The power strategy update command is used to generate the corresponding underlying inverter drive signal to physically adjust the active power output setpoint and reactive power compensation of the distributed energy control node, thereby eliminating the tail oscillation of the control output when approaching the power balance point of the grid.

[0006] Preferably, the update process of the parameter sensitivity vector follows these rules: In each power dispatch cycle, the squared values ​​of each component element of the original strategy gradient vector are extracted as the instantaneous fluctuation energy representation of the control signal in the current cycle; using a preset smoothing factor, a weighted iterative operation is performed on the parameter sensitivity vector and the instantaneous fluctuation energy representation of the previous cycle to generate the parameter sensitivity vector of the current cycle; wherein, the value of the smoothing factor is set between 0.9 and 0.999, so that the parameter sensitivity vector has low-pass filtering characteristics for high-frequency random jitter of the grid frequency, thereby locking in the parameter topography features that reflect the long-term evolution law of the power system.

[0007] Preferably, the step of performing weighted iterative calculations using a preset smoothing factor specifically involves updating the parameter sensitivity vector based on the following mathematical relationship: ,in, This is the parameter sensitivity vector for the current scheduling period. Let be the parameter sensitivity vector of the previous scheduling cycle, and β be the smoothing factor. The original policy gradient vector for the current scheduling cycle; ⊙ represents the Hadamard product operation; parameter sensitivity vector. Each component directly corresponds to the updated resistance physical quantity of a specific weight in the power control strategy network.

[0008] Preferably, the construction steps of the virtual momentum anchor vector include: establishing a historical gradient storage queue of length N in the memory to store the effective gradient vectors generated by the distributed energy control nodes in the past N scheduling cycles; performing a weighted summation operation based on the time decay factor on the vectors in the historical gradient storage queue to generate a momentum trend vector characterizing the long-term evolution direction of the grid power flow; and using the momentum trend vector as a reference axis for smoothing out instantaneous electrical noise interference to verify whether the original policy gradient vector of the current cycle deviates from the convergence trajectory of the system towards steady state.

[0009] Preferably, the step of applying non-uniform stiffness damping includes: mapping each component value of the parameter sensitivity vector to a preset stiffness coefficient range based on the grid frequency fluctuation spectral density, generating a stiffness coefficient vector that corresponds one-to-one with the control parameters; for components in the parameter sensitivity vector whose values ​​exceed a preset high-frequency threshold, setting their corresponding stiffness coefficients as enhanced damping values, wherein the enhanced damping values ​​are positively correlated with the sensitivity amplitude of the component; and using the stiffness coefficient vector to perform element-wise division scaling on the original strategy gradient vector, thereby forcibly attenuating the update amplitude of the parameter dimension that is highly sensitive to random disturbances of grid voltage or frequency.

[0010] Preferably, after the step of determining the gradient-trend conflict state of power regulation, the method further includes: if the value of the dot product is greater than or equal to 0, determining that the current control is in the trend-following regulation state; in the trend-following regulation state, triggering the stiffness damping bypass logic, directly using the original strategy gradient vector superposition to multiply the virtual momentum anchoring vector by the preset perturbation coefficient to obtain the perturbation component as the power strategy update instruction, so as to accelerate the convergence speed of the distributed energy control node to the target power operating point.

[0011] Preferably, the steps for constructing a power loss function under non-stationary dynamics based on the supply-demand deviation of the entire network include: obtaining marginal electricity price information and line load rate information exchanged between adjacent distributed energy control nodes through a communication interface; constructing a composite objective function that includes a voltage limit penalty term and a frequency deviation penalty term; and taking the partial derivative of the composite objective function with respect to the control strategy parameters to obtain gradient information reflecting the adjustment direction of each parameter under the current grid condition, which is used as an environmental feedback signal under non-stationary dynamics.

[0012] Preferably, the step of generating the corresponding underlying inverter drive signal using the power strategy update instruction includes: decomposing the reconstructed power strategy update instruction into the duty cycle adjustment amount of the inverter's pulse width modulation signal; inputting the duty cycle adjustment amount to the underlying drive circuit of the distributed energy control node; and before executing the duty cycle adjustment amount, verifying whether the adjusted output power is within the physical safety constraint range of the node, and if it exceeds the range, performing a truncation operation based on the boundary gradient.

[0013] Preferably, the method is used to solve the collaborative control problem of large-scale photovoltaic clusters under the scenario of severe power fluctuations caused by cloud shading; non-stationary dynamic environment specifically refers to the physical process in which the output power of the photovoltaic array exhibits intermittent and random jumps due to the rapid movement of clouds; tail oscillation specifically refers to the phenomenon of continuous high-frequency power oscillation of inverter output power near the theoretical maximum value caused by the time delay of multi-machine collaborative gradient during the maximum power point tracking process.

[0014] Preferably, after adjusting the power output setpoint of the distributed energy control node using the power strategy update command, the method further includes: real-time monitoring of the frequency change rate at the grid common connection point; when the absolute value of the frequency change rate is less than a preset steady-state judgment threshold, gradually reducing the stiffness damping corresponding to the integral control term in the parameter sensitivity vector to restore the system's dynamic response capability to small load changes; when the absolute value of the frequency change rate exceeds the steady-state judgment threshold, identifying it as a large disturbance state in the grid, executing emergency frequency response control or switching to a preset safety protection mode to maintain the voltage and frequency stability of the local grid.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. In the strategy optimization of agent cooperative game, based on the steady-state control of the system anchored by time-series inertia, this invention introduces the cooperative momentum feature reflecting the historical evolution trend into the strategy update loop of the distributed control unit, and constructs a damping control mechanism with time-series viscoelasticity. It uses the integral effect of historical control signals to smooth high-frequency random disturbances in instantaneous feedback, so that each control node can lock the evolution direction according to the long-term accumulated consensus trend when facing non-stationary fluctuations in the system environment, rather than just responding to the current instantaneous gradient. This logical anchoring based on physical inertia eliminates the tail oscillation phenomenon caused by mutual overcompensation in the high-frequency interaction of multiple nodes, ensures the convergence stability of the control strategy when the large-scale distributed system approaches the supply and demand equilibrium point, and avoids the risk of system divergence caused by the spatiotemporal misalignment of signals.

[0016] 2. Adaptive safety protection based on parameter space stiffness distribution: This invention establishes a parameter stiffness distribution field reflecting the vulnerability of the system by real-time monitoring of the second-order sensitivity response of each parameter point in the control network to the error gradient. When executing strategy updates, it can automatically identify and suppress the parameter adjustment amplitude in the high-sensitivity region, constraining the evolution path of the control signal within a low-sensitivity flat manifold space. This non-uniform gating based on the physical properties of parameters provides an inherent self-healing protection for nonlinear control systems, preventing the collapse of global control logic induced by small local parameter mutations under extreme conditions, and achieving a dynamic balance between rapid response to environmental changes and maintaining the intrinsic stability of the system.

[0017] 3. Based on orthogonal feature decoupling, this invention utilizes a geometric projection method to construct a feature protection subspace orthogonal to the global collaborative trend. This enables the physical separation and parallel processing of information in different dimensions of the control signal. While forcing each sub-node to follow global consistency constraints to maintain system synchronization, it retains the heterogeneous feature components projected in the orthogonal space through asymmetric gain adjustment. This allows each distributed node to maintain its independent adjustment capability for local special conditions while executing unified scheduling instructions, avoiding the degradation of local node functional characteristics due to excessive pursuit of global consensus. Thus, it achieves bidirectional optimization of system-level objectives and node-level characteristics in complex asymmetric network environments. Attached Figure Description

[0018] Figure 1 Main flowchart of the strategy optimization method for introducing a stiffness damping mechanism in this invention; Figure 2 This is a functional principle and logical architecture diagram of the strategy optimization method of the present invention.

[0019] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0021] A strategy optimization method for agent cooperative game includes the following steps: By deploying electrical measurement units at the output end of distributed energy control nodes, real-time voltage, current data and power flow status at the grid connection point are acquired. Based on the supply and demand deviation of the entire network and the thermal stability constraints of the lines, a power loss function under non-stationary dynamic environment is constructed, and the original strategy gradient vector of the current power dispatch cycle is calculated accordingly. A parameter sensitivity vector consistent with the power control strategy parameter dimension is maintained synchronously. The parameter sensitivity vector, as a physical field quantity characterizing the local curvature characteristics of the control parameter space under random disturbances of grid load, is obtained by updating the exponentially weighted moving average based on the second-order moment estimation of the historical power error gradient. Calculate the dot product of the original strategy gradient vector and the virtual momentum anchor vector that represents the long-term steady-state power flow direction of the power grid. When the value of the dot product is less than 0, it is identified that the adjustment direction of the current control node is in reverse opposition to the long-term evolution trend of the power grid, and it is determined to be a gradient-trend conflict state of power regulation. In response to the gradient-trend conflict state, non-uniform stiffness damping is applied to the control parameter update space under this state based on the parameter sensitivity vector. By suppressing the update step size of high-sensitivity parameter sites, the power strategy update command that can smooth out grid disturbances is reconstructed. The power strategy update command is used to generate the corresponding underlying inverter drive signal to physically adjust the active power output setpoint and reactive power compensation of the distributed energy control node, thereby eliminating the tail oscillation of the control output when approaching the power balance point of the grid.

[0022] Preferably, the update process of the parameter sensitivity vector follows these rules: In each power dispatch cycle, the squared values ​​of each component element of the original strategy gradient vector are extracted as the instantaneous fluctuation energy representation of the control signal in the current cycle; using a preset smoothing factor, a weighted iterative operation is performed on the parameter sensitivity vector and the instantaneous fluctuation energy representation of the previous cycle to generate the parameter sensitivity vector of the current cycle; wherein, the value of the smoothing factor is set between 0.9 and 0.999, so that the parameter sensitivity vector has low-pass filtering characteristics for high-frequency random jitter of the grid frequency, thereby locking in the parameter topography features that reflect the long-term evolution law of the power system.

[0023] Preferably, the step of performing weighted iterative calculations using a preset smoothing factor specifically involves updating the parameter sensitivity vector based on the following mathematical relationship: ,in, This is the parameter sensitivity vector for the current scheduling period. Let be the parameter sensitivity vector of the previous scheduling cycle, and β be the smoothing factor. The original policy gradient vector for the current scheduling cycle; ⊙ represents the Hadamard product operation; parameter sensitivity vector. Each component directly corresponds to the updated resistance physical quantity of a specific weight in the power control strategy network.

[0024] Preferably, the construction steps of the virtual momentum anchor vector include: establishing a historical gradient storage queue of length N in the memory to store the effective gradient vectors generated by the distributed energy control nodes in the past N scheduling cycles; performing a weighted summation operation based on the time decay factor on the vectors in the historical gradient storage queue to generate a momentum trend vector characterizing the long-term evolution direction of the grid power flow; and using the momentum trend vector as a reference axis for smoothing out instantaneous electrical noise interference to verify whether the original policy gradient vector of the current cycle deviates from the convergence trajectory of the system towards steady state.

[0025] Preferably, the step of applying non-uniform stiffness damping includes: mapping each component value of the parameter sensitivity vector to a preset stiffness coefficient range based on the grid frequency fluctuation spectral density, generating a stiffness coefficient vector that corresponds one-to-one with the control parameters; for components in the parameter sensitivity vector whose values ​​exceed a preset high-frequency threshold, setting their corresponding stiffness coefficients as enhanced damping values, wherein the enhanced damping values ​​are positively correlated with the sensitivity amplitude of the component; and using the stiffness coefficient vector to perform element-wise division scaling on the original strategy gradient vector, thereby forcibly attenuating the update amplitude of the parameter dimension that is highly sensitive to random disturbances of grid voltage or frequency.

[0026] Preferably, after the step of determining the gradient-trend conflict state of power regulation, the method further includes: if the value of the dot product is greater than or equal to 0, determining that the current control is in the trend-following regulation state; in the trend-following regulation state, triggering the stiffness damping bypass logic, directly using the original strategy gradient vector superposition to multiply the virtual momentum anchoring vector by the preset perturbation coefficient to obtain the perturbation component as the power strategy update instruction, so as to accelerate the convergence speed of the distributed energy control node to the target power operating point.

[0027] Preferably, the steps for constructing a power loss function under non-stationary dynamics based on the supply-demand deviation of the entire network include: obtaining marginal electricity price information and line load rate information exchanged between adjacent distributed energy control nodes through a communication interface; constructing a composite objective function that includes a voltage limit penalty term and a frequency deviation penalty term; and taking the partial derivative of the composite objective function with respect to the control strategy parameters to obtain gradient information reflecting the adjustment direction of each parameter under the current grid condition, which is used as an environmental feedback signal under non-stationary dynamics.

[0028] Preferably, the step of generating the corresponding underlying inverter drive signal using the power strategy update instruction includes: decomposing the reconstructed power strategy update instruction into the duty cycle adjustment amount of the inverter's pulse width modulation signal; inputting the duty cycle adjustment amount to the underlying drive circuit of the distributed energy control node; and before executing the duty cycle adjustment amount, verifying whether the adjusted output power is within the physical safety constraint range of the node, and if it exceeds the range, performing a truncation operation based on the boundary gradient.

[0029] Preferably, the method is used to solve the collaborative control problem of large-scale photovoltaic clusters under the scenario of severe power fluctuations caused by cloud shading; non-stationary dynamic environment specifically refers to the physical process in which the output power of the photovoltaic array exhibits intermittent and random jumps due to the rapid movement of clouds; tail oscillation specifically refers to the phenomenon of continuous high-frequency power oscillation of inverter output power near the theoretical maximum value caused by the time delay of multi-machine collaborative gradient during the maximum power point tracking process.

[0030] Preferably, after adjusting the power output setpoint of the distributed energy control node using the power strategy update command, the method further includes: real-time monitoring of the frequency change rate at the grid common connection point; when the absolute value of the frequency change rate is less than a preset steady-state judgment threshold, gradually reducing the stiffness damping corresponding to the integral control term in the parameter sensitivity vector to restore the system's dynamic response capability to small load changes; when the absolute value of the frequency change rate exceeds the steady-state judgment threshold, identifying it as a large disturbance state in the grid, executing emergency frequency response control or switching to a preset safety protection mode to maintain the voltage and frequency stability of the local grid.

[0031] Example 1: In a typical industrial application scenario where a distributed photovoltaic (PV) cluster with multiple grid-connected inverters is connected to a weak power grid, the grid faces the challenge of intermittent power fluctuations in the PV array output caused by rapid cloud movement. At this time, each distributed energy control node not only needs to track the maximum power point to ensure power generation efficiency, but also must respond to frequency and voltage fluctuation commands at the grid's point of common coupling. This multi-objective coupling leads to high-frequency power oscillations caused by communication delays and physical inertia misalignment when each node attempts to quickly eliminate supply-demand discrepancies. When the system is in such a non-stationary dynamic environment, the strategy optimization method of this invention is activated. Electrical measurement units deployed on the output side of each inverter collect three-phase voltage and current data at the grid connection point in real time with a sampling period of 10ms. The processor uses these discretized electrical physical quantities to construct a power loss function characterizing the current grid supply-demand balance state, and calculates the original strategy gradient vector for the current scheduling cycle by taking the partial derivative of this loss function with respect to the weight parameters in the control strategy network. To eliminate the dimensional gap between the abstract mathematical gradient and the underlying control commands, the system obtains the maximum absolute value of all components in the original policy gradient vector and divides each component by this maximum value to achieve dimensionless normalization. The normalized components are then multiplied by 0.01 times the current inverter's rated power as the step size reference, thus mapping the gradient values ​​to specific active power adjustment milliwatt values. However, the system does not directly use this gradient to update parameters; instead, it synchronously calls the parameter sensitivity vector maintained in memory. This is used as a physical field quantity characterizing the local curvature features of the control parameter space. Its value is not statically preset, but is based on the formula. Real-time iterative updates are performed; the smoothing factor β is set to 0.99, and ⊙ represents the Hadamard product operation. This calculation process accumulates the second moment information of the gradient of each parameter point within the historical period and solidifies it into a sensitivity index, thereby mathematically quantifying the response sensitivity of each control parameter to grid frequency jitter.

[0032] Next, the processor reads gradient data from the past 50 cycles in the historical gradient storage queue and generates a virtual momentum anchoring vector representing the long-term evolution direction of the power grid flow through exponential decay weighted calculation. The system performs the crucial conflict identification step and calculates the gradient vector of the original policy. With virtual momentum anchoring vector The dot product of the parameters indicates that the current regulation command generated based on instantaneous observations is inversely opposed to the long-term steady-state evolution trend of the power grid. The system determines that it is in a high-risk oscillation range of gradient-trend conflict. In this state, the core contradiction resolution mechanism is triggered: the processor determines the parameter sensitivity vector based on the parameter sensitivity vector. The distribution characteristics of the control parameters are used to apply non-uniform spatial stiffness damping to the control parameter space. The median value exceeds the preset high-frequency threshold. For high-sensitivity components, the system reduces the corresponding gradient update step size to 10% to 20% of the original value, while retaining the original update amplitude for low-sensitivity components. This differentiated processing based on the physical properties of the parameter space achieves a non-obvious synergistic effect: on the one hand, the virtual momentum anchoring vector d is used to lock the overall trajectory of the system towards the power equilibrium point, ensuring the global correctness of the adjustment direction; on the other hand, the parameter sensitivity vector... Precise suppression of high-risk parameter points eliminates tail oscillations caused by excessive responses of local parameters. Ultimately, the reconstructed power strategy update command is converted into a PWM duty cycle signal for the inverter's underlying drive circuit, driving the power semiconductor devices. In the command mapping stage, the system increments the reconstructed active power update command in 1000W steps, with each 1000W power adjustment requirement corresponding to a 0.002 PWM duty cycle increment. This increment is then directly applied to the power transistor drive circuit via the underlying timer comparison register. The duty cycle adjustment range is rigidly limited to between 0.05 and 0.95. In actual grid-connected testing, even under extreme conditions where the irradiance fluctuates drastically at a rate of 200W / m² / s, the active power output smoothness of this distributed photovoltaic cluster is improved, and the frequency deviation at the point of common coupling remains within a safe range of ±0.05Hz. The system achieves adaptive suppression and steady-state convergence of grid non-stationary disturbances without reducing the overall dynamic response speed.

[0033] Example 2: To verify the engineering effectiveness of the optimization method of the present invention under non-stationary dynamic environments and the boundary rationality of key control parameters, a real-time digital simulation hardware-in-the-loop test platform for a microgrid with 33 physical nodes was built. The topology followed the IEEE 33-node distribution system standard. Nodes 5, 12, 18, 22, and 33 were connected to distributed photovoltaic inverter control units with a rated capacity of 50kW. The test environment was configured with a programmable AC power supply to simulate the grid background of the weak grid point of common coupling (PCC). A signal generator was introduced to inject Gaussian white noise with a signal-to-noise ratio of 20dB into the voltage and current acquisition loops. At the same time, a random delay following a Poisson distribution was loaded into the communication links between the distributed nodes, with an average delay set to 25ms, to simulate electromagnetic interference and communication lag conditions in industrial settings. The experiment verified the selection logic of the core control parameters, namely the smoothing factor β and the high-frequency threshold κ. Parameter β determines the parameter sensitivity vector. The memory length of historical gradient information needs to be balanced between the ability to suppress short-term noise and the ability to track sudden changes in system state. If the β value is too low, the sensitivity vector will fluctuate drastically with the instantaneous gradient, leading to false triggering of stiffness damping; if the β value is too high, the system will not be able to perceive the time-varying characteristics of parameter curvature in a timely manner. Based on the constraints of the Nyquist sampling theorem and the power grid frequency cycle, the sampling period was set to 10ms. In the first stage of parameter optimization experiments, three different smoothing factor β values ​​were set for comparison: 0.90, 0.99, and 0.999. Under the condition of a 50% step change in photovoltaic output, the data shows that when β is 0.90, the variance of the sensitivity vector reaches This causes frequent jumps in the damping coefficient; when β is 0.999, the system response time increases to 3.5s; while when β is set to 0.99, the variance of the sensitivity vector decreases to Furthermore, the system response time remains within the preferred range of 1.2s. Based on this, β is locked at 0.99 in this experiment. Similarly, for the high-frequency threshold κ, it is determined through gradient testing. To effectively distinguish the critical inflection point values ​​of high-sensitivity parameters from those of conventional parameters, a comparative experimental system was constructed under this benchmark parameter, comprising the sample group of this invention, a completely missing control group, and a partially missing control group. The sample group of this invention employs a complete strategy optimization method, including virtual momentum anchoring and a sensitivity-based stiffness gating mechanism. The completely missing control group (control group A) uses the standard distributed multi-agent reinforcement learning algorithm (MADDPG) without introducing any momentum anchoring or stiffness gating. The partially missing control group (control group B) retains only the virtual momentum anchoring mechanism and removes the sensitivity-based stiffness gating logic. The experiment simulates the dynamic process of a 0.2Hz frequency drop in the power grid due to a sudden increase in load.

[0034] In the initial phase after the experiment started, i.e., within the interval of 0 to 0.5 seconds, all three control strategies detected the frequency deviation signal and generated the original strategy gradient vector aimed at improving the active power output. Due to the injection of Gaussian white noise, the gradient vector generated by the control group A exhibited severe jitter on the time axis, resulting in high-frequency oscillations in the inverter output power with an amplitude of 15kW. The frequency deviation at the PCC point fluctuated within ±0.15Hz and failed to converge after 2.0s, exhibiting obvious tail oscillation characteristics. In contrast, the sample group of this invention demonstrated drastically different dynamic behavior. At the instant of detecting a frequency drop, the processor calculated the virtual momentum anchoring vector d based on the gradient history of the past 50 cycles. Although the instantaneous noise caused the current original policy gradient vector to... A sudden change occurred, deviating from the steady-state direction, causing The dot product of d and d results in -0.08 (less than 0), indicating a gradient-trend conflict. The system then calls the parameter sensitivity vector. By scanning the weight parameters in the control network, the sensitivity component values ​​of a specific subset of parameters responsible for voltage loop control were identified. Higher than the preset The threshold is then set, and the control logic forcibly decays the update step size of these highly sensitive parameters from the reference value of 0.01 to 0.001. Actual test data shows that after the above processing, the inverter output power curve of the sample group of this invention did not jump significantly with the noise, but instead smoothly rose along the long-term trend indicated by the virtual momentum anchoring vector d. Finally, at 1.2s, the frequency deviation at the PCC point converged to within ±0.02Hz, and the power oscillation amplitude during the adjustment process was limited to below 2kW. The test results of control group B show that although the introduction of virtual momentum anchoring corrected the overall direction of the gradient to some extent and reduced the frequency deviation fluctuation range to ±0... The system operates at 0.08 Hz, but due to the lack of a stiffness-gated mechanism for high-sensitivity parameters, it still experiences subsynchronous oscillations with a frequency of 5 Hz and a duration exceeding 3 seconds as it approaches the power equilibrium point due to the overreaction of individual sensitive parameters. This result, through direct comparison with the sample group of this invention, provides strong evidence for the synergistic effect: that is, without step size adaptation (stiffness gating), direction correction (momentum anchoring) alone cannot achieve complete steady-state convergence of the system in a strong noise environment; only the organic combination of the two can eliminate tail oscillations while ensuring the adjustment speed. In addition, to verify the rationality of the parameter boundaries, an out-of-range control group C was set up, with its high-frequency threshold... Set as (Higher than the preferred value), the experimental results show that, due to the excessively high threshold, most high-sensitivity parameters failed to be identified and damped, causing the system's dynamic performance to converge with that of control group A, and the frequency oscillation amplitude to rise back to Hz, conversely, when the threshold is set to When the frequency recovery time is below the optimal value, the system enters an overdamped state, and the frequency recovery time is extended to more than 5.0 seconds, which fails to meet the time response requirements of the primary frequency regulation of the power grid. This set of data objectively confirms this. As an optimal operating window determined by engineering considerations, it avoids the hysteresis caused by overdamping and the oscillation caused by underdamping.

[0035] Example 3: This example provides a deep engineering disclosure of the internal construction logic of the core algorithm module and the field calibration procedures for key control parameters in the strategy optimization method. This ensures the reproducibility and adaptability of the technical solution under different microgrid topologies. Regarding the construction of the aforementioned power loss function, the system does not simply track a single index, but uses a multi-constraint penalty mechanism based on the Lagrange multiplier method to quantify the merits of the current control strategy. Within each scheduling cycle t, the processor collects the voltage amplitude at the grid connection point. Frequency deviation and line temperature at key transmission sections Based on these physical quantities, the total loss function L(θ) is constructed, and its specific formula is defined as follows: ,in, The rated voltage reference value. This refers to the thermal stability limit temperature of the circuit. It is a linear rectification function used to introduce a penalty term only when the temperature exceeds the limit; These are weighting coefficients for frequency, voltage, and thermal constraints, respectively. In the typical microgrid scenario of this embodiment, based on the priority of the primary frequency regulation response, these coefficients are normalized to 0.5:0.3:0.2. The system then uses the chain rule and an automatic differentiation engine to calculate the first-order partial derivative of the loss function with respect to the policy network parameter θ, thereby obtaining the accurate original policy gradient vector. Based on this, to eliminate the uncertainty in the parameter sensitivity vector mapping process, this scheme introduces an explicit nonlinear damping transfer function in the stiffness coefficient mapping stage. The system reads the sensitivity vector maintained by the exponentially weighted moving average algorithm at the current time. For each component in the vector (Representing the sensitivity of the i-th control parameter), the system uses the formula Calculate the corresponding stiffness attenuation coefficient. Where γ is the stiffness gain factor, which is set to 100 in this embodiment; κ is the aforementioned high-frequency threshold. This formula ensures that when the parameter sensitivity is lower than the threshold κ, the attenuation coefficient... It remains at 1 (i.e., no decay); however, when the sensitivity exceeds the threshold... As the sensitivity decreases rapidly inversely, it achieves a smooth, continuous, and powerful suppression of the update step size of the high-sensitivity parameter in mathematical logic, thus constituting a precise physical realization of non-uniform spatial stiffness damping.

[0036] Furthermore, regarding the high-frequency threshold κ for determining the sensitivity of parameters, this invention strictly prohibits the use of fixed empirical values. Instead, it uses a background noise spectrum analysis calibration procedure to determine the threshold, adapting to the differences in electromagnetic environments at different industrial sites. This calibration procedure is triggered when the system is first connected to the grid or when there is a significant change in the grid topology. The specific execution steps are as follows: The system enters a silent observation mode and, while maintaining a constant power output, continuously collects a grid frequency fluctuation data sequence for N periods, such as N=1000. The processor performs a Fast Fourier Transform (FFT) on the sequence to extract the noise power spectral density in the high-frequency band (greater than 5Hz). The integral mean of this high-frequency noise power spectral density is calculated, and three times this mean is defined as the reference noise floor. Using historical gradient data to simulate playback, calculations were performed in the presence of only background noise. The parameter sensitivity vector baseline value under disturbance is used, and the average value of the top 5% of the components with the largest amplitude in the baseline vector is locked as the effective high-frequency threshold κ under the current operating environment. Through this procedure, the system can dynamically distinguish between natural fluctuations caused by environmental white noise and oscillation trends caused by strategy instability, ensuring that the stiffness gating mechanism is only triggered when a real gradient-trend conflict risk occurs, thus avoiding malfunctions during normal adjustment.

[0037] Example 4: Based on the aforementioned core algorithm and calibration procedure, this example provides a standardized pre-deployment calibration and initialization debugging procedure to address the engineering adaptability issues when implementing this invention on physical objects with different power grid topologies, capacity levels, and equipment aging degrees. Before initiating the main logic of intelligent agent collaborative game strategy optimization control, the system executes a parameter baseline calibration procedure based on on-site electrical characteristics. Voltage and frequency background noise data for no less than 1000 power frequency cycles are continuously collected by a measurement unit deployed at the grid connection point. The processor performs a fast Fourier transform on the background noise data to extract the power spectral density of high-frequency components above 5Hz, and directly locks the high-frequency threshold κ to three times the integral mean of this power spectral density. This establishes a parameter sensitivity judgment base independent of human experience and anchored to the current electromagnetic environment noise level of the power grid. For the engineering tuning of the smoothing factor β, the control system injects a step excitation signal with an amplitude of 0.5% to 1% of the rated power into the distributed energy control node in the open-loop state, and records the step response curve of the system's active power output in real time. The processor calculates the overshoot and oscillation attenuation ratio of the response curve. When the calculated overshoot is greater than 5% or the oscillation attenuation ratio is less than 0.2, the smoothing factor β is monotonically increased in steps of 0.005 and the step test is retried until the damping characteristics of the system converge to the critical damping range of 0.707 to 0.9. This value is used as the parameter to update the inertia weight during formal operation. Since the distributed energy nodes (DERs) in the actual deployment environment have inherent heterogeneity in response characteristics, communication delay and electrical parameters, the strategy network that relies solely on offline training often cannot directly achieve the optimal control effect.

[0038] Therefore, before the system is put into closed-loop operation, it executes a zero-time adaptive initialization process consisting of three stages. The first stage is parameter sensitivity baseline scanning. In this stage, the system sequentially injects a set of pseudo-random binary sequences (PRBS) with an amplitude of 1% of the rated power as excitation signals into each control node, and continuously monitors the voltage and frequency response of the grid-connected points in the open-loop state. The processor uses the cross-correlation function to analyze the time-domain correlation between the input excitation and the output response, thereby identifying the actual transfer function gain of each node under the current physical connection. Based on this gain data, the system performs a zero-time adaptive initialization process on the aforementioned parameter sensitivity vector. The initial values ​​are preset to ensure that the initial sensitivity distribution objectively reflects the physical control weights of each node, rather than defaulting to a uniform distribution. The second stage involves online fine-tuning of the stiffness and damping coefficients. Although the aforementioned embodiments have provided calibration procedures based on noise spectrum analysis, in actual load operation, the dynamic characteristics of the load may introduce additional nonlinear disturbances. Therefore, within the first 10 minutes after the system's initial closed-loop connection, the control logic enters a self-learning calibration mode. In this mode, the system calculates the original strategy gradient vector in real time. If the cosine of the angle between the system and the virtual momentum anchoring vector d is statistically found to be less than -0.5 for 50 consecutive cycles, it indicates that the current stiffness damping is too strong, causing the system response to be sluggish. Conversely, if the cosine of the angle flips frequently between positive and negative, it indicates that the damping is insufficient. Based on this, the system constructs a simple proportional-integral (PI) adjustment loop to dynamically correct the stiffness gain factor γ until the gradient direction of the system and the momentum trend reach the preset statistical alignment standard.

[0039] Example 5: This example provides a standardized pre-deployment calibration and engineering model construction procedure to address the poor adaptability of strategy optimization models due to differences in initial conditions at different industrial sites. Regarding the initialization of the virtual momentum anchoring vector and the on-site tuning of the stiffness threshold mentioned in the previous examples, this invention defines a two-stage engineering procedure including offline data acquisition and feature extraction, and online parameter adaptive iteration. In the first stage, the system enters a passive monitoring mode, continuously collecting historical load data and frequency fluctuation data of the target power grid under normal operating conditions for at least 24 hours. The processor performs spectral analysis based on Fast Fourier Transform (FFT) on the collected time-series data, extracting low-frequency components characterizing the inherent inertial characteristics of the power grid and high-frequency noise components characterizing random disturbances. Based on the extracted low-frequency components, the system constructs a baseline predictor based on an autoregressive moving average (ARMA) model to generate the initial virtual momentum anchoring vector. To ensure that the momentum direction at system startup is consistent with the long-term steady-state trend of the power grid, the system also calculates the variance of high-frequency noise components. And substitute it into the formula In the calculation, the initial high-frequency threshold κ is obtained, where α is an empirical coefficient derived from a large amount of historical engineering data, and is preferably 3.5 in this embodiment.

[0040] In the second stage, the system switches to active detection and parameter fine-tuning mode. In this mode, the system injects a set of pseudo-random binary sequences (PRBS) with small amplitudes (not exceeding 0.5% of the rated power) into the power grid as an excitation signal, and monitors the frequency and voltage response characteristics of the power grid in real time. The processor uses a cross-correlation function to analyze the time-domain correlation between the input excitation and the output response, and identifies the actual transfer function gain under the current power grid topology. Based on this gain data, the system corrects the high-frequency threshold κ determined in the first stage. The correction formula is as follows: Where ΔG is the deviation rate between the identification gain and the preset reference gain, γ is the correction factor, and the corrected threshold is... This will serve as the stiffness gating criterion during actual runtime. In addition, the system also includes online anomaly monitoring logic to calculate the policy gradient vector in real time. If the cosine of the angle between the current parameter setting and the momentum anchoring vector d is lower than a preset alarm threshold such as -0.8 within 10 consecutive scheduling cycles, the system will determine that the current parameter setting deviates from the optimal operating point and automatically trigger a new round of parameter adaptive iteration process. Through the above-mentioned standardized calibration and fine-tuning procedures, the present invention can ensure that it can quickly converge to the optimal control state under different power grid environments, eliminating the uncertainty in engineering implementation caused by the reliance on experience in parameter setting.

[0041] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A strategy optimization method for agent cooperative game, characterized in that, Includes the following steps: By deploying electrical measurement units at the output end of distributed energy control nodes, real-time voltage, current data and power flow status at the grid connection point are acquired. Based on the supply and demand deviation of the entire network and the thermal stability constraints of the lines, a power loss function under non-stationary dynamic environment is constructed, and the original strategy gradient vector of the current power dispatch cycle is calculated accordingly. A parameter sensitivity vector consistent with the power control strategy parameter dimension is maintained synchronously. The parameter sensitivity vector, as a physical field quantity characterizing the local curvature characteristics of the control parameter space under random disturbances of grid load, is obtained by updating the exponentially weighted moving average based on the second-order moment estimation of the historical power error gradient. Calculate the dot product of the original strategy gradient vector and the virtual momentum anchor vector that represents the long-term steady-state power flow direction of the power grid. When the value of the dot product is less than 0, it is identified that the adjustment direction of the current control node is in reverse opposition to the long-term evolution trend of the power grid, and it is determined to be a gradient-trend conflict state of power regulation. In response to the gradient-trend conflict state, non-uniform stiffness damping is applied to the control parameter update space under this state based on the parameter sensitivity vector. By suppressing the update step size of high-sensitivity parameter sites, the power strategy update command that can smooth out grid disturbances is reconstructed. The power strategy update command is used to generate the corresponding underlying inverter drive signal to physically adjust the active power output setpoint and reactive power compensation amount of the distributed energy control node.

2. The strategy optimization method for agent cooperative game as described in claim 1, characterized in that, The update process of the parameter sensitivity vector follows these rules: In each power dispatch cycle, the squared values ​​of each component element of the original strategy gradient vector are extracted as the instantaneous fluctuation energy representation of the control signal in the current cycle; using a preset smoothing factor, a weighted iterative operation is performed on the parameter sensitivity vector and the instantaneous fluctuation energy representation of the previous cycle to generate the parameter sensitivity vector of the current cycle; the value of the smoothing factor is set between 0.9 and 0.999, so that the parameter sensitivity vector has low-pass filtering characteristics for high-frequency random jitter of the power grid frequency.

3. The strategy optimization method for agent cooperative game as described in claim 2, characterized in that, The steps of performing weighted iterative calculations using a preset smoothing factor are specifically based on updating the parameter sensitivity vector according to the following mathematical relationship: ,in, This is the parameter sensitivity vector for the current scheduling period. Let be the parameter sensitivity vector of the previous scheduling cycle, and β be the smoothing factor. The original policy gradient vector for the current scheduling cycle; ⊙ represents the Hadamard product operation; parameter sensitivity vector. Each component directly corresponds to the updated resistance physical quantity of a specific weight in the power control strategy network.

4. The strategy optimization method for agent cooperative game as described in claim 1, characterized in that, The steps for constructing the virtual momentum anchor vector include: establishing a historical gradient storage queue of length N in memory to store the effective gradient vectors generated by the distributed energy control nodes in the past N scheduling cycles; performing a weighted summation operation based on the time decay factor on the vectors in the historical gradient storage queue to generate a momentum trend vector characterizing the long-term evolution direction of the grid power flow; and using the momentum trend vector as a benchmark reference axis for smoothing out instantaneous electrical noise interference to verify whether the original policy gradient vector of the current cycle deviates from the convergence trajectory of the system towards steady state.

5. The strategy optimization method for agent cooperative game as described in claim 1, characterized in that, The steps for applying non-uniform stiffness damping include: mapping the values ​​of each component of the parameter sensitivity vector to a preset stiffness coefficient range based on the grid frequency fluctuation spectral density, generating stiffness coefficient vectors that correspond one-to-one with the control parameters; for components in the parameter sensitivity vector whose values ​​exceed a preset high-frequency threshold, setting their corresponding stiffness coefficients as enhanced damping values, with the enhanced damping value being positively correlated with the sensitivity amplitude of the component; and using the stiffness coefficient vector to perform element-wise division scaling on the original strategy gradient vector, thereby forcibly attenuating the update amplitude of parameter dimensions that are highly sensitive to random disturbances in grid voltage or frequency.

6. The strategy optimization method for agent cooperative game as described in claim 1, characterized in that, After determining the gradient-trend conflict state of power regulation, the process further includes: if the value of the dot product is greater than or equal to 0, determining that the current control is in a trend-following regulation state; in the trend-following regulation state, triggering the stiffness damping bypass logic, directly using the original strategy gradient vector superposition to multiply the virtual momentum anchoring vector by the preset perturbation coefficient to obtain the perturbation component as the power strategy update instruction, so as to accelerate the convergence speed of the distributed energy control node towards the target power operating point.

7. The strategy optimization method for agent cooperative game as described in claim 1, characterized in that, The steps for constructing a power loss function under non-stationary dynamics based on the supply-demand deviation of the entire network include: obtaining marginal electricity price information and line load rate information exchanged between adjacent distributed energy control nodes through communication interfaces; constructing a composite objective function that includes voltage over-limit penalty term and frequency deviation penalty term; taking the partial derivative of the composite objective function with respect to the control strategy parameters to obtain gradient information reflecting the adjustment direction of each parameter under the current grid state, which serves as the environmental feedback signal under non-stationary dynamics.

8. The strategy optimization method for agent cooperative game as described in claim 5, characterized in that, The steps for generating the corresponding underlying inverter drive signal using the power strategy update instruction include: decomposing the reconstructed power strategy update instruction into the duty cycle adjustment amount of the inverter's pulse width modulation signal; inputting the duty cycle adjustment amount to the underlying drive circuit of the distributed energy control node; and before executing the duty cycle adjustment amount, verifying whether the adjusted output power is within the physical safety constraints of the node, and if it exceeds the range, performing a truncation operation based on the boundary gradient.

9. The strategy optimization method for agent cooperative game as described in claim 1, characterized in that, The method is used to solve the collaborative control problem of large-scale photovoltaic clusters under the scenario of severe power fluctuations caused by cloud shading; non-stationary dynamic environment specifically refers to the physical process in which the output power of the photovoltaic array exhibits intermittent and random jumps due to the rapid movement of clouds; tail oscillation specifically refers to the phenomenon of continuous high-frequency power oscillation of inverter output power near the theoretical maximum value caused by the time delay of multi-machine collaborative gradient during the maximum power point tracking process.

10. A strategy optimization method for agent cooperative game as described in claim 1, characterized in that, After adjusting the power output setpoint of the distributed energy control node using the power strategy update command, the process also includes: real-time monitoring of the frequency change rate at the grid's point of common coupling; when the absolute value of the frequency change rate is less than a preset steady-state judgment threshold, gradually reducing the stiffness damping corresponding to the integral control term in the parameter sensitivity vector to restore the system's dynamic response capability to small load changes; when the absolute value of the frequency change rate exceeds the steady-state judgment threshold, identifying it as a large disturbance state in the grid, executing emergency frequency response control or switching to a preset safety protection mode to maintain the voltage and frequency stability of the local grid.

Images