A back-to-back flexible direct current control method and system based on reinforcement learning and harmonic injection

Abstract

The application discloses a back-to-back flexible direct current control method and system based on reinforcement learning and harmonic injection, and comprises the following steps: acquiring system operation state data, and dividing fault drop and recovery states according to the voltage amplitude of a grid-connected point; inputting the state data into a lightweight strategy neural network, and solving a PI parameter dynamic adjustment amount in real time and updating the parameters of a bottom controller; collecting a voltage modulation ratio and a power factor angle in real time; constructing a bridge arm power coupling equation containing third harmonic voltage and second harmonic current injection parameters, judging the existence of an equation set solution, and outputting an optimal or suboptimal harmonic intervention parameter combination; according to the output parameter combination type, corresponding harmonic control instructions are generated and branched, and are respectively superimposed to a voltage modulation link and issued to a circulating current controller, so that adaptive and collaborative suppression of sub-module voltage fluctuation is realized. The transient stability margin and operation reliability of the back-to-back flexible direct current transmission system under large disturbance are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of flexible DC transmission technology, and in particular to a back-to-back flexible DC control method and system based on reinforcement learning and harmonic injection. Background Technology

[0002] Back-to-back flexible direct current transmission systems (VSC-HVDC) are key equipment for the grid connection of new energy sources and the interconnection of asynchronous power grids. Their underlying control widely adopts a voltage and current dual closed-loop vector control framework based on a synchronous rotating coordinate system. In this framework, the parameter tuning of the proportional-integral (PI) controller usually relies on offline static calculations using a small-signal model to obtain stable control performance at typical operating points.

[0003] However, in actual power grid operation, AC receiving-end systems frequently suffer from large disturbances such as short circuits and grounding faults. During these nonlinear transient processes, the system experiences two distinctly different physical phases: "fault drop" and "fault recovery." In the fault drop phase, the system requires extremely high dynamic response bandwidth to suppress short-circuit current peaks; however, in the voltage recovery phase, high bandwidth parameters significantly reduce the system's equivalent damping ratio, easily inducing subsynchronous oscillations or high-frequency resonances. This fundamental conflict between dynamic response capability and damping stability margin makes it difficult for fixed PI parameters to maintain global optimum throughout the entire transient process.

[0004] Even more serious is the fact that, in engineering practice, to prevent the integral term from accumulating and causing control commands to exceed the linear modulation region, microcontrollers are often forcibly integrated with integral freeze logic. When the system encounters a deep fault, the fixed PI parameter can easily and quickly reach the hardware limiting boundary, forcing the controller into a saturation dead zone. During the voltage recovery period, the controller frequently jumps between the dead zone and the linear region, causing severe voltage overshoot and power backlash, threatening the safety of the converter valve.

[0005] In recent years, deep reinforcement learning (DRL) has provided a new technical path to overcome the aforementioned bottlenecks. Existing research attempts to directly map AI control to PWM duty cycles, forming a "black box" controller. However, such methods are completely outside the power electronic hardware safety protection system, lacking explicit modeling of physical constraints such as integral limiting and modulation saturation, which poses a risk of instantaneous system divergence and collapse. In addition, the microsecond-level computational step size of electromagnetic transient (EMT) simulation results in extremely large computational overhead for joint training of reinforcement learning, seriously hindering the iterative verification and engineering implementation of control strategies.

[0006] Meanwhile, modular multilevel converters (MMCs) are widely used in flexible DC transmission systems. In the AC / DC energy exchange process, MMCs employing half-bridge submodules experience continuous instantaneous power imbalance within the bridge arms. This directly leads to significant low-frequency voltage fluctuations (mainly fundamental and second harmonic components) in the submodule capacitors, exacerbating electrical stress on the devices and deteriorating the dynamic control performance and operational stability of the converter valves.

[0007] To suppress the aforementioned low-frequency voltage fluctuations, existing technologies often employ single-dimensional intervention strategies, such as simply injecting zero-sequence third harmonic voltage into the modulating wave or actively injecting second harmonic circulating current into the bridge arm. However, the voltage and current within the MMC exhibit strong coupling characteristics, and adjusting a single variable can easily induce new cross-coupled power components, thus limiting the suppression effect. Simply combining these two methods, due to a lack of in-depth analysis of the multi-dimensional harmonic cross-coupling mechanism and insufficient consideration of the physical boundary constraint that the half-bridge submodule lacks the ability to output negative levels, not only fails to solve for the globally optimal intervention parameters under complex operating conditions but also easily induces overmodulation, leading to system instability.

[0008] In summary, the existing technology has the following drawbacks:

[0009] 1. Poor adaptability of transient PI parameters: Fixed PI parameters cannot take into account the differentiated control requirements during the fault drop period and recovery period, and are prone to coupling with integral freeze logic, causing control dead zone and voltage overshoot;

[0010] 2. AI control lacks physical safety boundaries: Existing DRL control schemes directly generate placeholder control signals without integrating with the underlying PI architecture and hardware limiting mechanism, which poses a safety risk;

[0011] 3. Low training efficiency: The joint training of electromagnetic transient models and reinforcement learning has high communication overhead and low sample throughput, making it difficult to meet the requirements of engineering deployment;

[0012] 4. Limited harmonic suppression methods: Existing third-harmonic injection or second-harmonic injection methods lack a collaborative optimization mechanism and fail to achieve joint control of multidimensional harmonics from the power coupling mechanism level. The suppression effect is limited and overmodulation is easily induced. Summary of the Invention

[0013] To address the shortcomings in the prior art, this invention provides a back-to-back flexible DC transmission control method and system based on reinforcement learning and harmonic injection. The method achieves adaptive adjustment of transient PI parameters through reinforcement learning, and combines multi-dimensional harmonic partitioning decision-making and branch execution mechanisms to significantly improve the transient stability margin and operational reliability of the back-to-back flexible DC transmission system under large disturbances.

[0014] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows:

[0015] Firstly, a back-to-back flexible CRT control method based on reinforcement learning and harmonic injection includes:

[0016] S1: Obtain the current status data of the system operation; wherein, the status data includes at least the grid connection point voltage, current, error, integral freeze flag, and grid connection point voltage amplitude monitored by the phase-locked loop;

[0017] S2: Based on the comparison between the voltage amplitude at the grid connection point and the preset threshold, the transient process of the system is divided into fault drop state and fault recovery state.

[0018] S3: Input the current state data and state recognition results into the lightweight policy neural network that has been trained offline, and calculate and output the dynamic adjustment amount of the proportional-integral (PI) parameter in real time;

[0019] S4: The dynamic adjustment is superimposed on the preset static reference parameters, the proportional-integral (PI) parameters of the underlying dual-closed-loop vector controller are updated, and voltage and current closed-loop control is executed.

[0020] S5: Real-time acquisition of the voltage modulation ratio and power factor angle of the system;

[0021] S6: Construct the bridge arm instantaneous power coupling equation containing the third harmonic voltage injection parameters and the second harmonic current injection parameters, and solve the optimal or suboptimal harmonic intervention parameter set online according to the voltage modulation ratio and power factor angle.

[0022] S7: Based on the parameter combination type output by S6, control commands are generated using the corresponding harmonic intervention parameters and sent to the execution stage:

[0023] If the global optimal parameter combination is output, the third harmonic voltage injection parameter in the optimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

[0024] If the output is a suboptimal parameter combination, the third harmonic voltage injection parameter in the suboptimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

[0025] Furthermore, in S3, the output layer of the lightweight strategy neural network does not contain a nonlinear activation function, and an absolute value mapping constraint is applied to limit the dynamic adjustment of the PI parameter to a non-negative value, satisfying:

[0026] ;

[0027] ;

[0028] in, This is the updated proportional gain; This is the updated integration time constant; These are the preset static reference parameters corresponding to the proportional gain; These are the preset static reference parameters corresponding to the integration time constant; The dynamic adjustment amount of the proportional gain output of the policy neural network; The integral time constant of the policy neural network output Dynamic adjustment amount; The network weights corresponding to the proportional gain; The network weights are the integral time constants.

[0029] Furthermore, the offline training of the lightweight policy neural network described in S3 employs a proximal policy optimization algorithm, whose objective function incorporates a probability ratio truncation mechanism, specifically as follows:

[0030] ;

[0031] ;

[0032] in, The objective function is... These are the weight parameters of the current policy neural network; Expected based on experience; The probability ratio between the old and new strategies; This is the estimated value of the dominance function at time t; It is a forced boundary limiting function; To truncate hyperparameters; For new strategy networks; For old policy networks; This is the state vector.

[0033] Furthermore, the offline training of the lightweight policy neural network described in S3 uses a combined reward function to guide the convergence of the policy neural network. The combined reward function is specifically as follows:

[0034] ;

[0035] Among them, among them, For combined reward functions; For status indicators; The reward function for the fault drop period includes rate limiting penalties and points freeze / refund penalties. This is the fault recovery period reward function, which includes a transient differential damping penalty term; This is a penalty for smoothing out actions.

[0036] Furthermore, motion smoothing penalty items The low-pass filter comparison stage is constructed as follows:

[0037] ;

[0038] in, To smooth out the penalty coefficient, used to reduce the high-frequency chattering components in the policy network output; The total dimension of the action commands output by the policy neural network (i.e., the total number of underlying control parameters that are adjusted). This is the i-th action instruction; This is the historical trajectory of the low-pass filter.

[0039] Furthermore, in S6, the instantaneous power coupling equation of the bridge arm includes the third harmonic voltage injection parameter, i.e., the amplitude coefficient. Phase angle And the second harmonic current injection parameter, i.e., the amplitude coefficient Phase angle The system determines whether the equation set has a real solution based on the voltage modulation ratio and power factor angle. If it does, the system solves and outputs the globally optimal parameter combination. If it does not, the system uses the constraint of ensuring that the fundamental frequency fluctuation component is zero to solve and output the suboptimal parameter combination that minimizes the second harmonic fluctuation component.

[0040] Furthermore, the objective function satisfied by solving for the suboptimal parameter combination is specifically:

[0041] ;

[0042] in, This represents the amplitude of the second harmonic power fluctuation component; This is the voltage modulation ratio; The power factor angle; The amplitude coefficient for the second harmonic current injection; The amplitude coefficient for the third harmonic voltage injection; This is the second harmonic current amplitude mapping function under the constraint that the fundamental frequency power fluctuation component is zero; Let be the third harmonic voltage amplitude mapping function under the constraint that the fundamental frequency power fluctuation component is zero; The phase angle injected for the second harmonic current; The phase angle injected for the third harmonic voltage;

[0043] The solution for the suboptimal parameter combination also satisfies the optimization constraint that the fundamental frequency fluctuation component is zero, as well as the physical boundary constraint that the reference voltage of the half-bridge submodule is non-negative.

[0044] Furthermore, in S7, when the third harmonic voltage injection parameter is superimposed on the voltage modulation stage, a zero-sequence third harmonic voltage component is injected into the basic sinusoidal modulation wave to reduce the peak value of the bridge arm reference voltage and reshape the bridge arm power distribution; when the second harmonic current injection parameter is sent to the circulating current controller, a second harmonic circulating current with a specified amplitude and phase angle is actively injected into the bridge arm to interact with the bridge arm voltage to generate a compensation power component, thereby offsetting the inherent low-frequency power fluctuations of the system.

[0045] Secondly, an adaptive control system for a back-to-back flexible DC transmission system based on reinforcement learning and harmonic injection, the system being used to execute the method described above, including:

[0046] Data acquisition module: Acquires the current status data of the system operation; the status data includes at least the grid connection point voltage, current, error, integral freeze flag, and grid connection point voltage amplitude monitored by the phase-locked loop;

[0047] State division module: Based on the comparison between the voltage amplitude at the grid connection point and the preset threshold, the transient process of the system is divided into fault drop state and fault recovery state;

[0048] The proportional-integral (PI) parameter update module: inputs the current state data and state recognition results into the lightweight strategy neural network that has been trained offline, calculates and outputs the dynamic adjustment of the proportional-integral (PI) parameters in real time; superimposes the dynamic adjustment onto the preset static reference parameters, updates the proportional-integral (PI) parameters of the underlying dual-loop vector controller, and executes voltage and current closed-loop control.

[0049] Harmonic intervention parameter set generation module: Real-time acquisition of the voltage modulation ratio and power factor angle of the system; construction of bridge arm instantaneous power coupling equations including third harmonic voltage injection parameters and second harmonic current injection parameters; and online calculation of the optimal or suboptimal harmonic intervention parameter set based on the voltage modulation ratio and power factor angle.

[0050] Command Issuance Module: Based on the parameter combination type output by the harmonic intervention parameter group generation module, it generates control commands using the corresponding harmonic intervention parameters and issues them to the execution stage.

[0051] If the global optimal parameter combination is output, the third harmonic voltage injection parameter in the optimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

[0052] If the output is a suboptimal parameter combination, the third harmonic voltage injection parameter in the suboptimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

[0053] This invention proposes a back-to-back flexible direct current control method and system based on reinforcement learning and harmonic injection. The method has the following beneficial effects:

[0054] 1. The underlying PI control logic is mapped to a physically constrained deep reinforcement learning policy network. By introducing an integral freezing flag and dividing the transient operating state, combined with offline accelerated iteration and online deployment, the transient stability margin of the system under large disturbances is improved.

[0055] 2. By superimposing the third harmonic at the voltage modulation end to optimize the voltage constraint condition, and actively injecting the second harmonic circulating current at the current control end to construct the offset power, and relying on online optimization to synchronously solve the joint parameters, multi-dimensional harmonic adaptive collaborative control is realized, which significantly suppresses low-frequency voltage fluctuations inside the converter valve.

[0056] 3. Design a combined reward function that integrates physical limiting constraints, introduce current limiting and integral freeze concession penalties for the drop period, impose transient differential damping penalties for the recovery period, and globally nest low-pass filter action smoothing term;

[0057] 4. Build a cross-platform hybrid parallel training environment, compile the electromagnetic transient model into a dynamic link library, accelerate the iteration offline in a distributed architecture, and deploy the lightweight policy network online for dynamic adjustment after convergence. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the adaptive control method for a back-to-back flexible DC transmission system based on reinforcement learning and harmonic injection, provided in an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of the principle of the Proximal Policy Optimization (PPO) algorithm engine and mathematical constraint mechanism in an embodiment of the present invention;

[0061] Figure 3 This is a transparent equivalent mapping structure block diagram of the policy neural network and the underlying PI controller provided in an embodiment of the present invention, wherein, Figure 3 (a) is the current inner loop dq axis decoupling control diagram of the bottom-level dual closed-loop vector controller; Figure 3 (b) is a diagram of the computational structure of a standard neural network neuron containing a nonlinear activation function; Figure 3(c) is a diagram showing the transparent and equivalent mapping between the policy neural network and the underlying PI controller after the activation function is removed.

[0062] Figure 4 This is the execution logic diagram of the combined reward function that integrates transient state switching and multidimensional constraints provided in this embodiment of the invention;

[0063] Figure 5 This is a schematic diagram of the main circuit topology of the modular multilevel converter (MMC) for a back-to-back flexible DC transmission system provided in an embodiment of the present invention.

[0064] Figure 6 This is a block diagram of capacitor fluctuation suppression control for a back-to-back flexible DC transmission system submodule provided in an embodiment of the present invention;

[0065] Figure 7 This is a flowchart of the optimal solution for suppressing capacitance fluctuations in a back-to-back flexible DC transmission system provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0067] like Figure 5 The diagram shows the main circuit topology of the modular multilevel converter (MMC) for a back-to-back flexible DC transmission system provided in this embodiment of the invention. The AC side of the modular multilevel converter is connected to a three-phase AC power grid (shown as an equivalent AC voltage source in the diagram). , , and the equivalent inductance of the grid side (Indicated), the DC side is connected to the DC bus of the back-to-back system. The main body of the converter consists of three phase units: A, B, and C. Each phase unit includes an upper bridge arm and a lower bridge arm.

[0068] Taking phase A as an example, its upper bridge arm consists of the bridge arm equivalent resistance R11, the bridge arm reactor L11, and N half-bridge sub-modules (marked as SM1 to SMN in the figure) connected in series. The upper bridge arm current is denoted as... The overall output voltage of the upper bridge arm is denoted as Its lower bridge arm consists of the bridge arm equivalent resistance R21, the bridge arm reactor L21, and N half-bridge sub-modules connected in series. The lower bridge arm current is denoted as... The overall output voltage of the lower bridge arm is denoted as Among them, the bridge arm reactor is mainly used to suppress the circulating current generated inside the converter valve and limit the rate of rise of the system fault current.

[0069] like Figure 5 As shown in the enlarged dashed box, the half-bridge sub-modules in the embodiments all adopt a standard half-bridge topology, including two insulated-gate bipolar transistors with anti-parallel diodes (denoted as VT11 and VT12 in the figure, and the corresponding anti-parallel diodes are denoted as VD11 and VD12 respectively) and a parallel energy storage DC capacitor C.

[0070] The method described in this invention ultimately controls the DC capacitor C inside the half-bridge submodule. During normal AC / DC energy exchange in the MMC, the instantaneous power imbalance in the bridge arms will inevitably induce significant low-frequency (mainly fundamental and harmonic frequencies) voltage fluctuations on capacitor C. This embodiment, based on the main circuit hardware topology, actively constructs counteracting power through coordinated control of the modulation and circulating current terminals, thereby suppressing the low-frequency voltage fluctuations of capacitor C. The embodiment is described in detail below.

[0071] Example 1

[0072] like Figure 1 As shown, this embodiment provides a back-to-back flexible direct current control method based on reinforcement learning and harmonic injection, including:

[0073] S1: Obtain the current system status data. The specific process is as follows:

[0074] S1.1: The instantaneous values ​​of the three-phase voltage at the grid connection point are acquired through a high-frequency status acquisition module with a microsecond-level synchronous cycle. , , and instantaneous values ​​of three-phase current , , .

[0075] S1.2: Calculate the voltage error. With current error This serves as the input deviation signal for the underlying PI controller. Specifically, the voltage error is the difference between the reference voltage and the actual voltage, and the current error is the difference between the reference current and the actual current.

[0076] S13: Read the integral freeze flag in the underlying controller The integral freeze flag is set when the accumulated integral term reaches the hardware limit boundary, indicating whether the controller has entered a saturation dead zone state.

[0077] S14: Real-time monitoring of grid connection point voltage amplitude via phase-locked loop (PLL) And output it to the status recognition module.

[0078] S15: Integrate the data collected above into a multi-dimensional state vector. , as input to the reinforcement learning agent.

[0079] S2: Based on the comparison between the grid connection point voltage amplitude and the preset threshold, the system transient process is divided into fault drop state and fault recovery state. The specific process is as follows:

[0080] S2.1: The preset voltage threshold is 0.9 per unit, which serves as the switching boundary between fault drop and recovery;

[0081] S2.2: Real-time reading of the grid connection point voltage amplitude output by the phase-locked loop. ;

[0082] S2.3: If the voltage amplitude falls below 0.9 per unit for the first time, i.e. If the state machine is locked in a fault fall state, the control strategy will be biased towards current limiting to prevent overload and reactive power excitation support.

[0083] S2.4: If the voltage amplitude rises from below 0.9 per unit to above the preset voltage threshold, i.e. Furthermore, the preceding state is a fault fall, the state machine switches to the fault recovery state without disturbance, and the control strategy shifts to suppressing subsynchronous oscillations and transient power echoes.

[0084] S2.5: Output status indicator Among them, when This indicates a fault drop condition, when , indicating the fault recovery state, is used for subsequent reward function calculation and conditional branches of the policy network.

[0085] S3: Input the current state data and state recognition results into the lightweight policy neural network trained offline, and calculate and output the dynamic adjustment amount of the proportional-integral (PI) parameter in real time. The specific process is as follows:

[0086] S3.1: The state vector output by S1 Status flags output by S2 Concatenate the data to obtain the input to the policy neural network;

[0087] S3.2: Use a lightweight strategy neural network for prediction and output the original dynamic adjustment amount of the PI parameter;

[0088] Specifically, this embodiment uses a proximal policy optimization (PPO) algorithm engine and mathematical constraint mechanisms. For example... Figure 2As shown, the agent acquires the system's operating status in real time during its interaction with the underlying controller and stores the collected state and reward sequences in a trajectory buffer. The evaluation network outputs a value prediction based on the current state. This is used to calculate the advantage function and evaluate the merits of the current strategy. Simultaneously, the action network generates action instructions based on the current state and directly sends them to the underlying controller for dynamic adjustment.

[0089] Specifically, the lightweight policy neural network constructed in this embodiment relies on a multilayer perceptron (MLP) topology framework and includes an input layer, two hidden layers, and an output layer. To balance the system's nonlinear approximation capability for high-dimensional state parameters with the extremely fast computation timeliness of microsecond-level control cycles, the neuron size of both hidden layers is reduced and tuned to 64, and the Tanh hyperbolic tangent function is introduced as a nonlinear activation operator. Simultaneously, to address the physical continuity requirement of control parameter adjustment, the network output layer strictly eliminates nonlinear activation functions, aiming to reconstruct and maintain an absolute linear mapping relationship with the underlying dual-loop PI controller.

[0090] In the offline co-simulation training phase of the policy model, the Adam optimization algorithm is used to drive the gradient update of the global network weights. The core hyperparameter group is specifically tuned as follows: the initial learning rate is set to... It also incorporates a convergence mechanism that decays linearly with the number of iterations; the global batch size for empirical trajectory sampling is configured to 2048; the gradient update mini-batch is set to 64; and a forward reward discount factor is used to measure long-term control returns. Set to 0.99; Generalized Dominance Estimation (GAE) smoothing coefficient Set to 0.95; to prevent drastic changes in control parameters during a single iteration, the policy probability ratio is truncated to the boundary hyperparameter. The calibration is set to 0.2 to avoid the risk of transient instability from the mathematical level. For online closed-loop control deployment in practical engineering, the strategy neural network completely eliminates the computational overhead of backpropagation and directly solidifies the optimal static weight matrix after offline convergence. The control unit (such as DSP or FPGA) only needs to perform a lightweight forward tensor multiplication and addition derivation with a time complexity of $O(1)$ within a single sampling interruption cycle to achieve deterministic and low-latency distribution of the dynamic compensation of PI parameters, which is highly adaptable to the high real-time performance and wide bandwidth requirements of back-to-back flexible DC transmission systems for large disturbance suppression.

[0091] To suit the extremely high stability requirements of power electronic control systems, an objective function with an embedded probability ratio truncation mechanism is set. In each discrete iteration of the interaction between the agent and the power grid environment, this embodiment defines a probability ratio. To measure the new policy network Compared to the old policy network Among them, the old policy network A fixed reference network for generating trajectory cache data for agents during the environmental interaction phase; a new policy network. This refers to the network that is currently optimizing its parameters during the gradient iterative update step of the current policy.

[0092] In state Take action below Degree of deviation:

[0093] .

[0094] To ensure that each step of the policy update approaches the safety baseline, the objective function is constructed as follows:

[0095] ;

[0096] in, The objective function is... These are the weight parameters of the current policy neural network. Expected based on experience; The probability ratio between the old and new strategies; This is the advantage function, used to evaluate the additional benefit of the current parameter adjustment action compared to the average level; It is a forced boundary limiting function; To truncate the hyperparameter (usually set to 0.2). When the new policy leads to a probability ratio... Exceeding When the confidence interval is reached, the penalty mechanism is implemented through... The function forcibly removes stimuli that attempt to cause drastic parameter jumps. The system uses the `clip` function to perform forced truncation and takes the minimum value between the untrunculated surrogate term and the trunculated surrogate term, thereby constructing a conservative near-end update mechanism based on the truncation surrogate objective. This suppresses abrupt changes in control parameters caused by excessively large single-iteration amplitudes of the strategy. This mechanism reduces the risk of excessive updates to the strategy network driven by abnormal transient samples and, in conjunction with action smoothing penalties, PI parameter physical boundary constraints, and underlying amplitude limiting protection, improves the converter's operational stability under grid distortion conditions.

[0097] S3.3: Remove the nonlinear activation function from the network output layer, maintain the linear characteristics of the PI controller, and ensure that the neural network output and the underlying PI controller achieve a transparent and equivalent mapping; such as Figure 3 The diagram shows the transparent equivalent mapping structure between the policy neural network and the underlying PI controller in this embodiment. To perfectly reconstruct the linear characteristics of the PI controller, the nonlinear activation function at the output end of the policy network is removed. Shaft voltage control command For example, the tensor multiplication and addition operation of a neural network can be equivalently transformed into:

[0098] ;

[0099] ;

[0100] in, The proportional gain of the underlying controller; This represents the deviation of the d-axis current. The integral gain of the underlying controller; In the policy neural network, the weight constants corresponding to the voltage feedforward branch are fixed. This represents the feedforward component of the grid voltage on the d-axis. This represents the feedforward component of the grid voltage on the q-axis. The fundamental angular frequency of the power grid; It is the equivalent inductance on the grid side (or the equivalent inductance of the bridge arm reactor). This is the actual measured current value along the q-axis; This represents the actual measured current value along the d-axis.

[0101] S3.4: Apply absolute value mapping constraints to the output adjustment amount, and dynamically adjust the PI parameter (i.e., and The constraint is non-negative and satisfies the following relationship:

[0102]

[0103]

[0104] in, This is the dynamic adjustment amount of the proportional gain of the final output of the policy neural network. This is the dynamic adjustment amount of the integral gain of the final output of the policy neural network; The dynamic adjustment of the original proportional gain of the policy neural network before absolute value mapping constraints; The dynamic adjustment amount of the original integral gain of the policy neural network before it has undergone absolute value mapping constraints;

[0105] S3.5: Output the final dynamic adjustment value of the PI parameter. and .

[0106] The execution logic diagram of the combined reward function mechanism integrating transient state switching and multidimensional constraints in this embodiment is as follows: Figure 4 As shown, the combined reward function integrates phased penalties and continuous trajectory constraints. The combined reward function, which drives the convergence of the guided policy neural network, is as follows:

[0107]

[0108] in, For global combined reward function; The voltage amplitude at the AC grid connection point; The reward function for the fault fallout period includes steady-state tracking, physical current limiting penalty, and points freeze and yield penalty. This is the fault recovery period reward function, which includes a transient differential damping penalty term; This is a penalty for smoothing out actions.

[0109] When the grid connection point voltage drops below the safety threshold ( Entering the failure drop phase, It not only includes steady-state tracking error penalty and physical current limiting penalty, but also introduces integral freeze yield penalty, the specific mathematical expression of which is:

[0110]

[0111] in, , , This corresponds to the penalty weight coefficient; and These are the reference current and the actual current of the control system, respectively. The overcurrent protection limit for the converter valve hardware; This is the state of the integral freeze flag (1 when saturation is triggered, 0 in the linear region). This mechanism drives the agent to use a yielding strategy that reduces the integral gain in the early stages of error divergence to avoid the hardware nonlinear limiting boundary.

[0112] After the external fault is cleared and the fault recovery period begins ( The system is highly susceptible to secondary power oscillations. A transient damping constraint term is introduced, encompassing the small time-varying rates of AC / DC voltage and power. The specific formula is as follows:

[0113]

[0114] in, , All are transient differential damping penalty coefficients; This is the DC bus voltage; This represents the active power at the grid connection point. This item represents the forced injection of broadband virtual damping into the system to suppress high-frequency active power oscillation echoes.

[0115] Smooth motion penalty It is constructed through an embedded low-pass filter (LPF) comparison feedback loop, as follows:

[0116] in, To smooth out the penalty coefficient, used to reduce the high-frequency chattering components in the policy network output; The total dimension of the action instructions output by the policy network (i.e., the number of PI parameters being tuned). For the first One action command; For the historical trajectory of a first-order discrete low-pass filter, its iterative update logic satisfies:

[0117]

[0118] in, Let be the filter coefficients, satisfying , For the sampling period of the underlying microcontroller digital control (e.g.) ), This is the filter time constant (e.g., 5ms).

[0119] S4: The dynamic adjustment is superimposed on the preset static reference parameters, the proportional-integral (PI) parameters of the underlying dual-loop vector controller are updated, and voltage and current closed-loop control is executed. The specific process is as follows:

[0120] S4.1: Obtain the preset static reference parameters and ;

[0121] S4.2: Superimpose the dynamic adjustment of the PI parameter output from S3 with the corresponding static reference parameter to calculate the updated PI parameter, satisfying:

[0122]

[0123]

[0124] in, This is the updated proportional gain; This is the updated points gain; These are the preset static reference parameters corresponding to the proportional gain; These are the preset static baseline parameters corresponding to the integral gain. Since the absolute value mapping function has been forcibly applied in the preceding steps, completely eliminating the risk of negative gain output by the AI ​​algorithm, the superposition process here ensures that the fine-tuning iteration always runs within the absolutely safe physical positive quadrant.

[0125] S4.3: Write the updated PI parameters into the registers of the underlying dual-loop vector controller;

[0126] S4.4: The controller performs closed-loop regulation of the voltage outer loop and current inner loop based on the new PI parameters;

[0127] S4.5: Outputs a PWM modulated wave to the power device of the converter valve to complete real-time control of voltage and current.

[0128] S5: Real-time acquisition of the voltage modulation ratio and power factor angle of the system. The block diagram of the capacitor fluctuation suppression control for the back-to-back flexible DC transmission system submodule is shown below. Figure 6 As shown. The specific process is as follows:

[0129] S5.1: Obtain the current modulation amplitude value during the operation of the control system. With DC bus voltage Calculate the voltage modulation ratio :

[0130] ;

[0131] S5.2: Based on the current active power of the control system Power and reactive power Calculate the power factor angle :

[0132] ;

[0133] S5.3: The interval is based on the control cycle (generally 100-200). Synchronous update of voltage modulation ratio and power factor angle .

[0134] S6: Construct the bridge arm instantaneous power coupling equation including the third harmonic voltage injection parameters and the second harmonic current injection parameters, and calculate the optimal or suboptimal harmonic intervention parameter set online based on the voltage modulation ratio and power factor angle. The flowchart for finding the optimal solution for capacitor fluctuation suppression in a back-to-back flexible DC transmission system is shown below. Figure 7 As shown. The specific process is as follows:

[0135] S6.1: Construct a system that includes third harmonic voltage injection parameters (i.e., amplitude coefficients). Phase angle ) and second harmonic current (i.e., amplitude coefficient) Phase angle Instantaneous power coupling equations for the injected parameters of the bridge arms:

[0136]

[0137] in, For the instantaneous power of the bridge arm, and The voltage and current of the rear bridge arm are injected as a mixture of second-harmonic circulating current and third-harmonic voltage. and For DC bus voltage and current, This is the voltage modulation ratio. The power factor angle, The fundamental angular frequency, and The coefficients and phase angles for the third harmonic voltage injection. and The coefficients and phase angles injected for the second harmonic current.

[0138] S6.2: Based on the voltage modulation ratio and power factor angle The fundamental frequency fluctuation component equation and the second harmonic frequency fluctuation component equation of the bridge arm power are obtained:

[0139] The equation for the fundamental frequency fluctuation component of the bridge arm power is:

[0140]

[0141] in, This represents the fundamental frequency fluctuation component of the instantaneous power of the bridge arm. and For DC bus voltage and current, This is the voltage modulation ratio. The power factor angle, The fundamental angular frequency, and The coefficients and phase angles for the third harmonic voltage injection. and The coefficients and phase angles injected for the second harmonic current.

[0142] The equation for the second harmonic wave component is:

[0143]

[0144] in, This represents the second harmonic fluctuation component of the instantaneous power of the bridge arm. and For DC bus voltage and current, This is the voltage modulation ratio. The power factor angle, The fundamental angular frequency, and The coefficients and phase angles for the third harmonic voltage injection. and The coefficients and phase angles injected for the second harmonic current.

[0145] Specifically, to suppress the fundamental frequency fluctuation component of the bridge arm power to zero, the following conditions must be met:

[0146] ;

[0147] in, and The coefficients and phase angles injected for the second harmonic current. and The coefficients and phase angles for the third harmonic voltage injection. This is the voltage modulation ratio. The power factor angle, ω is the fundamental angular frequency.

[0148] To suppress the second harmonic fluctuation component of the bridge arm power to zero, the following condition must be met:

[0149]

[0150] in, and The coefficients and phase angles injected for the second harmonic current. and The coefficients and phase angles for the third harmonic voltage injection. This is the voltage modulation ratio. The power factor angle, ω is the fundamental angular frequency.

[0151] S6.3: Assess whether the current operating condition is within the preset ideal complete cancellation region: A joint equation system with both the fundamental frequency fluctuation component and the second harmonic fluctuation component simultaneously set to 0, based on the current... Does a real solution exist under the operating conditions, simultaneously satisfying the physical boundary constraint that the reference voltage of the half-bridge submodule is non-negative? Specifically, the joint equations that set both the fundamental frequency fluctuation component and the second harmonic fluctuation component to zero are:

[0152] ;

[0153] S6.4: If a real solution exists, solve the system of equations directly and output the globally optimal parameter combination;

[0154] S6.5: If no real solution exists, then suppressing the fundamental frequency fluctuation component of the bridge arm power to 0 will be used as an optimization constraint. The suboptimal parameter set that minimizes the second harmonic fluctuation component will be solved, where the objective function for optimization is:

[0155] ;

[0156] in, This represents the amplitude of the second harmonic power fluctuation component;

[0157] S6.6: Output parameter combination type identifier (i.e., optimal / suboptimal) and corresponding harmonic interference parameters (i.e., ... , , , ).

[0158] S7: Based on the parameter combination type output by S6, control commands are generated using the corresponding harmonic intervention parameters and sent to the execution stage:

[0159] If the global optimal parameter combination is output, the third harmonic voltage injection parameter in the optimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

[0160] If the output is a suboptimal parameter combination, the third harmonic voltage injection parameter in the suboptimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

[0161] S7.1: Obtain the parameter combination type identifier (optimal or suboptimal) output from S6, i.e., the corresponding harmonic interference parameters (i.e. , , , );

[0162] S7.2: If it is the optimal parameter combination, execute S7.3-S7.5; if it is the suboptimal parameter combination, execute S7.6-S7.8.

[0163] S7.3 (Optimal): Inject the third harmonic voltage from the optimal parameter combination into the parameters. , Convert to zero-sequence third harmonic voltage command ;

[0164] S7.4 (Optimal): Superimpose the third harmonic voltage command onto the base sinusoidal modulation wave of the voltage modulation stage. Generate the corrected bridge arm reference voltage :

[0165] ;

[0166] S7.5 (Optimal): Inject the second harmonic current from the optimal parameter combination into the parameters. , Convert to zero-sequence second harmonic current command The signal is sent to the circulation controller, replacing the original circulation suppression target, and proceeds to S7.9;

[0167] S7.6 (Suboptimal): Inject the third harmonic voltage from the suboptimal parameter combination into the parameters. , Convert to zero-sequence third harmonic voltage command ;

[0168] S7.7 (Second-optimal): The third harmonic voltage command is superimposed onto the fundamental sinusoidal modulation wave of the voltage modulation stage. Generate the corrected bridge arm reference voltage :

[0169] S7.8 (Suboptimal): Inject the second harmonic current from the suboptimal parameter combination into the parameters. , Convert to zero-sequence second harmonic current command The command is sent to the circulating controller and enters S7.9;

[0170] S7.9: The voltage modulation loop and circulating current controller respond to the command in the next control cycle (this control cycle can be set to a fixed duration within the range of 50μs to 200μs according to the actual hardware performance) to complete the multi-dimensional harmonic collaborative injection. Through the above method, the third harmonic voltage injection reduces the peak value of the bridge arm reference voltage and reshapes the bridge arm power distribution. The second harmonic circulating current interacts with the bridge arm voltage to generate a compensating power component, offsetting the inherent low-frequency power fluctuations of the system and realizing adaptive collaborative suppression of submodule voltage fluctuations.

[0171] Preferably, it also includes S8: non-volatilely backing up the transient waveform slice data of the entire process from power grid fault to smoothing out and the parameter tuning trajectory of the reinforcement learning agent to the panoramic operation storage module, providing offline traceability support for the second-dimensional upgrade iteration of the model.

[0172] Example 2

[0173] This embodiment provides an adaptive control system for a back-to-back flexible DC transmission system based on reinforcement learning and harmonic injection. The system is used to execute the method described above, including:

[0174] Data acquisition module: Acquires the current status data of the system operation; the status data includes at least the grid connection point voltage, current, error, integral freeze flag, and grid connection point voltage amplitude monitored by the phase-locked loop;

[0175] State division module: Based on the comparison between the voltage amplitude at the grid connection point and the preset threshold, the transient process of the system is divided into fault drop state and fault recovery state;

[0176] The proportional-integral (PI) parameter update module: inputs the current state data and state recognition results into the lightweight strategy neural network that has been trained offline, calculates and outputs the dynamic adjustment of the proportional-integral (PI) parameters in real time; superimposes the dynamic adjustment onto the preset static reference parameters, updates the proportional-integral (PI) parameters of the underlying dual-loop vector controller, and executes voltage and current closed-loop control.

[0177] Harmonic intervention parameter set generation module: Real-time acquisition of the voltage modulation ratio and power factor angle of the system; construction of bridge arm instantaneous power coupling equations including third harmonic voltage injection parameters and second harmonic current injection parameters; and online calculation of the optimal or suboptimal harmonic intervention parameter set based on the voltage modulation ratio and power factor angle.

[0178] Command Issuance Module: Based on the parameter combination type output by the harmonic intervention parameter group generation module, it generates control commands using the corresponding harmonic intervention parameters and issues them to the execution stage.

[0179] If the global optimal parameter combination is output, the third harmonic voltage injection parameter in the optimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

[0180] If the output is a suboptimal parameter combination, the third harmonic voltage injection parameter in the suboptimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

[0181] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0182] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A back-to-back flexible direct current control method based on reinforcement learning and harmonic injection, characterized in that, include: S1: Obtain the current status data of the system operation; wherein, the status data includes at least the grid connection point voltage, current, error, integral freeze flag, and grid connection point voltage amplitude monitored by the phase-locked loop; S2: Based on the comparison between the voltage amplitude at the grid connection point and the preset threshold, the transient process of the system is divided into fault drop state and fault recovery state. S3: Input the current state data and state recognition results into the lightweight policy neural network that has been trained offline, and calculate and output the dynamic adjustment amount of the proportional-integral (PI) parameter in real time; S4: The dynamic adjustment is superimposed on the preset static reference parameters, the proportional-integral (PI) parameters of the underlying dual-closed-loop vector controller are updated, and voltage and current closed-loop control is executed. S5: Real-time acquisition of the voltage modulation ratio and power factor angle of the system; S6: Construct the bridge arm instantaneous power coupling equation containing the third harmonic voltage injection parameters and the second harmonic current injection parameters, and solve the optimal or suboptimal harmonic intervention parameter set online according to the voltage modulation ratio and power factor angle. S7: Based on the parameter combination type output by S6, control commands are generated using the corresponding harmonic intervention parameters and sent to the execution stage: If the global optimal parameter combination is output, the third harmonic voltage injection parameter in the optimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller. If the output is a suboptimal parameter combination, the third harmonic voltage injection parameter in the suboptimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

2. The method according to claim 1, characterized in that, In S3, the output layer of the lightweight strategy neural network does not contain a nonlinear activation function, and an absolute value mapping constraint is applied to limit the dynamic adjustment of the PI parameter to a non-negative value, satisfying: ； ； in, This is the updated proportional gain; This is the updated integral time constant; These are the preset static reference parameters corresponding to the proportional gain; These are the preset static reference parameters corresponding to the integration time constant; The dynamic adjustment amount of the proportional gain output of the policy neural network; The integral time constant of the policy neural network output Dynamic adjustment amount; The network weights corresponding to the proportional gain; The network weights are the integral time constants.

3. The method according to claim 1, characterized in that, The lightweight policy neural network described in S3 employs a proximal policy optimization algorithm for offline training, with its objective function embedding a probability ratio truncation mechanism, specifically: ； ； in, The objective function is... These are the weight parameters of the current policy neural network; Expected based on experience; The probability ratio between the old and new strategies; This is the estimated value of the dominance function at time t; It is a forced boundary limiting function; To truncate hyperparameters; For new strategy networks; For old policy networks; This is the state vector.

4. The method according to claim 1, characterized in that, The offline training of the lightweight policy neural network described in S3 uses a combined reward function to guide the convergence of the policy neural network. The combined reward function is as follows: ； Among them, among them, For combined reward functions; For status indicators; The reward function for the fault drop period includes rate limiting penalties and points freeze / refund penalties. This is the fault recovery period reward function, which includes a transient differential damping penalty term; This is a penalty for smoothing out actions.

5. The method according to claim 4, characterized in that, Smooth motion penalty The low-pass filter comparison stage is constructed as follows: ； in, To smooth out the penalty coefficient, used to reduce the high-frequency chattering components in the policy network output; The total dimension of the action commands output by the policy neural network; This is the i-th action instruction; This is the historical trajectory of the low-pass filter.

6. The method according to claim 1, characterized in that, In S6, the instantaneous power coupling equation of the bridge arm includes the third harmonic voltage injection parameter, i.e., the amplitude coefficient. Phase angle And the second harmonic current injection parameter, i.e., the amplitude coefficient Phase angle The system determines whether the equation set has a real solution based on the voltage modulation ratio and power factor angle. If it does, the system solves and outputs the globally optimal parameter combination. If it does not, the system uses the constraint of ensuring that the fundamental frequency fluctuation component is zero to solve and output the suboptimal parameter combination that minimizes the second harmonic fluctuation component.

7. The method according to claim 6, characterized in that, The objective function satisfied by solving for the suboptimal parameter combination is as follows: ； in, This represents the amplitude of the second harmonic power fluctuation component; This is the voltage modulation ratio; The power factor angle; The amplitude coefficient for the second harmonic current injection; The amplitude coefficient for the third harmonic voltage injection; This is the second harmonic current amplitude mapping function under the constraint that the fundamental frequency power fluctuation component is zero; Let be the third harmonic voltage amplitude mapping function under the constraint that the fundamental frequency power fluctuation component is zero; The phase angle injected for the second harmonic current; The phase angle injected for the third harmonic voltage; The solution for the suboptimal parameter combination also satisfies the optimization constraint that the fundamental frequency fluctuation component is zero, as well as the physical boundary constraint that the reference voltage of the half-bridge submodule is non-negative.

8. The method according to claim 1, characterized in that, In S7, when the third harmonic voltage injection parameter is superimposed on the voltage modulation stage, a zero-sequence third harmonic voltage component is injected into the basic sinusoidal modulation wave to reduce the peak value of the bridge arm reference voltage and reshape the bridge arm power distribution; when the second harmonic current injection parameter is sent to the circulating current controller, a second harmonic circulating current with a specified amplitude and phase angle is actively injected into the bridge arm to interact with the bridge arm voltage to generate a compensation power component and offset the inherent low-frequency power fluctuations of the system.

9. An adaptive control system for a back-to-back flexible DC transmission system based on reinforcement learning and harmonic injection, the system being used to execute the method as described in any one of claims 1-8, characterized in that, include: Data acquisition module: Acquires the current status data of the system operation; the status data includes at least the grid connection point voltage, current, error, integral freeze flag, and grid connection point voltage amplitude monitored by the phase-locked loop; State division module: Based on the comparison between the voltage amplitude at the grid connection point and the preset threshold, the transient process of the system is divided into fault drop state and fault recovery state; The proportional-integral (PI) parameter update module: inputs the current state data and state recognition results into the lightweight strategy neural network that has been trained offline, calculates and outputs the dynamic adjustment of the proportional-integral (PI) parameters in real time; superimposes the dynamic adjustment onto the preset static reference parameters, updates the proportional-integral (PI) parameters of the underlying dual-loop vector controller, and executes voltage and current closed-loop control. Harmonic intervention parameter set generation module: Real-time acquisition of the voltage modulation ratio and power factor angle of the system; construction of bridge arm instantaneous power coupling equations including third harmonic voltage injection parameters and second harmonic current injection parameters; and online calculation of the optimal or suboptimal harmonic intervention parameter set based on the voltage modulation ratio and power factor angle. Command Issuance Module: Based on the parameter combination type output by the harmonic intervention parameter group generation module, it generates control commands using the corresponding harmonic intervention parameters and issues them to the execution stage. If the global optimal parameter combination is output, the third harmonic voltage injection parameter in the optimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller. If the output is a suboptimal parameter combination, the third harmonic voltage injection parameter in the suboptimal parameter combination is superimposed on the voltage modulation stage, and the second harmonic current injection parameter is sent to the circulating current controller.

Images