Self-adaptive power grid inertia and frequency modulation control method and equipment for aluminum electrolysis cell

Abstract

The invention relates to the technical field of power grid frequency modulation, in particular to a self-adaptive power grid inertia and frequency modulation control method and device for an aluminum electrolysis cell, and the method comprises the steps: collecting the operation data and power grid frequency signals of the aluminum electrolysis cell in real time, and forming a historical operation database; respectively constructing a state space and an action space of reinforcement learning based on the operation data and the power grid frequency signal; constructing a composite reward function based on the power grid frequency deviation, the frequency change rate, the electrolytic cell operation stability index and the energy consumption; based on a state space and an action space, an intelligent agent is constructed by using a depth deterministic strategy gradient framework, and in the process of constructing the intelligent agent, adaptive noise based on a constraint violation risk is introduced. When the aluminum electrolysis cell participates in power grid frequency modulation, on the premise that operation safety and process stability of the aluminum electrolysis cell are guaranteed, rapid, self-adaptive and high-performance supporting on the power grid frequency is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power grid frequency modulation, in particular to a kind of aluminum electrolytic cell adaptive power grid inertia and frequency modulation control method and equipment. BACKGROUND

[0002] With the large-scale access of wind power, photovoltaic and other volatile renewable energy, the inertia level of power system decreases, frequency fluctuation intensifies, and the demand for fast frequency modulation capacity is increasingly urgent. Traditional generation side frequency modulation resources are limited by response speed and capacity, and it is difficult to meet the requirements. Digging the potential of flexible load on the load side has become an important direction. As a high-energy industrial process, the total power of aluminum electrolysis process is huge and has certain thermal inertia. In theory, it can provide short-time power support by adjusting current or pole distance, and it is a potential adjustable load.

[0003] However, aluminum electrolytic cell participating in power grid frequency modulation faces severe challenges: its operation is highly dependent on stable electrothermal balance, and current, voltage, pole distance and other parameters are strictly constrained by process. Severe or improper power regulation can easily cause production accidents, increase energy consumption and shorten the service life of the cell. This leads to a fundamental contradiction between the "fast frequency modulation demand of the power grid" and the "safe and stable operation of the electrolytic cell". In addition, the system has strong nonlinear, multivariable coupled dynamic characteristics, and needs to coordinate multiple objectives such as frequency support effect, operation stability and energy economy. Traditional fixed control strategy is difficult to achieve adaptive optimization under complex conditions. SUMMARY

[0004] Therefore, the purpose of the present application is to provide an aluminum electrolytic cell adaptive power grid inertia and frequency modulation control method and equipment to solve the problem that in the prior art, when aluminum electrolytic cell participates in power grid frequency modulation, it cannot realize fast, adaptive and high-performance support for power grid frequency under the premise of ensuring its operation safety and process stability.

[0005] According to the first aspect of the present application, an aluminum electrolytic cell adaptive power grid inertia and frequency modulation control method is provided, comprising: real-time collection and storage of operation data of aluminum electrolytic cell and power grid frequency signal to form a historical operation database; constructing a state space and an action space of reinforcement learning according to the historical operation database; constructing a composite reward function based on power grid frequency deviation, frequency change rate, electrolytic cell operation stability index and energy consumption; constructing an agent using a deep deterministic policy gradient framework according to the state space and the action space; in the process of constructing the agent, an adaptive noise based on constraint violation risk is introduced, and the action space is made to meet the physical constraints of aluminum electrolytic cell and the power grid frequency modulation protocol through a component-level safety projection function; An electrolytic cell and power grid interaction simulation environment is constructed, and a state space, an action space and a composite reward function are jointly input into a deep deterministic policy gradient framework; The agent is trained offline using a historical operation database, so that the agent obtains an optimal power regulation strategy, and the optimal power regulation strategy is converted into a control instruction.

[0006] Preferably, the state space and the action space of the reinforcement learning are constructed according to the historical operation database, including: According to the electrothermal coupling operation mechanism of the aluminum electrolytic cell, electrolytic cell operation characteristics are extracted from the operation data, including current, cell voltage and pole distance; According to the power system frequency dynamic equation, power grid frequency characteristics are extracted from the power grid frequency signal, including frequency deviation, frequency change rate and inertia response characteristics; According to the historical operation database, historical state characteristics are obtained, including the power regulation amount at the last time and the electrolytic cell operation stability index; The electrolytic cell operation characteristics, the power grid frequency characteristics and the historical state characteristics are jointly defined as the state space of the reinforcement learning agent; According to the adjustment response characteristics of the electrolytic cell to the power grid frequency disturbance, the action variables of the reinforcement learning agent are constructed, including: power regulation action, pole distance adjustment action and current adjustment action; The action space of the reinforcement learning agent is generated according to the action variables.

[0007] Preferably, a composite reward function based on the power grid frequency deviation, the frequency change rate, the electrolytic cell operation stability index and the energy consumption is constructed, including: According to the real-time collected frequency deviation and frequency change rate of the power grid, a frequency emergency index is constructed; According to the real-time frequency deviation and frequency change rate, the current operation mode of the power grid and the frequency change directionality index are obtained, and dynamic weights are assigned to the frequency quality, operation stability and energy consumption economy of the current operation mode; For the frequency quality, operation stability and energy consumption economy, respectively, a reward function with time decay characteristics is constructed; A reward shaping term is constructed according to the frequency change directionality index; Each reward function is weighted and fused according to the dynamic weights, and then weighted and fused with the reward shaping term to obtain the composite reward function.

[0008] Preferably, for the frequency quality, operation stability and energy consumption economy, respectively, a reward function with time decay characteristics is constructed, including: A time decay factor is constructed based on the action or event duration and the decay coefficient; Based on the real-time collected frequency deviation and frequency change rate of the power grid, a frequency quality reward function is constructed, which introduces a time decay factor; Based on the actual current, rated current, actual electrolysis efficiency and optimal efficiency of the electrolytic cell, an operation stability reward function is constructed, which introduces a time decay factor; Based on the current power and optimal power, an energy consumption economy reward function is constructed, which introduces a time decay factor.

[0009] Preferably, according to the state space and action space, an agent is constructed using a deep deterministic policy gradient framework, comprising: S41, the parameters of the policy network and the value network of the deep deterministic policy gradient framework are initialized by using a Gaussian random distribution; S42, the target policy network and the target value network are set as auxiliary networks, and through soft updating, the parameters of the target policy network and the target value network gradually follow the smooth evolution of the parameters of the policy network and the value network; S43, an experience replay pool is established for storing four-tuple samples generated by the interaction of the agent in the electrolytic cell and the power grid interaction simulation environment; S44, in the deterministic action output by the policy network, an adaptive noise based on constraint violation risk is introduced, and through a multi-constraint safety projection function, all deterministic actions meet the physical constraints of the aluminum electrolytic cell and the power grid frequency modulation protocol, generating constraint corrected actions and interaction samples, and storing the constraint corrected actions and interaction samples in the experience replay pool; S45, using the interaction samples containing constraint corrected actions in the experience replay pool, the value network parameters are updated and trained by minimizing the time difference error, and the policy network parameters are updated and trained by maximizing the expected Q value output by the value network; S46, steps S42 to S45 are repeatedly executed for training until a preset training round is reached; S47, when the preset training round is reached, the agent forms a policy network structure, and the agent construction is completed.

[0010] Preferably, in the deterministic action output by the policy network, an adaptive noise based on constraint violation risk is introduced, and through a multi-constraint safety projection function, all deterministic actions meet the physical constraints of the aluminum electrolytic cell and the power grid frequency modulation protocol, comprising: According to the aluminum electrolytic cell operation mechanism and the power grid frequency modulation requirements, the physical boundary and dynamic change range of each control component in the action space are constructed to form a set of action hard constraints; Based on the set of action hard constraints, a component-level safety projection function is constructed; In the action generation stage, the relative distance of each action component and its corresponding constraint boundary is calculated, and a system-level real-time constraint risk index is established; Generate a noise scaling coefficient based on a system-level real-time constraint risk indicator; Introduce the noise scaling coefficient into the deterministic action output by the policy network; input the deterministic action after introducing the noise scaling coefficient into the component-level safety projection function for correction to generate a corrected execution action; Generate an interaction sample based on the corrected execution action.

[0011] Preferably, an electrolytic cell and power grid interaction simulation environment is constructed, and the state space, action space and composite reward function are jointly input into a deep deterministic policy gradient framework, including: Based on the current-voltage-pole distance electrothermal coupling equation of the aluminum electrolytic cell, an electrolytic cell power output dynamic model is established, and the electrolytic cell power output dynamic model is used as a dynamic node on the load side of the power grid; Combined with the power grid frequency dynamic equation, a bidirectional coupling relationship reflecting the interaction mechanism between power grid frequency change and electrolytic cell power regulation is established; Set a sampling period, and construct a time-domain simulation framework for electrolytic cell and power grid interaction based on the bidirectional coupling relationship; In each sampling period, according to the state space and action space, the immediate influence of the execution action output by the agent on the electrolytic cell power and the power grid frequency is calculated, and the next time state is calculated based on the bidirectional coupling relationship. At the same time, according to the composite reward function, the immediate return value of the agent at this time is calculated to form an interaction sample four-tuple and stored in an experience replay pool; In the time-domain simulation framework, random grid frequency disturbance and load fluctuation scenarios are introduced; Set an upper limit on the step length of a single environment interaction, and according to the frequency deviation recovery time, the electrolytic cell operation stability index and whether the energy consumption deviation reaches the threshold, determine whether the electrolytic cell and power grid interaction is terminated.

[0012] Preferably, the agent is trained offline using a historical operation database to enable the agent to obtain an optimal power regulation strategy, and the optimal power regulation strategy is converted into a control instruction, including: The aluminum electrolytic cell operation parameters and power grid frequency signals collected historically in the historical operation database are used as a historical sample set to establish an offline training sample set containing four-tuples of state, action, reward and next state; Based on the deep deterministic policy gradient framework, the historical sample set is input, and batch replay mechanism is used for offline training; When the agent converges in the offline training phase, an optimal power regulation strategy is formed; The optimal strategy model is input into the time-domain simulation framework for electrolytic cell and power grid interaction, and the actual operation state signal is input for strategy verification; When the grid frequency deviates from the rated value, the intelligent agent outputs a power adjustment instruction according to the real-time observed state quantity.

[0013] Preferably, the operation data of the aluminum electrolytic cell comprises: current, cell voltage, pole distance and power.

[0014] According to a second aspect of the present application, an aluminum electrolytic cell adaptive grid inertia and frequency modulation control device is provided, comprising: a host controller, and a memory connected to the host controller; the memory, wherein the memory stores program instructions; The host controller is used to execute the program instructions stored in the memory to execute the method of any one of the above.

[0015] The technical scheme provided by the embodiments of the present application can include the following beneficial effects: It can be understood that the technical scheme of the present application can collect the operation data of the aluminum electrolytic cell and the grid frequency signal in real time to form a historical operation database; based on the operation data and the grid frequency signal, a state space and an action space of reinforcement learning are constructed respectively; a composite reward function based on the grid frequency deviation, the frequency change rate, the electrolytic cell operation stability index and the energy consumption is constructed; based on the state space and the action space, an intelligent agent is constructed using the deep deterministic policy gradient framework, and in the process of constructing the intelligent agent, an adaptive noise based on constraint violation risk is introduced. It can be understood that this technical scheme can independently map each control action to its physical boundary, ensuring that the electrolytic cell operation parameters always meet the safety constraints under the electro-thermal coupling mechanism. Considering the grid frequency quality, the electrolytic cell operation stability and the energy consumption economy, long-term stable regulation incentive and trend feedforward guidance are realized. The constructed intelligent agent can learn the optimal power regulation strategy that adapts to various disturbance scenarios, significantly improving the frequency deviation suppression capability, the inertia support response speed and the energy efficiency management level, and realizing high-performance and sustainable closed-loop control of industrial load participating in power system auxiliary services.

[0016] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0017] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the present application, and together with the specification, serve to explain the principles of the present application.

[0018] Figure 1 is a step schematic diagram of an aluminum electrolytic cell adaptive grid inertia and frequency modulation control method according to an exemplary embodiment. DETAILED DESCRIPTION

[0019] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not intended to represent all embodiments in accordance with the present application. Rather, they merely represent typical embodiments in accordance with a portion of the application, as detailed in the appended claims.

[0020] In an embodiment, Figure 1 is a schematic diagram of steps of an aluminum electrolytic cell adaptive grid inertia and frequency control method according to an exemplary embodiment, referring to Figure 1 , an aluminum electrolytic cell adaptive grid inertia and frequency control method is provided, comprising: Step S1, real-time collection and storage of operation data of the aluminum electrolytic cell and grid frequency signals to form a historical operation database.

[0021] In this embodiment, the operation data of the aluminum electrolytic cell at least includes: current, cell voltage, pole distance and power.

[0022] Step S2, constructing state space and action space of reinforcement learning according to the historical operation database.

[0023] In a preferred embodiment, this step includes: Step S21, extracting electrolytic cell operation features from operation data according to the electro-thermal coupling operation mechanism of the aluminum electrolytic cell, the electrolytic cell operation features including current , cell voltage and pole distance .

[0024] Step S22, extracting grid frequency features from the grid frequency signal according to the power system frequency dynamic equation, the grid frequency features including frequency deviation , frequency change rate and inertia response characteristics, wherein, is the grid frequency, is time, is the differential symbol.

[0025] Step S23, obtaining historical state features from the historical operation database, the historical state features including last time power adjustment amount and electrolytic cell operation stability index .

[0026] Step S24, jointly defining the electrolytic cell operation features, the grid frequency features and the historical state features as the state space of the reinforcement learning intelligent agent:

[0027] Step S25, according to the adjustment response characteristics of the electrolytic cell to the grid frequency disturbance, the action variables of the reinforcement learning agent are constructed, including: power adjustment action (for indicating the power rise and fall amplitude achieved by changing the direct current or the pole distance of the electrolytic cell), pole distance adjustment action (for fine-tuning the pole distance to indirectly control the cell voltage drop and energy consumption), and current adjustment action (for controlling the distribution of the current channel to optimize the power response speed).

[0028] Step S26, generating the action space of the reinforcement learning agent according to the action variables :

[0029] Step S3, constructing a composite reward function based on the grid frequency deviation, the frequency change rate, the electrolytic cell operation stability index, and the energy consumption.

[0030] Preferably, in order to ensure the dimensional consistency of the reward function, all physical quantities are normalized before calculation: the frequency-related quantities are normalized based on the rated frequency of 50Hz, the power-related quantities are normalized based on the rated power of the electrolytic cell, the current-related quantities are normalized based on the rated current of the electrolytic cell, and the pole distance-related quantities are normalized based on the rated pole distance of the electrolytic cell. Through normalization, it is ensured that the dimensions of each component in the reward function are consistent, and the reward value is a dimensionless quantity.

[0031] In a preferred embodiment, step S3 includes: Step S31, constructing a frequency emergency index according to the real-time collected frequency deviation and frequency change rate of the grid .

[0032] The frequency emergency index is used to quantify the frequency risk level of the current system, providing a quantitative basis for subsequent reward allocation.

[0033] This step first collects the grid frequency deviation and the frequency change rate in real time, and combines the system's preset maximum allowed deviation and the maximum frequency change rate for normalization processing. Then, according to the grid frequency regulation requirements, the weighting coefficients and are set, the deviation amplitude and the change rate are weighted and summed, and the cumulative deviation integral term (inertia response characteristic) weighting coefficient can be introduced to reflect the cumulative frequency risk of the system, and finally the comprehensive quantitative index is obtained, and the emergency adjustment factor​​​ The frequency quality weight is increased, while the operational stability weight is correspondingly reduced. The energy consumption and economic weight is obtained through normalization calculation.

[0034] Step S32: Based on the real-time frequency deviation and frequency change rate, the current operating mode of the power grid and the frequency change direction index are obtained. Dynamic weights are assigned to the frequency quality, operating stability and energy consumption economy of the current operating mode to achieve adaptive switching of the reward center.

[0035] The current operating mode of the power grid can be set according to different frequency deviation ranges, inertia support stages, and energy consumption constraint stages.

[0036] Specifically: First, based on the real-time frequency deviation of the power grid... and its rate of change Determine the current operating mode (including three operating modes: emergency frequency adjustment mode, inertia support mode, and economic optimization mode). For example: frequency deviation range (large deviation, emergency frequency adjustment mode), inertia support stage (rapid frequency change), energy consumption constraint stage (small deviation but energy consumption needs to be optimized).

[0037] Then there are three types of rewards: frequency quality Operational stability Energy consumption economy Assign dynamic weights , , The weight value is adaptively adjusted according to changes in the operating mode, such as increasing it when the frequency deviation is large. To enhance frequency stability, increase the amount of inertia support during the inertia support phase. To ensure stable operation and improve efficiency during energy-constrained phases. To optimize economic efficiency and ensure that the sum of the three weights is 1, the reward focus is dynamically switched according to the grid operation status, thereby guiding the agent to adopt different priority adjustment strategies under different modes.

[0038] In this embodiment, a corresponding basic weight vector is set for each operating mode, wherein: under the emergency frequency modulation mode, the frequency quality weight is... Set to 0.7, runtime stability weight The energy consumption economic weight is 0.25. The weights are set to 0.05 for the three types of modes: 0.4, 0.4, and 0.2 for the inertia support mode and 0.1, 0.3, and 0.6 for the economic optimization mode.

[0039] In addition, this step calculates the frequency change directionality index based on the real-time frequency deviation and frequency change rate, which is used to construct the reward shaping item in step S34.

[0040] Step S33: Construct reward functions with time decay characteristics for frequency quality, operational stability, and energy consumption economy, respectively.

[0041] This step addresses three aspects: frequency quality, operational stability, and energy economy. It constructs reward functions with time decay characteristics to ensure that continuous and stable adjustment behaviors receive higher rewards, while transient fluctuations gradually decay, thereby improving the dynamic stability of the system.

[0042] In aluminum electrolytic cells, when providing frequency regulation services to the power grid, the significant thermal and electrochemical inertia causes the effects of power regulation actions to persist for a period of time. Traditional reinforcement learning's instantaneous reward mechanism cannot effectively distinguish between "brief but drastic fluctuations" and "smooth and sustained regulation." This easily leads the agent to output frequent and oscillating control commands in pursuit of instantaneous high rewards. Although this may improve frequency indicators momentarily, such "short-sighted" behavior severely disrupts the thermal balance and chemical stability of the electrolytic cell and increases overall energy consumption due to repeated equipment operation, failing to achieve truly optimal long-term control effects. The core advantage of introducing time decay characteristics lies in shifting the agent's optimization objective from "pursuing immediate rewards" to "pursuing continuous and stable long-term benefits." Its function is to act as an inherent "smoothener" and "stabilizer". Through the exponential decay factor, behaviors that can maintain the stable operation of the system for a long time and generate lasting positive benefits will receive higher cumulative rewards. This will naturally guide the agent to learn a smooth control strategy that responds to the grid demand while taking into account the inertia of the electrolyzer process. This effectively avoids frequent oscillations in control commands and ultimately significantly improves the operational stability and long-term energy economy of the electrolyzer while ensuring the grid frequency quality.

[0043] In a preferred embodiment, step S33 includes: S331. Construct a time decay factor based on the duration of an action or event and the decay coefficient. :

[0044] in, The attenuation coefficient controls the attenuation rate (unit: 1 / s), with a value range of [0.01, 0.1]. Duration of action or event, indicating the length of time (in seconds) of the current regulatory behavior.

[0045] S332, Frequency deviation of the power grid based on real-time data acquisition With frequency change rate Construct a frequency-quality reward function that incorporates a time decay factor:

[0046] in, The frequency quality is dynamically weighted and adjusted according to the operating mode. This represents the penalty coefficient for the rate of frequency change. In practice, the frequency quality reward can be calculated using the frequency quality reward function described above, based on real-time data collected from the power grid's frequency deviation and rate of frequency change. A time decay factor is introduced during the calculation to encourage the agent to perform frequency recovery actions, thus preventing control command oscillations caused by obtaining high rewards for short-term fluctuations.

[0047] S333, Based on the actual current of the electrolytic cell Rated current Actual electrolysis efficiency and optimal efficiency A runtime stability reward function incorporating a time decay factor is constructed. :

[0048] in, Dynamic weights for operational stability, Rated current, The current deviation sensitivity coefficient, This is the current stability bonus coefficient, used to adjust the contribution of current deviation from the rated value to the operational stability bonus. This is the penalty coefficient for the pole distance / efficiency deviation.

[0049] In practice, the above-mentioned operational stability reward function can be used to determine the actual current of the electrolytic cell. Rated current Actual electrolysis efficiency and optimal efficiency The calculation of operational stability rewards involves introducing a time decay factor to guide the agent to prioritize maintaining the long-term stable operation of the electrolyzer, rather than pursuing short-term precise matching of the operating point.

[0050] S334, Based on current power With optimal power An energy consumption economic reward function incorporating a time decay factor was constructed. :

[0051] in, As a dynamic weight for energy consumption economy, This represents the current electrolytic cell power. The optimal power target (unit: MW). This is a penalty factor (unit: 1 / MW²), used to suppress excessive power deviation. This is the reward coefficient (dimensionless), used to encourage power to approach the optimal value.

[0052] In practice, the energy consumption economic reward function described above can be used to determine the current power output. With optimal power Calculate the energy consumption economic reward In the calculation process, a time decay factor is introduced. It is used to optimize long-term energy efficiency during frequency regulation, enabling the agent to learn strategies that bring the average power of the electrolyzer closer to the optimal economic point while meeting the grid demand.

[0053] Step S34: Construct reward shaping items based on the frequency change directionality index.

[0054] When calculating the frequency change directionality index, the system first acquires the grid frequency deviation in real time. and frequency change rate Then through the symbolic function Determine the direction of the frequency change (increasing or decreasing), and then calculate the frequency deviation amplitude. With dynamic weights Multiplying them together yields the directional index:

[0055] This indicator reflects the trend of the current frequency deviation over time, and is used to reward shaping items. Provides feedforward reference, enabling the agent to adjust its action strategy based on the rising or falling frequency trend to accelerate frequency recovery or stabilize operation.

[0056] Constructing reward shaping items based on frequency change direction indicators:

[0057] in, The bonus shaping coefficient (dimensionless) has a value range of [0,1] and is used to adjust the intensity of the feedforward adjustment. The optimal value is determined through offline parameter tuning.

[0058] Step S35: Weight and fuse each reward function according to dynamic weights, and then weight and fuse it with the reward shaping term to obtain a composite reward function.

[0059] The reward shaping term Φ(t) is weighted and fused with the composite reward function to adjust the value of the composite reward function according to the change of frequency deviation over time. This achieves a feedforward response to the frequency deviation trend, that is, the magnitude and direction of the frequency deviation are converted into numerical adjustment amounts and added to the composite reward function. This means that when the frequency deviation increases over time, the penalty value in the reward function is increased, and when the frequency deviation decreases over time, the positive reward value in the reward function is increased. Thus, in the reinforcement learning process, the agent can make feedforward adjustments to the action strategy according to the frequency change trend, while maintaining operational stability and energy economy constraints.

[0060] It is understood that the technical solution of this embodiment enables the intelligent agent to automatically switch the primary control objective under different power grid conditions, just like an experienced expert, through "dynamic weight allocation," thereby achieving dynamic priority adaptation. At the same time, by introducing a directional indicator of frequency change through "reward shaping term," the intelligent agent is provided with "trend prediction" information, enabling it to upgrade from a passive response of "reducing the current deviation" to an active intervention of "suppressing the deterioration trend and accelerating the recovery process." This significantly improves the dynamic performance and control quality of the frequency regulation process and guides the intelligent agent to learn a smarter and more forward-looking optimal strategy.

[0061] Step S4: Based on the state space and action space, construct an agent using the Deep Deterministic Policy Gradient (DDPG) framework; during the construction of the agent, introduce adaptive noise based on constraint violation risk, and use component-level safety projection functions to ensure that the action space satisfies the physical constraints of the aluminum electrolytic cell and the power grid frequency regulation protocol.

[0062] In a preferred embodiment, step S4, which involves constructing an agent using a deep deterministic policy gradient framework based on the state space and action space, includes: S41. The parameters of the policy network (Actor) and value network (Critic) of the deep deterministic policy gradient framework are initialized using a Gaussian random distribution to ensure that the initial weights of the network have reasonable randomness and learnability.

[0063] In practice, the first step is to initialize all weight parameters of the policy network (Actor) and value network (Critic) to ensure that the network has reasonable randomness and learnability in the early stages of training. Specifically, the weight parameters of each network layer are initialized with a mean of zero and a variance of... Gaussian distribution Initial values ​​are generated by random sampling, and the bias parameters can be set to zero or subject to small random perturbations to avoid excessive deviations in the initial action output or value assessment. This Gaussian distribution-based initialization method can break the network symmetry, ensure the output differences of each neuron in the early stage of training, enable the policy network and value network to have sufficient exploration capabilities, and provide stable gradient signals for subsequent iterative updates based on experience playback, thereby promoting the efficient convergence of the DDPG agent in the electrolyzer-grid interaction environment.

[0064] S42. Set up the target policy network and the target value network as auxiliary networks, and use soft update methods (such as parameter update rate) to update them. This allows the parameters of the target policy network and the target value network to evolve smoothly in accordance with the parameters of the policy network and the value network, thereby avoiding gradient oscillations and improving training stability.

[0065] In practice, firstly, corresponding target networks (Target Actor and Target Critic) are established for the main policy network (Actor) and the main value network (Critic) respectively, serving as auxiliary networks to calculate the target values ​​required for training. In each training iteration, the parameters of the target network gradually converge to the parameters of the main network through a soft update mechanism.

[0066] Specifically, this is achieved in the following way: Target policy network parameters according to renew; Target value network parameters according to renew; in The parameter update rate is used to control the smoothness of changes in the target network parameters.

[0067] Through this mechanism, the target network parameters evolve slowly with the main network, making the target Q value calculated by the Critic during training more stable. The Actor obtains a reliable reference in policy updates, thereby avoiding gradient oscillations, improving training convergence, and ensuring that the agent's adaptive power regulation strategy can be stably optimized in the electrolyzer and power grid interaction environment.

[0068] S43. Establish an experience playback pool to store quadruple samples generated by the agent in the simulation environment of interaction between the electrolyzer and the power grid.

[0069] Specifically, to support offline batch training of the DDPG agent, an experience replay buffer is first initialized to store the interaction data generated by the agent in the electrolyzer-grid simulation environment. Whenever the agent outputs an action based on the current state and applies it to the simulation environment, the environment returns an immediate reward and the next state, forming a quadruple sample. This quadruple is stored sequentially in the experience replay buffer, and its capacity is managed according to the first-in-first-out (FIFO) rule. When the buffer is full, the earliest sample is overwritten. During training, the DDPG agent randomly selects a small batch of samples from the replay buffer to update the Critic and Actor networks, thereby breaking the temporal correlation between samples, reducing training variance, and enhancing the generalization ability of the policy network to different combinations of states and actions, thus improving the stability and convergence of the agent in the electrolyzer-grid dynamic environment.

[0070] S44. In the deterministic actions output by the policy network, adaptive noise based on constraint violation risk is introduced. Through the multi-constraint safety projection function, all deterministic actions are made to satisfy the physical constraints of the aluminum electrolytic cell and the power grid frequency regulation protocol. Constraint-corrected actions and interaction samples are generated and stored in the experience replay pool to provide data support for subsequent network training.

[0071] This embodiment primarily addresses the critical issue that when a reinforcement learning agent explores and learns in an aluminum electrolysis cell-grid interaction environment, its output control actions may violate the physical safety constraints of the aluminum electrolysis cell itself (such as exceeding current and electrode spacing limits) and the grid frequency regulation protocol (such as exceeding power regulation limits). Specifically, the random exploration mechanism of the traditional DDPG algorithm is blind, resulting in invalid or even dangerous actions that are harmful to the equipment or unacceptable to the grid. This not only leads to low training sample efficiency and difficulty in learning convergence but also hinders the practical application of reinforcement learning algorithms in such safety-critical industrial processes. By using "adaptive noise based on constraint violation risk" to intelligently suppress exploration in high-risk directions before action generation, and then using "component-level safety projection function" to force safety correction on the final output action, this approach ensures the absolute safety of actions executed at every moment, avoids learning oscillations that may be caused by simple limiting, and guides the agent to learn the optimal strategy for safety compliance more efficiently than ex-post punishment. This significantly improves training efficiency and the feasibility and reliability of the final strategy in real industrial environments.

[0072] In a preferred embodiment, in step S44, adaptive noise based on constraint violation risk is introduced into the deterministic actions output by the policy network. A multi-constraint safety projection function is used to ensure that all deterministic actions satisfy the physical constraints of the aluminum electrolytic cell and the power grid frequency regulation protocol, including: Step S441: Based on the operating mechanism of aluminum electrolysis cells and the frequency regulation requirements of the power grid, construct the physical boundaries and dynamic change ranges of each control component in the action space to form a set of hard constraints for the action.

[0073] This step, based on the operating mechanism of aluminum electrolytic cells and the frequency regulation requirements of the power grid (the operating mechanism of aluminum electrolytic cells is mainly based on the electrothermal coupling relationship of current-voltage-electrode spacing; its power output is affected by the current density, cell voltage, and electrode spacing adjustment of the electrolytic cell, and is also constrained by the electrolyte temperature and electrode state, ensuring the stability and optimal efficiency of the electrolysis process; the frequency regulation requirements of the power grid stipulate that the load-side response should be able to provide inertia support and primary frequency regulation capability when the power grid frequency deviation occurs, that is, to limit the frequency deviation amplitude and improve the frequency recovery speed by rapidly adjusting the power response, while ensuring that the power adjustment amplitude and rate are within the physical safety range of the electrolytic cell, thereby realizing the dynamic support and stable regulation of the power grid frequency by the electrolytic cell), constructs the physical boundaries and dynamic change ranges of each control component in the action space, forming a set of hard constraints for the action. :

[0074] In the formula, This represents the maximum permissible range of power regulation (unit: MW). This is the minimum polar distance (unit: m). This represents the maximum polar distance (unit: m). This is the minimum allowable current (unit: kA). This represents the maximum allowable current (unit: kA). For time, This is the maximum allowable value for the rate of change of current (unit: kA / s).

[0075] Step S442: Based on the action hard constraint set Ω, construct the component-level safe projection function:

[0076] In the formula, The first output of the agent policy network Each motion component (without constraint correction). For the first Each action component in the action hard constraint set The lower bound in the middle, For the first Each action component in the action hard constraint set The upper bound of the range.

[0077] Component-level safety projection functions are used to independently correct each motion component, ensuring that it always remains within the dynamic safety boundary. This ensures the physical feasibility of the action output and the safety of system operation; that is, it independently maps the action components to their corresponding dynamic safety intervals. When an action exceeds the boundary, the projection function automatically truncates it to the boundary value, ensuring that each action component will not exceed the physical limit or violate the grid frequency regulation constraint at any time. This enables component-by-component constraint correction based on the action output by the strategy network, ensuring the safety and feasibility of the final action in the electrolyzer-grid interaction environment.

[0078] Step S443: In the action generation stage, calculate the relative distance between each action component and its corresponding constraint boundary. And establish system-level real-time constraint risk indicators. .

[0079] In practice, the motion generation stage first processes each motion component. Calculate its relationship with the corresponding constraint boundary. relative distance This is used to quantify the degree to which an action component deviates from the center of the safe zone, specifically the normalized minimum value of the distance between the action and the upper and lower boundaries. Then, the relative distances of all action components are averaged and this value is subtracted from 1 to form a system-level real-time constraint risk indicator. It is used to comprehensively reflect the overall risk level of the agent's current output action approaching the constraint boundary, and to provide a reference for subsequent adaptive noise scaling and safe projection, thereby assessing and controlling potential constraint violation risks in real time during the action generation process.

[0080] Among them, relative distance For (the first) (Normalized minimum distance from each action component to its constraint boundary)

[0081] System-level real-time constraint risk indicators (Indicates the degree of proximity between the current action and the constraint boundary):

[0082] In the formula, Output the action for the current time-to-time strategy. This represents the total number of action components in the action vector.

[0083] Step S444: Based on system-level real-time constraint risk indicators Generate noise scaling factor :

[0084] in, This is the risk sensitivity coefficient, with a value range of [1, 10]. It is determined through offline parameter tuning and is used to adjust the sensitivity of the noise scaling factor to constraint risk; as the action gradually approaches the boundary (i.e., When the noise scaling factor increases, it decays exponentially, suppressing the risk of constraint violation caused by overexploration; when the system is in a safe region, the noise intensity remains at a high level to enhance the policy exploration.

[0085] Step S445: Introduce a noise scaling factor into the deterministic actions output by the policy network. The deterministic motion, after introducing a noise scaling factor, is input into the component-level secure projection function. Perform corrections and generate corrected execution actions. .

[0086] The noise scaling factor is calculated using the following expression:

[0087] in, Indicates Gaussian noise. For the policy network (Actor) in state The deterministic action of the output, The noisy initial action output by the policy network (without constraint projection correction).

[0088] The action is represented as:

[0089] Ensure that the final executed action strictly meets the physical and frequency modulation constraints; when At that time, the system generates a projection flag signal. This indicates that the action was truncated due to constraint modification.

[0090] Step S446: Generate interaction samples based on the corrected execution actions:

[0091] In the formula, For state vectors, For the intelligent agent to perform actions The instant reward obtained afterward For the intelligent agent to perform actions The state at the next time step after the environment transition is stored in the experience replay pool for subsequent training of the Critic and Actor networks.

[0092] S45. Using the interaction samples of the constraint-corrected actions contained in the experience replay pool, update the parameters of the value network (Critic) by minimizing the temporal difference (TD) error, and update the parameters of the training policy network (Actor) by maximizing the expected Q value of the value network (Critic) output.

[0093] In practice, the DDPG agent first randomly selects interaction samples containing constrained and corrected actions from the experience replay pool. Each sample includes the current state, action, reward, and next state. The agent calculates the target action and target Q-value for the next state through the target network and compares it with the Q-value output by the main value network to obtain the temporal difference (TD) error. Then, it minimizes this error to update the Critic network parameters. Subsequently, the Critic network is used to evaluate the value of the current policy output action, and the Actor network parameters are updated by maximizing the expected Q-value through gradient ascent. This allows the policy network to gradually optimize action selection while satisfying action constraints, enabling the agent to learn adaptive and stable power regulation strategies in the electrolyzer-grid interaction environment.

[0094] S46. Repeat steps S42 to S45 for training until the preset training round n is reached. That is, the agent continuously performs a cyclical iterative process of state perception, action selection, environmental interaction, reward acquisition, experience storage, and network parameter update during training. The agent updates the Critic and Actor networks by repeatedly drawing samples from the experience replay pool and using the target network for soft updates, gradually optimizing the policy until the policy converges.

[0095] S47. After reaching the preset training round n, the agent forms a policy network structure and can autonomously output the optimal frequency regulation control command according to the power grid status in real time, thus completing the construction of the agent.

[0096] Step S5: Construct an interactive simulation environment between the electrolyzer and the power grid, and jointly input the state space, action space, and composite reward function into the Deep Deterministic Policy Gradient (DDPG) framework.

[0097] In a preferred embodiment, step S5 includes: Step S51: Based on the current-voltage-electrode spacing electrothermal coupling equation of the aluminum electrolytic cell, establish a dynamic model of the electrolytic cell power output, and use this dynamic model as a dynamic node on the grid load side to realize real-time mapping between the electrolytic cell operating state and grid load changes. The dynamic model of the electrolytic cell power output is used to describe the influence of current density, cell voltage, and electrode spacing changes on electrolytic power and thermal balance.

[0098] In practical application, firstly, based on the electrothermal coupling mechanism of aluminum electrolytic cells, a description of the cell current is established. Slot voltage With polar distance The dynamic equations relating the two components are used to calculate the instantaneous electrolysis power. (i.e., the dynamic model of electrolytic cell power output), combined with the heat balance equation of the electrolysis process, considers the influence of current density changes on temperature, electrolyte conductivity and cell voltage; then, the dynamic model is mapped to the grid load side as an adjustable dynamic load node to realize the real-time response of electrolytic cell power output to grid frequency disturbances, so that the agent can select power adjustment actions according to the current state in the simulation environment and observe the immediate impact of the actions on electrolytic cell power and grid frequency, thus forming a closed-loop interactive simulation.

[0099] Step S52: Combining the power grid frequency dynamic equation, establish a two-way coupling relationship that reflects the interaction mechanism between power grid frequency changes and electrolytic cell power regulation (i.e., the two-way coupling relationship between electrolytic cell power output and power grid frequency response), which is used to describe the inertial support and primary frequency regulation effect of electrolytic cell power adjustment on power grid frequency response.

[0100] In practice, The dynamic equation for power grid frequency is:

[0101] in, For grid mechanical power input, For grid load power output, This is the frequency damping coefficient. For frequency deviation, This is the equivalent inertia.

[0102] First, based on the dynamic equation of the power grid frequency, the dynamic model of the electrolytic cell power output is established. Dynamic power at load nodes A portion is introduced into the power grid frequency equation, while also incorporating the power grid frequency deviation. Feedback is fed back to the electrolyzer power regulation strategy, which is then implemented by the intelligent agent based on... and frequency change rate Output power regulation action This enables real-time adjustment of the electrolytic cell power to the grid frequency; changes in the electrolytic cell power, in turn, affect... By connecting the power grid frequency, a closed-loop bidirectional coupling is formed, enabling the electrolyzer to provide inertial support for frequency deviations and perform primary frequency regulation, while ensuring dynamic interaction and stable response between the power grid and the electrolyzer.

[0103] Step S53: Set the sampling period and construct a time-domain simulation framework for the interaction between the electrolyzer and the power grid based on the bidirectional coupling relationship.

[0104] In practice, this step sets a uniform sampling period. Furthermore, a time-domain simulation framework for the interaction between the electrolyzer and the power grid is constructed based on the bidirectional coupling relationship, enabling simulation of the system state space. Action space and composite reward function The simulation involves continuous iterative calculation and dynamic updating. The time-domain simulation framework for the interaction between the electrolyzer and the power grid includes five stages: state observation, action selection, physical model response, reward calculation, and state update. State observation is derived from the real-time state variables of the electrolyzer and the power grid. Action selection is performed by the DDPG agent, which outputs deterministic control commands based on the current state. These commands are then corrected by a secure projection function and applied to the simulation model. The simulation environment updates the power grid frequency and electrolyzer operating parameters at each moment, forming a closed-loop interactive structure.

[0105] Step S54, within each sampling period According to the state space With action space The execution action output by the computational agent. The immediate impact on electrolyzer power and grid frequency is determined, and the state at the next moment is calculated based on the bidirectional coupling relationship. Meanwhile, according to the composite reward function Calculate the agent's instantaneous reward value at that moment to form an interaction sample quadruple. The data is stored in the experience replay pool for use in training and updating the policy network and value network.

[0106] Step S55: In the time-domain simulation framework, randomly introduce power grid frequency disturbance and load fluctuation scenarios.

[0107] In the time-domain simulation framework, scenario parameters are randomly sampled during the training phase. By changing the initial frequency deviation, frequency modulation coefficient, and electrolytic cell thermal inertia parameters, the adaptive and steady-state recovery capabilities of the agent under different frequency disturbance conditions are enhanced. Specifically, in the time-domain simulation framework, to enhance the agent's adaptive capability under different frequency disturbance conditions, the initial frequency deviation of the power grid is first randomly generated. Grid frequency regulation coefficient and the thermal inertia parameters of the electrolytic cell Key system parameters were used to construct various disturbance and load fluctuation scenarios; then, within each simulation step, random disturbance signals were superimposed onto the dynamic equation of the power grid frequency. Or change the power of the load node The instantaneous values ​​are used to simulate load fluctuations and sudden events in actual power grid operation. The agent performs state observation, action selection and environmental interaction in these random scenarios, so that the generated experience samples cover a variety of frequency deviations, rates of change and power fluctuations, thereby improving the steady-state recovery capability and dynamic adaptability of the policy network under different operating modes.

[0108] Step S56: Set the upper limit of the step size for a single environmental interaction. Based on the frequency deviation recovery time, the electrolyzer's operational stability indicators, and whether the energy consumption deviation reaches a threshold, a determination is made as to whether the interaction between the electrolyzer and the power grid has terminated. Preferably, the power grid frequency recovery time is also recorded. Steady-state error Current fluctuation rate and energy consumption deviation rate These performance indicators provide a quantitative basis for evaluating the training effect and strategy optimization of the agent.

[0109] In practice, the step size of each single environmental interaction Then, monitor the current status; if all conditions are met simultaneously... , and If the interaction between the electrolyzer and the power grid terminates prematurely, the performance indicators are recorded; if the interaction between the electrolyzer and the power grid does not meet the above conditions, but the number of steps reaches [a certain threshold], [the following applies]. The interaction between the electrolyzer and the power grid also terminates, and the final state is recorded.

[0110] In each interaction cycle between the electrolyzer and the power grid, the agent operates according to the state space. Select Action The time-domain simulation framework calculates the new system state and reward, and transmits the interaction samples to the experience replay pool in real time. The DDPG framework uses the experience samples to update the parameters of the Actor and Critic networks, and achieves stable convergence through soft updates of the target network. After training convergence, the strategy output in the time-domain simulation framework is the optimal power regulation strategy for the electrolyzer and power grid system under different operating modes. By embedding the trained and converged agent into the time-domain simulation framework, real-time frequency disturbance input and dynamic power response output are achieved.

[0111] Step S6: Use the historical operation database to perform offline training on the agent, so that the agent can obtain the optimal power adjustment strategy, and convert the optimal power adjustment strategy into control commands.

[0112] In a preferred embodiment, step S6 includes: Step S61: Using the historical operating parameters of aluminum electrolytic cells and power grid frequency signals collected in the historical operating database as a historical sample set, establish an offline training sample set containing a quadruple of state, action, reward, and next state. This sample set is used to characterize the dynamic response of the electrolytic cell under different frequency perturbation conditions, providing data support for the agent's strategy optimization.

[0113] Step S62: Based on the Deep Deterministic Policy Gradient (DDPG) framework, offline training is performed using a batch replay mechanism with historical sample sets as input.

[0114] In practice, the historical sample set is input into the experience replay pool, and a batch replay mechanism is adopted. Each time, a certain number of samples are randomly selected from the replay pool to form a training batch. In each training batch, the parameters of the value network (Critic) are updated by minimizing the temporal difference (TD) error, and the expected Q value output by the Critic network is used to optimize the parameters of the policy network (Actor) using gradient ascent. At the same time, the target policy network and the target value network are softly updated to ensure that the training process is smooth and converges stably. By continuously iterating and extracting batches and updating network parameters, the policy network learns the optimal power regulation strategy under the historical data distribution, realizing the mapping relationship between state and action, thereby providing a reliable model for the agent to output the optimal frequency modulation control command in real time operation.

[0115] Step S63: After the agent converges during the offline training phase, an optimal power adjustment strategy is formed.

[0116] In practice, the training error changes of the policy network (Actor) and value network (Critic) are monitored. When the variation of the policy output action and the corresponding Q value predicted by the value network tends to stabilize over several consecutive training rounds, and the TD error converges to a preset small threshold 'a', the agent training is considered to have converged. Subsequently, the policy network parameters are fixed. As the optimal policy function This policy function can be based on the input state vector The optimal power adjustment action is adaptively output, which means obtaining the mapping relationship between state and action.

[0117] The input state vector is represented as:

[0118] The mapping relationship is as follows:

[0119] in, For the corresponding optimal power regulation action (at least including power regulation) Pole pitch adjustment Current regulation This strategy can adaptively adjust the amplitude and direction of output power according to the real-time status, achieving dynamic optimal control that balances frequency stability and energy consumption.

[0120] Step S64: Input the optimal strategy model into the time-domain simulation framework for the interaction between the electrolyzer and the power grid, and input actual operating state signals for strategy verification. After verifying that the strategy performance meets the control objectives by evaluating the frequency deviation recovery time, current fluctuation amplitude, and energy consumption indicators, the strategy model is deployed to the online control module of the cell control system.

[0121] In practical application, the optimal power adjustment action will be implemented. After correction using a component-level secure projection function, the simulation model is applied. The simulation framework calculates the electrolyzer power and grid frequency response based on the grid frequency dynamic equation and updates the state for the next time step. Within each simulation step, a composite reward function is simultaneously utilized. Evaluate the strategy performance and record key metrics such as frequency recovery time. Current fluctuation amplitude and energy consumption deviation By comparing these indicators with control objectives, the stability, frequency regulation effect, and energy consumption economy of the optimal strategy under actual conditions are verified (e.g., whether the frequency deviation is within the allowable range, whether the recovery time meets the frequency regulation requirements, whether the current fluctuation is stable, and whether the energy consumption is close to the optimal value), providing a basis for the deployment of the strategy to the tank control system.

[0122] Step S65: When the grid frequency deviates from the rated value, the intelligent agent outputs a power adjustment command based on the real-time observed state variables.

[0123] When the grid frequency deviates from the rated value k, the agent adjusts its state variables based on real-time observations. Output power adjustment command (That is: inputting it into the optimal policy network obtained through training) The strategy network, based on the state-action mapping relationship learned during offline training, performs feature extraction and nonlinear mapping calculation on the current electrolyzer operating state and grid frequency characteristics, and outputs the corresponding power regulation amplitude; the cell control system converts this regulation command into a current setpoint. (Specifically: adjust the power range) Converted into current increment according to the mapping relationship (The mapping relationship is as follows:) , The slot voltage is determined by the current. and polar distance The decision (reflecting the electrothermal coupling characteristics of the electrolytic cell) ensures that current changes can accurately achieve the required power adjustment, which is executed by the power control unit to realize the inertia support and rapid frequency regulation response of the aluminum electrolytic cell to the power grid.

[0124] It is understandable that the technical solution of the present invention has the following beneficial effects: By constructing a component-level safety projection function and an adaptive noise mechanism based on constraint violation risk, dual safety guarantees for the action space are achieved within the Deep Deterministic Policy Gradient (DDPG) framework. On one hand, the component-level safety projection function independently maps each control action (such as power adjustment, electrode spacing change, and current adjustment) to its physical boundary, ensuring that the electrolytic cell operating parameters (such as current, voltage, and electrode spacing) always meet the safety constraints under the electrothermal coupling mechanism. On the other hand, a system-level real-time constraint risk index dynamically assesses the risk of actions approaching the boundary and adaptively adjusts the exploration noise intensity through an exponentially decaying noise scaling factor—suppressing exploration when the system approaches the safety limit and enhancing exploration capabilities when in a safe region. This mechanism, while ensuring the long-term stable operation of the aluminum electrolytic cell, improves the agent's dynamic response capability and policy robustness under complex power grid disturbances, effectively avoiding equipment damage or process instability problems caused by blind exploration in traditional reinforcement learning.

[0125] This invention designs a dynamic weighted composite reward function based on operational mode recognition, comprehensively considering three types of indicators: grid frequency quality, electrolyzer operational stability, and energy consumption economy. It introduces a time decay factor and a reward shaping term to achieve long-term stable regulation incentives and trend feedforward guidance. Through a dynamic weight allocation function, the system can automatically switch the control focus based on the current frequency deviation and rate of change—for example, prioritizing frequency recovery under large disturbances and optimizing energy consumption in steady state. Simultaneously, the reward shaping term uses the directional information of frequency changes to feedforward adjust the reward, encouraging the agent to take reverse adjustment actions in advance, accelerating the frequency recovery speed. Combining an electrolyzer-grid bidirectional coupling simulation environment with an offline-online collaborative training mechanism, the agent can learn optimal power regulation strategies to adapt to various disturbance scenarios, significantly improving frequency deviation suppression capability, inertia-supported response speed, and energy efficiency management level, achieving high-performance, sustainable closed-loop control of industrial loads participating in power system ancillary services.

[0126] In another embodiment, an adaptive power grid inertia and frequency modulation control device for aluminum electrolysis cells is provided, comprising: The main controller and the memory connected to the main controller; The memory stores program instructions; The main controller is used to execute program instructions stored in the memory and perform any of the methods described above.

[0127] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0128] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0129] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0130] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0131] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0132] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0133] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0134] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0135] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for adaptive power grid inertia and frequency modulation control of aluminum electrolytic cells, characterized in that, include: Real-time acquisition and storage of operating data of aluminum electrolytic cells and power grid frequency signals to form a historical operating database; The state space and action space for reinforcement learning are constructed based on the historical operational database. Construct a composite reward function based on grid frequency deviation, frequency change rate, electrolyzer operation stability index, and energy consumption; Based on the state space and action space, an intelligent agent is constructed using a deep deterministic policy gradient framework. In the process of constructing the intelligent agent, adaptive noise based on constraint violation risk is introduced, and the action space is made to satisfy the physical constraints of the aluminum electrolytic cell and the power grid frequency regulation protocol through a component-level safety projection function. A simulation environment for interaction between the electrolyzer and the power grid is constructed, and the state space, action space, and composite reward function are jointly input into a deep deterministic policy gradient framework. The agent is trained offline using a historical operating database to obtain the optimal power regulation strategy, which is then converted into control commands.

2. The method according to claim 1, characterized in that, Based on the historical operational database, construct the state space and action space for reinforcement learning, including: Based on the electrothermal coupling operation mechanism of aluminum electrolytic cells, the operating characteristics of the electrolytic cells are extracted from the operating data. The operating characteristics of the electrolytic cells include current, cell voltage, and electrode spacing. Based on the power system frequency dynamic equation, the power grid frequency characteristics are extracted from the power grid frequency signal. The power grid frequency characteristics include frequency deviation, frequency change rate and inertia response characteristics. Historical state characteristics are derived from the historical operation database, including the power regulation amount and the tank operation stability index at the previous moment. The electrolyzer operating characteristics, grid frequency characteristics, and historical state characteristics are jointly defined as the state space of the reinforcement learning agent. Based on the regulation response characteristics of the electrolyzer to grid frequency disturbances, action variables of the reinforcement learning agent are constructed, including: power regulation action, electrode spacing regulation action, and current regulation action. The action space of the reinforcement learning agent is generated based on the action variables.

3. The method according to claim 1, characterized in that, A composite reward function is constructed based on grid frequency deviation, frequency change rate, electrolyzer operating stability index, and energy consumption, including: A frequency urgency index is constructed based on the real-time collected frequency deviation and frequency change rate of the power grid. Based on real-time frequency deviation and frequency change rate, the current operating mode of the power grid and the frequency change directionality index are obtained, and dynamic weights are assigned to the frequency quality, operating stability and energy economy of the current operating mode. Reward functions with time decay characteristics are constructed for frequency quality, operational stability, and energy consumption economy, respectively. Reward shaping items are constructed based on the directional indicators of frequency changes; The individual reward functions are weighted and fused according to dynamic weights, and then weighted and fused with the reward shaping term to obtain a composite reward function.

4. The method according to claim 3, characterized in that, Reward functions with time decay characteristics are constructed for frequency quality, operational stability, and energy efficiency, respectively, including: Construct a time decay factor based on the duration of an action or event and the decay coefficient; Based on real-time collected data on the frequency deviation and frequency change rate of the power grid, a frequency quality reward function incorporating a time decay factor is constructed. Based on the actual current, rated current, actual electrolysis efficiency, and optimal efficiency of the electrolyzer, an operational stability reward function incorporating a time decay factor is constructed. Based on the current power and the optimal power, an energy consumption economic reward function with a time decay factor is constructed.

5. The method according to claim 1, characterized in that, Based on the state space and action space, an agent is constructed using a deep deterministic policy gradient framework, including: S41. The parameters of the policy network and value network of the deep deterministic policy gradient framework are initialized using a Gaussian random distribution. S42. Set the target policy network and the target value network as auxiliary networks, and use soft update to make the parameters of the target policy network and the target value network evolve smoothly in accordance with the parameters of the policy network and the value network. S43. Establish an experience playback pool to store quadruple samples generated by the agent in the simulation environment of interaction between the electrolyzer and the power grid. S44. In the deterministic actions output by the policy network, adaptive noise based on constraint violation risk is introduced. Through the multi-constraint safety projection function, all deterministic actions are made to satisfy the physical constraints of the aluminum electrolytic cell and the power grid frequency regulation protocol. Constraint-corrected actions and interaction samples are generated and stored in the experience playback pool. S45. Using the interaction samples of the constraint-corrected actions contained in the experience replay pool, update the parameters of the value network by minimizing the temporal difference error, and update the parameters of the training policy network by maximizing the expected Q value of the value network output. S46. Repeat steps S42 to S45 for training until the preset number of training rounds is reached. S47. After reaching the preset number of training rounds, the agent forms a policy network structure, completing the agent construction.

6. The method according to claim 5, characterized in that, In the deterministic actions output by the policy network, adaptive noise based on constraint violation risk is introduced. A multi-constraint safety projection function ensures that all deterministic actions satisfy the physical constraints of the aluminum electrolytic cell and the power grid frequency regulation protocol, including: Based on the operating mechanism of aluminum electrolytic cells and the frequency regulation requirements of the power grid, the physical boundaries and dynamic change ranges of each control component in the action space are constructed to form a set of hard constraints for the action. Construct a component-level safe projection function based on the set of hard constraints for actions; During the action generation phase, the relative distance between each action component and its corresponding constraint boundary is calculated, and a system-level real-time constraint risk index is established. Noise scaling factor is generated based on system-level real-time constraint risk indicators; A noise scaling factor is introduced into the deterministic action output by the policy network; the deterministic action with the noise scaling factor is then input into the component-level safe projection function for correction, generating the corrected execution action. Interactive samples are generated based on the corrected execution actions.

7. The method according to claim 1, characterized in that, A simulation environment for the interaction between the electrolyzer and the power grid is constructed, and the state space, action space, and composite reward function are jointly input into a deep deterministic policy gradient framework, including: Based on the current-voltage-electrode spacing electrothermal coupling equation of aluminum electrolytic cells, a dynamic model of electrolytic cell power output is established, and the dynamic model of electrolytic cell power output is used as a dynamic node on the load side of the power grid. By combining the dynamic equation of power grid frequency, a two-way coupling relationship is established to reflect the interaction mechanism between power grid frequency changes and electrolytic cell power regulation. Set the sampling period and construct a time-domain simulation framework for the interaction between the electrolyzer and the power grid based on the bidirectional coupling relationship; Within each sampling period, based on the state space and action space, the immediate impact of the agent's output actions on the electrolyzer power and grid frequency is calculated, and the state at the next moment is calculated based on the bidirectional coupling relationship. At the same time, the immediate reward value of the agent at that moment is calculated based on the composite reward function, forming an interactive sample quadruple, which is stored in the experience replay pool. In the time-domain simulation framework, scenarios of power grid frequency disturbances and load fluctuations are randomly introduced; Set an upper limit for the step size of a single environmental interaction, and determine whether the interaction between the electrolyzer and the power grid should be terminated based on the frequency deviation recovery time, the electrolyzer's operational stability index, and whether the energy consumption deviation reaches the threshold.

8. The method according to claim 1, characterized in that, The agent is trained offline using a historical operational database to obtain an optimal power regulation strategy. This optimal power regulation strategy is then converted into control commands, including: Using historical operating parameters of aluminum electrolytic cells and power grid frequency signals collected in the historical operating database as historical sample sets, an offline training sample set containing a quadruple of state, action, reward and next state is established. Based on the deep deterministic policy gradient framework, the system uses historical sample sets as input and employs a batch replay mechanism for offline training. Once the agent converges during the offline training phase, it will form the optimal power adjustment strategy. The optimal strategy model is input into the time-domain simulation framework of the interaction between the electrolyzer and the power grid, and the actual operating state signal is input for strategy verification. When the grid frequency deviates from the rated value, the agent outputs a power adjustment command based on the real-time observed state variables.

9. The method according to claim 1, characterized in that, The operating data of the aluminum electrolysis cell includes: current, cell voltage, electrode spacing, and power.

10. An adaptive power grid inertia and frequency modulation control device for aluminum electrolysis cells, characterized in that, include: The main controller and the memory connected to the main controller; The memory stores program instructions; The main controller is used to execute program instructions stored in the memory to perform the method as described in any one of claims 1 to 9.

Images