Power flow regulation generation method and device based on supervised fine-tuning and reinforcement learning

Abstract

The application provides a power flow regulation generation method and device based on supervised fine-tuning and reinforcement learning, which comprises the following steps: obtaining power system state observation data; inputting the power system state observation data into a power flow regulation generation model to obtain a regulation generation result output by the power flow regulation generation model; wherein the power flow regulation generation model is obtained by using a first observation training sample constructed by using forward disturbance sampling and trajectory inversion based on observation training data to supervise fine-tune an initial power flow regulation generation model trained, and by using a group relative strategy optimization algorithm to reinforce learning training of the model obtained by the supervision fine-tuning. The application can enable the model to deeply perceive the physical mechanism and dynamic boundary of the power system, avoid generalization failure when facing high-difficulty regulation tasks, ensure that the model can autonomously explore a better strategy in a multi-dimensional action space, and improve the accuracy, generalization ability and reliability of the model in power flow regulation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating power flow regulation based on supervised fine-tuning and reinforcement learning. Background Technology

[0002] With the continuous expansion of power system scale and the increasing complexity of its operation, ensuring the safe and stable operation of the power grid faces unprecedented challenges. Power flow calculation, as a core tool for analyzing the operating status of the power grid, has its convergence a prerequisite for assessing the static security of the system. In recent years, breakthroughs in artificial intelligence technology, especially in the reasoning capabilities of large language models, have provided new ideas for solving complex decision-making problems in power systems.

[0003] Currently, power flow regulation mainly employs two methods: traditional deep learning-based methods and general-purpose large language model-based methods. Traditional deep learning-based methods typically utilize deep neural networks (DNNs) to learn state-action mappings, or employ reinforcement learning algorithms such as soft actor-commentator (SAC) and proximal policy optimization (PPO) to train regulation strategies, attempting to establish a connection between input states and adjustment actions through data-driven approaches. General-purpose large language model-based methods, on the other hand, attempt to introduce general-purpose large models such as GPT-4, leveraging their powerful general reasoning capabilities to directly solve power system decision-making problems through zero-shot or few-shot hints, hoping to use the language understanding capabilities of large models to handle complex power grid regulation tasks.

[0004] However, traditional deep learning methods suffer from significant "black box" characteristics, failing to explain decision-making logic and making it difficult to gain the trust of power system operators. Furthermore, due to their fixed model structures (e.g., a fixed number of nodes in a fully connected network), they exhibit poor generalization ability, making them difficult to apply directly to systems of varying sizes. They also suffer from low training efficiency in sparse reward environments and lack the ability to handle complex logical reasoning tasks. On the other hand, methods based on general-purpose large language models suffer from insufficient domain knowledge due to the lack of power system-specific content in their pre-training corpora. This results in inconsistent output quality under zero-shot or few-shot prompts, leading to output instability. Moreover, models with extremely large parameter sets are costly to deploy, have slow inference speeds, and closed-source models cannot fine-tune parameters for specific tasks, limiting their efficient application in real-world power systems. Summary of the Invention

[0005] This invention provides a power flow regulation generation method and apparatus based on supervised fine-tuning and reinforcement learning, which addresses the shortcomings of existing technologies in lacking deep adaptability and controllability for specific tasks of power systems. It enables the model to deeply perceive the physical mechanisms and dynamic boundaries of the power system, preventing it from generalizing and failing when facing highly difficult regulation tasks. It ensures that the model can autonomously explore better strategies in a multi-dimensional action space, significantly improving the accuracy, robustness, generalization ability, and reliability of the model's power flow regulation.

[0006] This invention provides a power flow regulation generation method based on supervised fine-tuning and reinforcement learning, comprising: acquiring power system state observation data; inputting the power system state observation data into a power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model; wherein, the power flow regulation generation model is obtained by supervising fine-tuning an initial power flow regulation generation model trained using a first observation training sample, and then training the model obtained by reinforcement learning using a population relative policy optimization algorithm; the first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

[0007] According to the present invention, a power flow regulation generation method based on supervised fine-tuning and reinforcement learning includes the following steps before inputting power system state observation data into the power flow regulation generation model: acquiring observation training data and corresponding regulation labels, and constructing a first observation training sample based on the observation training data using positive perturbation sampling and trajectory reversal; using the first observation training sample and regulation labels, supervising fine-tuning the pre-trained initial power flow regulation generation model according to a preset iteration strategy; the preset iteration strategy includes: for each iteration step, inputting the first observation training sample into the initial power flow regulation generation model to obtain the regulation training result output by the initial power flow regulation generation model; determining the negative log-likelihood loss function based on the regulation training result and the corresponding regulation label; determining the first gradient based on the negative log-likelihood loss function, and updating the model parameters of the initial power flow regulation generation model using the first gradient; inputting the first observation training sample into the updated initial power flow regulation generation model, and repeating the iteration until the negative log-likelihood loss function converges, ending the training, and obtaining the supervised fine-tuned model.

[0008] According to the present invention, a power flow regulation generation method based on supervised fine-tuning and reinforcement learning is provided. Based on observational training data, a first observational training sample is constructed using forward perturbation sampling and trajectory reversal. The method includes: selecting initially converged observational training data; randomly selecting a control variable and applying a random perturbation to obtain observational perturbation data; the control variable is used to characterize the adjustment parameters of the adjustable devices at the corresponding nodes in the observational training data; based on the observational perturbation data, the convergence state of the power flow calculation is determined using a power flow model; the power flow model is constructed based on the operating state of the power grid; when the power flow calculation is determined to be converged, the corresponding successful state is recorded. The process involves obtaining successful observation disturbance data, randomly selecting control variables and applying random disturbances, and reusing the power balance model to determine the convergence state of the power flow calculation; or, if the power flow calculation is determined to be non-convergent, recording the corresponding failure state, obtaining failed observation disturbance data, and stopping the disturbance; applying random disturbances to the failed observation disturbance data according to a preset number of disturbance steps to generate non-convergent state data, and combining this with the successful observation disturbance data obtained from the disturbances prior to the failed observation disturbance data to obtain forward trajectory data; reversing the forward trajectory data, and combining the state transitions between adjacent data in the reversed trajectory to construct the corresponding first observation training sample.

[0009] According to the present invention, a power flow regulation generation method based on supervised fine-tuning and reinforcement learning determines the convergence state of power flow calculation based on observed disturbance data and a power flow model. The method includes: obtaining active power residuals and reactive power residuals using a preset iterative algorithm based on the observed disturbance data and the power flow model; determining that the convergence state of the power flow calculation is converged when both the active power residuals and reactive power residuals are less than corresponding preset thresholds; or determining that the convergence state of the power flow calculation is non-converged when the active power residuals and / or reactive power residuals are greater than or equal to corresponding preset thresholds. The forward trajectory data is reversed, and the state transitions between adjacent data points in the reversed trajectory are combined to construct the corresponding first observation training sample. This includes: reversing the forward trajectory data to obtain the reversed trajectory; determining the control command based on the difference between adjacent states in the reversed trajectory; and constructing the corresponding first observation training sample based on the previous state in the corresponding adjacent state.

[0010] According to the present invention, a power flow regulation generation method based on supervised fine-tuning and reinforcement learning is provided. The method inputs a first observation training sample into a pre-trained initial power flow regulation generation model to obtain the regulation training result output by the initial power flow regulation generation model. The method includes: inputting the first observation training sample into the pre-trained initial power flow regulation generation model; performing feature encoding and embedding processing on the first observation training sample to map it into a high-dimensional vector representation; based on the high-dimensional vector representation, capturing the correlation and dependency between power system state features and control variables using a multi-head self-attention mechanism to obtain a hidden state representation that incorporates contextual information; and based on an autoregressive generation method, predicting the probability distribution of the next term step-by-step according to the hidden state representation to generate the regulation training result.

[0011] According to the present invention, a power flow regulation generation method based on supervised fine-tuning and reinforcement learning is provided. Based on an autoregressive generation method, the method predicts the probability distribution of the next word element step-by-step according to the hidden state representation to generate regulation training results. The method includes: for each time step, obtaining the input word element of the corresponding time step, performing embedding mapping and position encoding on the input word element to obtain the word element vector of the current step; based on a multi-head self-attention mechanism, decoding the word element vector of the current step using the hidden state representation, and predicting the log probability of each word element in the vocabulary space; sampling or greedy search based on the log probability to determine the target word element of the corresponding time step; concatenating the target word element into the prefix output sequence as the input word element of the next time step, and repeating the above steps until an end marker is generated to obtain a regulation training result sequence containing intermediate result thought chains and regulation instructions.

[0012] According to the present invention, a power flow regulation generation method based on supervised fine-tuning and reinforcement learning includes the following steps after obtaining the supervised fine-tuned model: for each iteration step, based on the previously acquired observation training data, using the supervised fine-tuned model to obtain the regulation result; the regulation result includes the prediction result of the target quantity; based on each prediction result, determine the corresponding reward, and based on the corresponding reward of each prediction result, obtain the corresponding advantage function and determine the objective function; based on the objective function, backpropagate to determine the second gradient, and use the second gradient to update the supervised fine-tuned model, repeating the iteration until the objective function converges or the maximum number of iterations is reached, thereby obtaining the power flow regulation generation model.

[0013] According to the present invention, a power flow regulation generation method based on supervised fine-tuning and reinforcement learning is provided. The prediction results include regulation commands, corresponding response labels for regulation commands, intermediate result thought chains, and corresponding thought chain labels for intermediate result thought chains. Based on each prediction result, a corresponding reward is determined, including: verifying the completeness, correct order, data format, and data parsability of the thought chain labels and result labels in each prediction result; assigning values ​​to the corresponding prediction results based on the verification results and a preset binary reward to obtain a format reward value for each prediction result; performing power flow simulation based on each prediction result to determine the power flow convergence reward value for each prediction result; determining the boundary conditions of the control variables corresponding to the regulation commands in each prediction result based on each prediction result, and determining corresponding penalties or safety rewards based on the boundary conditions of each control variable to obtain a boundary reward value for each prediction result; and obtaining the corresponding reward for each prediction result based on the format reward value, power flow convergence reward value, and boundary reward value. Based on the rewards corresponding to each prediction result, the corresponding advantage function is obtained, and the objective function is determined, including: obtaining the importance sampling ratio using exponential operations based on the log probabilities of each prediction result obtained from observed training data under the previous model parameters; limiting the importance sampling ratio according to a preset pruning gradient threshold to obtain the pruning result; determining the mean and standard deviation of the reward based on the rewards corresponding to each prediction result, and normalizing the rewards of each prediction result according to a preset group size to determine the advantage function of each prediction result; the preset group size is used to characterize the number of prediction results generated by the model; determining the relative entropy KL divergence between the current model and the supervised fine-tuned model, and determining the KL divergence penalty term according to a preset penalty coefficient; and obtaining the objective function based on the importance sampling ratio, the pruning result, the advantage function of each prediction result, and the KL divergence penalty term.

[0014] According to the present invention, a power flow regulation generation method based on supervised fine-tuning and reinforcement learning is provided. The initial number of power flow regulation generation models is at least two. After obtaining the power flow regulation generation models, the method includes: acquiring the policy entropy of each power flow regulation generation model on a validation dataset, where the policy entropy measures the uncertainty of the model's output distribution; analyzing the exploration capability and convergence characteristics of each model based on its policy entropy; and selecting a target power flow regulation generation model from the at least two power flow regulation generation models based on its exploration capability and convergence characteristics, for use as input to power system state observation data.

[0015] This invention also provides a power flow regulation generation device based on supervised fine-tuning and reinforcement learning, comprising: a data acquisition module for acquiring power system state observation data; and a regulation generation module for inputting the power system state observation data into a power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model. The power flow regulation generation model is obtained by supervising the fine-tuning of an initial power flow regulation generation model trained using a first observation training sample, and then training the model obtained through reinforcement learning using a population relative policy optimization algorithm. The first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the power flow regulation generation method based on supervised fine-tuning and reinforcement learning as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the power flow regulation generation method based on supervised fine-tuning and reinforcement learning as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the power flow regulation generation method based on supervised fine-tuning and reinforcement learning as described above.

[0019] The present invention provides a power flow regulation generation method and apparatus based on supervised fine-tuning and reinforcement learning. By acquiring state observation data, it provides the subsequent model with raw inputs reflecting the current grid topology, load level, and equipment status, ensuring that regulation generation is based on real operating conditions. By constructing samples of forward disturbance sampling and trajectory reversal, it simulates the reverse process from fault state to normal state, thereby explicitly generating high-value samples covering the system's critical state, fault evolution, and recovery process. This enables the model to deeply perceive the physical mechanism and dynamic boundary of the power system, giving the model a deep adaptability to the complex and variable operating conditions of the power system. This prevents the model from generalizing and failing when facing high-difficulty regulation tasks. Furthermore, it utilizes group relative strategy optimization (GRPO) for reinforcement learning training, enabling the model to autonomously explore better strategies in a multi-dimensional action space. This ensures that the regulation generation results output by the model are physically convergent and compliant with safety regulations, thereby effectively addressing thorny issues such as voltage exceeding limits or power flow congestion, and significantly improving the model's power flow regulation accuracy, robustness, generalization ability, and reliability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the power flow regulation generation method based on supervised fine-tuning and reinforcement learning provided by the present invention. Figure 2 This is a schematic diagram of the power flow regulation generation device based on supervised fine-tuning and reinforcement learning provided by the present invention. Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Figure 1 This is a flowchart illustrating the power flow regulation generation method based on supervised fine-tuning and reinforcement learning provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: S11, acquire power system status observation data; S12, input the power system state observation data into the power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model; wherein, the power flow regulation generation model is obtained by supervising the fine-tuning of the initial power flow regulation generation model after training using the first observation training sample, and by using the group relative strategy optimization algorithm to train the model obtained by reinforcement learning; the first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

[0024] It should be noted that the following describes in detail the power flow regulation generation method based on supervised fine-tuning and reinforcement learning of the present invention.

[0025] Step S11: Obtain power system status observation data.

[0026] It should be noted that power system state observation data includes system description, operating status, sensitivity information, constraint information, and convergence status. Specifically: the system description includes topology information and node types. Topology information includes the number of nodes and branch connections. The number of nodes characterizes the number of buses in the power grid, and branch connections characterize the connection information between nodes, such as how two nodes are connected via transmission lines or transformers. Node types include active power voltage nodes, reactive power voltage nodes, and slack nodes. The operating status includes the voltage amplitude V, voltage phase angle θ, active power P, and reactive power Q of each node. Sensitivity information includes the residuals of the power flow equations and the sensitivity of each control variable. The residuals of the power flow equations can be determined based on the difference between the active power and reactive power values ​​obtained from the power flow simulation and the corresponding given power values. The control variables are used to characterize the adjustment parameters of adjustable equipment in the power system, including voltage, active power, reactive power, etc. The constraint information includes the upper and lower limits of each control variable. The convergence state is used to characterize whether the power flow calculation has converged, that is, whether the power imbalance is less than the corresponding threshold. The power imbalance is used to characterize the difference between the active power and reactive power values ​​obtained from the power flow simulation and the corresponding given power values.

[0027] In addition, after acquiring the power system state observation data, the power system state observation data is converted into a text description by the observer. The specific design can be designed according to the actual design requirements, and no further limitations are made here.

[0028] For example, suppose the power system is an IEEE 39-bus system with 39 bus nodes. The slack bus is bus number 30, with a voltage of 0.982 pu. Nodes 29 and 31-39 are PV nodes, and nodes 1-28 are PQ nodes. The convergence state is non-convergence, and the maximum power imbalance is 30883.26 pu. The current voltage of node 31 is 0.9841 pu, with high voltage sensitivity, an upward adjustment margin of 7.71%, and a downward adjustment margin of -2.34%. The current voltage of node 32 is 0.9790 pu, with high voltage sensitivity, an upward adjustment margin of 6.30%, and a downward adjustment margin of -1.87%. The corresponding text representation is as follows: # System Description Total buses: 39 - Slack bus: 30 (V=0.982 pu) - PV buses: 29, 31, 32, 33, 34, 35, 36, 37, 38, 39 - PQ buses: 1-28 # Current Status - Convergence: NO - Max mismatch: 30883.26 pu # PV Buses Detail Bus 31: V=0.9841, V_sensitivity=high, up_limit=7.71%, down_limit=-2.34% Bus 32: V=0.9790, V_sensitivity=high, up_limit=6.30%, down_limit=-1.87% ... Step S12: Input the power system state observation data into the power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model; wherein, the power flow regulation generation model is obtained by supervising the fine-tuning of the initial power flow regulation generation model after training using the first observation training sample, and by using the group relative strategy optimization algorithm to train the model obtained by reinforcement learning; the first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

[0029] It should be added that the principle of inputting power system state observation data into the power flow regulation generation model can be referred to in the following text on inputting the first observation training sample into the pre-trained initial power flow regulation generation model, and will not be repeated here. In addition, the regulation generation result includes regulation generation instructions and an intermediate reasoning chain. The intermediate reasoning chain includes the reasoning process of system state diagnosis, selection of controllable variables, and selection of regulation strategies.

[0030] In an optional embodiment, before inputting power system state observation data into the power flow regulation generation model, the process includes: acquiring observation training data and corresponding regulation labels, and constructing a first observation training sample based on the observation training data using positive perturbation sampling and trajectory reversal; using the first observation training sample and regulation labels, performing supervised fine-tuning on the pre-trained initial power flow regulation generation model according to a preset iteration strategy; the preset iteration strategy includes: for each iteration step, inputting the first observation training sample into the initial power flow regulation generation model to obtain the regulation training result output by the initial power flow regulation generation model; determining a negative log-likelihood loss function based on the regulation training result and corresponding regulation labels; determining a first gradient based on the negative log-likelihood loss function, and updating the model parameters of the initial power flow regulation generation model using the first gradient; inputting the first observation training sample into the updated initial power flow regulation generation model, and repeating the iteration until the negative log-likelihood loss function converges, ending the training, and obtaining the supervised fine-tuned model.

[0031] It should be noted that by constructing training samples using positive perturbation sampling and trajectory reversal mechanisms, sample trajectories with rich variations and conforming to physical laws can be generated from limited observation data. This effectively expands the diversity of the training dataset, providing sufficient data support for the refined management of the model. Furthermore, the pre-trained model is used as a base for fine-tuning, so that the fine-tuned model not only inherits the ability to understand the general characteristics of the power system, but also adapts to specific power flow control tasks through targeted training, thereby significantly improving training efficiency and the upper limit of model performance. In addition, by using the negative log-likelihood loss function to calculate the gradient, the difference between the predicted distribution and the true label distribution can be accurately measured from a probabilistic perspective, guiding the model parameters to be updated in the direction of maximizing the likelihood probability, thereby effectively improving the prediction accuracy and robustness of the model in complex power flow environments.

[0032] Furthermore, the negative log-likelihood loss function is expressed as: in, This represents the negative log-likelihood loss function; Indicates model parameters; This represents the total number of training samples for the first observation, which can be 1000-5000. The specific number of samples can be constructed according to the actual design requirements, and no further limitation is made here. This represents the i-th first observation training sample; This represents the control label of the i-th first observation training sample, which is obtained in the following text and will not be repeated here; This represents the length of the i-th first observation training sample, i.e., the number of lexical units.

[0033] Furthermore, the initial power flow regulation generation model can be updated using the Adam optimizer, and the corresponding update is expressed as: in, This represents the updated model parameters; Indicates the current model parameters; This represents the learning rate, for example, it can be set to 1×10. -5 ; This represents the first moment estimate of the first gradient. This represents the second-order moment estimate (variance) of the first gradient. This represents a numerically stable term, typically 10. -8 .

[0034] Specifically, the initial convergent observation training data is selected, a control variable is randomly selected, and a random perturbation is applied to obtain observation perturbation data; the control variable is used to characterize the adjustment parameters of the adjustable devices at the corresponding nodes in the observation training data; based on the observation perturbation data, the convergence state of the power flow calculation is determined using the power flow model; the power flow model is constructed based on the operating state of the power system grid; when the power flow calculation is determined to be convergent, the corresponding successful state is recorded to obtain successful observation perturbation data, and the control variable is randomly selected again and a random perturbation is applied, and the power balance model is reused to determine the convergence state of the power flow calculation; or, when the power flow calculation is determined to be non-convergent, the corresponding failure state is recorded to obtain failed observation perturbation data, and the perturbation is stopped; according to the preset perturbation steps, random perturbation is applied to the failed observation perturbation data to generate non-convergent state data, and combined with the successful observation perturbation data obtained from perturbing the failed observation perturbation data earlier, positive trajectory data is obtained; the positive trajectory data is reversed, and the state transition between two adjacent data in the reversed trajectory is combined to construct the corresponding first observation training sample.

[0035] It should be noted that by using the power flow model to verify the convergence state after disturbance in real time, the feasible domain boundary for maintaining stable system operation can be accurately identified, and physically non-convergent invalid states can be automatically filtered out. This ensures that the constructed trajectory data is always within the physical constraints of safe power system operation. Furthermore, by continuing to apply disturbances at failure points, i.e. non-convergence points, non-convergence sequences are generated and combined with previous successful data, thus constructing a complete evolution path from normal steady state to unstable critical state and then to fault state. By reversing the forward trajectory, the recovery process of gradually returning from the fault state to the normal state can be simulated. This reverse state transition information can effectively train the model to learn how to pull the system back from the edge of instability to the safe region, thereby significantly improving the model's control capability and recovery strategy generation level in the face of sudden faults.

[0036] It should be added that applying a random perturbation is represented as: in, represents the control variable for which random disturbance is applied when the power flow calculation fails to converge initially, and k represents the kth random selection; This indicates an assignment operation; Let represent a random variable that follows a uniform distribution, and its range of values ​​is limited to . between.

[0037] In addition, when the initial determination of non-convergence in power flow calculation is obtained, the corresponding observational disturbance failure data is also obtained. Then, using the aforementioned random perturbation, depth-non-convergent samples are generated. , n This indicates the preset number of perturbation steps. The preset number of perturbation steps can be configured according to actual design requirements and prior experience. For example, the preset number of perturbation steps is less than or equal to 7, and no further restrictions are imposed here.

[0038] Furthermore, based on the observed disturbance data, the convergence state of the power flow calculation is determined using the power flow model, including: obtaining the active power residual and reactive power residual using a preset iterative algorithm based on the observed disturbance data and the power flow model; determining that the convergence state of the power flow calculation is converged when both the active power residual and the reactive power residual are less than the corresponding preset thresholds; or, determining that the convergence state of the power flow calculation is non-converged when the active power residual and / or the reactive power residual are greater than or equal to the corresponding preset thresholds.

[0039] In addition, the forward trajectory data is reversed, and the state transitions between adjacent data in the reversed trajectory are combined to construct the corresponding first observation training sample, including: reversing the forward trajectory data to obtain the reversed trajectory; determining the control command based on the difference between adjacent states in the reversed trajectory, and constructing the corresponding first observation training sample in combination with the previous state in the corresponding adjacent state.

[0040] It should be added that the positive trajectory data is represented as follows: Accordingly, the reverse trajectory is represented as: In addition, the state transition between two adjacent data points can be determined using a power flow simulation simulator based on the current state and action in the reverse trajectory.

[0041] It is important to note that since the forward perturbation process does not have a thought chain of intermediate results (CoT), a large-scale inference model, such as Qwen3-32B, is required to generate the CoT. Specifically, this involves: constructing a CoT containing state descriptions and target control instructions; calling a large model to generate the CoT, requiring an explanation of why the control strategy was chosen; manually or automatically checking the quality of the CoT and removing unreasonable samples; and concatenating the CoT and control instructions to form a complete control label.

[0042] Furthermore, the first observation training sample is input into the pre-trained initial power flow regulation generation model to obtain the regulation training result output by the initial power flow regulation generation model. This includes: inputting the first observation training sample into the pre-trained initial power flow regulation generation model, performing feature encoding and embedding processing on the first observation training sample, and mapping the first observation training sample into a high-dimensional vector representation; based on the high-dimensional vector representation, capturing the correlation and dependency between power system state features and control variables based on a multi-head self-attention mechanism to obtain a hidden state representation that integrates contextual information; and based on an autoregressive generation method, predicting the probability distribution of the next term step-by-step according to the hidden state representation to generate the regulation training result.

[0043] It should be noted that the model maps complex observation samples into high-dimensional vectors to transform discrete equipment parameters and continuous state variables into a unified feature space representation, preserving the inherent correlation of the data. It also utilizes a multi-head self-attention mechanism to monitor the dynamic interaction between the state characteristics and control variables of different nodes in the power network in parallel, effectively identifying the correlation patterns that play a key role in power flow regulation, thereby accurately grasping the overall operating status of the system. Through word-by-word prediction using an autoregressive approach, the model can simulate the decision-making process of human experts, generating control instructions step by step in a logical order, ensuring the temporal consistency and rationality of the generated control strategies, and improving the interpretability and credibility of the model output.

[0044] It should be added that the initial power flow regulation generation model output regulation training results are expressed as follows: in, This represents the model parameters; there are approximately 7 billion parameters for the 7B model. Representation model; y The tuning training result of the model output is represented as ( This includes the CoT (Content of Thought) and regulatory instructions; x The input to the model, i.e., the first observed training sample, is represented as ( ; T This indicates the length of the output sequence, which is determined by the model itself.

[0045] In addition, the initial power flow regulation generation model adopts the Transformer architecture, the core of which is a multi-head self-attention mechanism, represented as: in, This represents the associated dependencies captured based on the multi-head self-attention mechanism; These represent the query vector, key vector, and value vector, respectively, which are obtained from the first observed training sample through linear transformation. The dimension of the key vector is typically 64 or 128. This represents the normalization function, ensuring that the sum of the attention weights is 1.

[0046] Furthermore, based on the autoregressive generation method, the probability distribution of the next word is predicted step-by-step according to the hidden state representation to generate the regulation training result. This includes: for each time step, obtaining the input word of the corresponding time step, performing embedding mapping and position encoding on the input word to obtain the word vector of the current step; based on the multi-head self-attention mechanism, decoding the word vector of the current step using the hidden state representation, and predicting the log probability of each word in the vocabulary space; sampling or greedy search based on the log probability to determine the target word of the corresponding time step; concatenating the target word into the prefix output sequence as the input word of the next time step, repeating the above steps until an end marker is generated, to obtain a regulation training result sequence containing the intermediate result thought chain and regulation instructions.

[0047] It should be noted that the tuning training results include the CoT (Cognitive Chain of Intermediate Results), the corresponding CoT labels, the tuning instructions, and the corresponding result labels. CoT inference decomposes complex problems into multiple sub-problems by inserting intermediate inference steps into the output. Research shows that the Transformer model can learn optimization algorithms similar to gradient descent through CoT. For example, the tuning training results can be represented as: <think> Step 1: Analyze the current system status - Power imbalance: 30883 pu - Jacobian matrix condition number: 1.05e20 (highly ill-conditioned) - Key node identification: Node 30 (balanced node, sensitivity 1.05e20) Step 2: Screening for controllable variables - Node 29: PV node, voltage 1.0499, upward adjustment margin 0.962% - Node 31: PV node, voltage 0.9841, upward adjustment margin 7.7126% - Node 32: PV node, voltage 0.9790, upward adjustment margin 6.2976% Step 3: Select a control strategy Based on sensitivity analysis, nodes 31 and 32 have considerable room for adjustment. - The overall system voltage is too low; a voltage boosting strategy is adopted. - Decision: Simultaneously increase the voltage of nodes 31 and 32 to the upper limit. < / think> <answer>[{"bus_id": 31, "adj_pct": 7.7126, "var_type": "V"}, {"bus_id": 32, "adj_pct": 6.2976, "var_type": "V"}]< / answer> in, <think>< / think> This represents a pair of thought chain labels; <answer>< / answer> This represents a pair of result labels; bus_id represents the node number to be adjusted (1 to n); adj_pct represents the adjustment percentage, with positive values ​​indicating an increase and negative values ​​indicating a decrease; var_type represents the variable type, including voltage (V), active power (P), and reactive power (Q).

[0048] It should be noted that CoT can be used for information integration to combine numerical observations (voltage, power, etc.) with domain knowledge (PV / PQ node types, constraint limits, etc.), and through the logical chain of "if...then...", it can simulate the decision-making process of experts to perform logical reasoning, and introduce intermediate conclusions (such as "node 31 is adjustable") to support the final decision, so as to generate text that can be understood by humans and improve interpretability.

[0049] in addition, <answer>< / answer> The information between them is the action space, which is a list of control instructions. It adopts JSON or other specified formats. The action space can adjust 1 to m variables at the same time (m is the number of PV nodes), and adj_pct can take values ​​continuously from the real number set and is subject to the physical limit of the control variable, for example, V ∈ [0.9, 1.1] pu.

[0050] It should be added that, after supervised fine-tuning (SFT), the initial power flow regulation generation model can output correct values. <think>< / think> and <answer>< / answer> The labels and outputs can be successfully parsed into valid control commands, identifying key nodes, determining the adjustment direction (increasing or decreasing voltage), and achieving a 50% convergence success rate on the training system.

[0051] In an optional embodiment, the base model for the initial power flow regulation generation model can be the general-purpose language model Qwen2.5-7B with 7 billion parameters; the fine-tuning method uses LoRA low-rank adaptation, training only about 0.5% of the parameters; the rank of the low-rank matrix LoRA is 32; the scaling factor α of the low-rank matrix LoRA is 64; and the learning rate of the Adam optimizer is 1×10⁻⁶. -5 The number of samples used in each update, i.e., the batch size, is 8; the number of training epochs, i.e., the number of times the training set is traversed, is 3-5 epochs; the maximum sequence length, i.e., the maximum number of words in the input and output, is 16384; and the number of training samples, i.e., the total number of samples in the first observation training sample generated, is 2000.

[0052] In an optional embodiment, after obtaining the supervised fine-tuned model, the process includes: for each iteration step, using the supervised fine-tuned model based on previously acquired observation training data, obtaining a control result; the control result includes a prediction result of the target quantity; determining a corresponding reward based on each prediction result, and obtaining a corresponding advantage function based on the corresponding reward for each prediction result, and determining an objective function; determining a second gradient based on the objective function through backpropagation, and updating the supervised fine-tuned model using the second gradient, repeating the iteration until the objective function converges or reaches the maximum number of iterations, thereby obtaining a power flow control generation model.

[0053] It should be noted that by generating multiple prediction results for the target number during the inference stage, various possible control paths are simulated, providing rich candidate samples for subsequent evaluation. This avoids the model getting trapped in a single decision-making mode with local optima. Furthermore, by introducing a reward mechanism to calculate the advantage function, the superiority of the current strategy relative to the average level is accurately quantified. This not only focuses on the correctness of the control results but also guides the model to evolve towards a better control strategy based on the level of reward, thereby significantly improving the model's decision-making quality under complex constraints. Additionally, by iteratively calculating gradients and updating the model, reinforcement learning mechanisms are used to further calibrate the model's decision boundaries, enabling it to adaptively learn the dynamic response characteristics of the power system. This ensures that the final generated model has higher response accuracy and generalization ability when facing complex and ever-changing power flow control tasks.

[0054] Specifically, the prediction results include control instructions, corresponding response labels for the control instructions, intermediate result thought chains, and corresponding thought chain labels for the intermediate result thought chains. Based on each prediction result, corresponding rewards are determined, including: verifying the completeness, correct order, data format, and parsability of the thought chain labels and result labels in each prediction result; assigning values ​​to the corresponding prediction results based on the verification results and a preset binary reward, obtaining the format reward value for each prediction result; performing power flow simulation based on each prediction result to determine the power flow convergence reward value for each prediction result; determining the boundary conditions of the control variables corresponding to the control instructions in each prediction result, and determining the corresponding penalties or safety rewards based on the boundary conditions of each control variable, obtaining the boundary reward value for each prediction result; and obtaining the corresponding reward for each prediction result based on the format reward value, power flow convergence reward value, and boundary reward value.

[0055] It should be added that the reward is represented as follows: in, Indicates a reward; Represents the format of the reward value; This represents the convergence reward value. This represents the boundary reward value.

[0056] Furthermore, the completeness, correct order, data format, and parsability of the thought chain tags and result tags in the corresponding prediction results are verified, including: determining whether the prediction results contain only one pair of thought chain tags, whether the control results contain only one pair of result tags, whether the positional relationship between the thought chain tags and result tags conforms to the preset positional relationship, whether the JSON format in the result tags is correct and can be parsed, and whether each object in the control instructions contains a target field, which includes the node number to be adjusted (bus_id), the adjustment percentage (adj_pct), and the variable type (var_type). If all of the above are met, the verification passes; otherwise, the verification fails.

[0057] Furthermore, format rewards are used to characterize the completeness, correct order, and parsability of the data format of the thought chain labels and outcome labels in the regulatory results. Represented as: Furthermore, the power flow convergence reward value is a tiered reward determined based on whether the power flow calculation has converged, whether numerical errors have occurred, or the degree of improvement in imbalance. Specifically, based on each prediction result, power flow simulation is performed to determine the power flow convergence reward value for each prediction result, including: for each prediction result, power flow simulation is performed separately; if the corresponding power flow calculation convergence state is determined to be converged, then the power flow convergence reward value is assigned according to the preset maximum reward value; or, if the corresponding power flow calculation convergence state is determined to be non-converged, based on the occurrence of serious errors in the power flow calculation, such as matrix singularity or numerical overflow, the power flow convergence reward value is assigned according to the first preset negative reward value; or, based on the absence of serious errors in the power flow calculation, the degree of improvement in the power imbalance is determined according to the current power imbalance and the previous power imbalance; if the current power imbalance increases or remains unchanged compared to the previous power imbalance, then the power flow convergence reward value is assigned according to the second preset negative reward value; otherwise, the power flow convergence reward value is determined according to the degree of improvement. The aforementioned tiered reward mechanism ensures that regardless of the initial imbalance, the reward remains the same for each order of magnitude reduction, thus preventing large imbalance samples from excessively dominating training.

[0058] It should be noted that the first and second preset negative reward values ​​can be designed according to actual design requirements and prior experience. For example, the first preset negative reward value can be -1 and the second preset negative reward value can be -0.5. No further restrictions are imposed here.

[0059] Furthermore, when the corresponding power flow calculation is in a non-convergent state and no serious errors occur in the power flow calculation, the power flow convergence reward value is... Represented as: in, The degree of improvement is indicated by the difference between the previous power imbalance and the current power imbalance. This indicates the prior power imbalance. This indicates the current power imbalance. The determination of the power imbalance can be referred to in the previous text, and will not be repeated here.

[0060] In addition, boundary reward value Represented as: in, This indicates the total number of control variables involved in this adjustment; This represents the adjusted value of the k-th control variable determined based on the prediction results; This indicates a reward for a safe action, and can generally be 1. Represents an exponential function, when hour, ,otherwise ; This represents the penalty coefficient for exceeding the boundary, which is typically 5. This indicates the out-of-bounds status of the k-th control variable; 0 indicates it is within the bounds, and greater than 0 indicates it is out of bounds. and These represent the upper and lower limits allowed for the k-th control variable, respectively. For example, the voltage can be 0.9-1.1 pu.

[0061] For example, if all control variables are within the bounds when K=3, that is... ,but If one of the control variables exceeds the limit by 10%, =0.1, then If all control variables are severely out of bounds, that is... 0.5, then .

[0062] Furthermore, the objective function is constructed based on maximizing the relative advantage of the group, using a detection mechanism to limit the policy update magnitude, and incorporating a KL divergence penalty term. Specifically, based on the reward corresponding to each prediction result, the corresponding advantage function is obtained, and the objective function is determined, including: based on each prediction result, combined with the log probability of each prediction result obtained from the observed training data under the previous model parameters, using exponential calculation to obtain the importance sampling ratio; based on a preset pruning gradient threshold, the importance sampling ratio is limited to obtain the pruning result; based on the reward corresponding to each prediction result, the reward mean and reward standard deviation are determined, and combined with a preset group size, the reward of each prediction result is normalized to determine the advantage function of each prediction result; the preset group size is used to characterize the number of prediction results generated by the model; the relative entropy KL divergence between the current model and the supervised fine-tuned model is determined, and combined with a preset penalty coefficient, the KL divergence penalty term is determined; based on the importance sampling ratio, the pruning result, and the advantage function of each prediction result, combined with the KL divergence penalty term, the objective function is obtained.

[0063] It should be added that the objective function is expressed as: in, Represent the objective function; Indicates model parameters; Indicates the model input; Represents the probability distribution of the model input; This represents the i-th prediction result output by the model; This indicates the model strategy in the previous version; G represents the group size, i.e., the number of objectives. Indicates the importance sampling ratio; This represents the advantage function corresponding to the i-th prediction result; This represents the clipping function and the corresponding clipping result; This indicates the preset clipping gradient threshold; This represents the KL divergence penalty coefficient; This represents the KL divergence penalty term; Indicates the current model policy; This represents the model after supervised fine-tuning.

[0064] Furthermore, importance sampling ratio Represented as: in, Indicates the current model; This refers to the model in the previous version; This indicates the current iteration step.

[0065] The clipping function is expressed as: It should be noted that by limiting the ratio to Within the range, to prevent performance crashes caused by excessively large policy updates.

[0066] The advantage function corresponding to the i-th prediction result Represented as: in, This represents the reward for the i-th prediction result; This represents the average reward. This represents the standard deviation of the reward. The advantage function measures the superiority of the i-th prediction relative to the group level.

[0067] The KL divergence penalty term is expressed as: in, This indicates the model strategy for freezing after supervised fine-tuning; This represents the current model strategy. The penalty term is used to constrain the current model from deviating from the reference model. (Usually the model after supervised fine-tuning and SFT) is too far away.

[0068] Furthermore, after obtaining the objective function, the second gradient determined by backpropagation is: The Adam optimizer is used to update the supervised fine-tuned model, and the corresponding update is expressed as: in, This represents the learning rate.

[0069] In one optional embodiment, the base model for training the Group Relative Policy Optimization (GRPO) algorithm is a supervised fine-tuned model, and training begins with the supervised fine-tuned model. The reference model is the supervised fine-tuned model in a frozen state, and the learning rate can be 1×10⁻⁶. -5 G can be 4-8; batch size B can be 128; preset clipping gradient threshold. It can be 0.2; KL divergence penalty coefficient The value can be 0.001; the training iterations can be 60, meaning the maximum number of iterations is 60.

[0070] Experiments have shown that the success rate of the model with only supervised fine-tuning is 58.16%. The success rate of the 7B model is as high as 74.35% and the success rate of the 14B model is as high as 83.37% after supervised fine-tuning and reinforcement learning GRPO training. It can be seen that the model performance can be significantly improved by performing reinforcement learning GRPO training after supervised fine-tuning.

[0071] In one optional embodiment, the initial number of power flow regulation generation models is at least two. After obtaining the power flow regulation generation models, the process includes: acquiring the policy entropy of each power flow regulation generation model on the validation dataset, where the policy entropy is used to measure the uncertainty of the model output distribution; analyzing the exploration capability and convergence characteristics of each model based on the policy entropy of each power flow regulation generation model; and selecting a target power flow regulation generation model from the at least two power flow regulation generation models based on the exploration capability and convergence characteristics, for use as input to power system state observation data.

[0072] It should be added that the policy entropy is expressed as: in, Represents policy entropy; D represents the model strategy; D represents the reference dataset or expert data distribution, used to provide model input. x With the corresponding model output y Distribution; This indicates the policy model at the current time step. Generating lexical units The probability of each word. In addition, at high entropy, H≈10, the model is very uncertain about the choice of each word and tends to explore diverse outputs; at low entropy, H≈1, the model is very certain about the output and tends to generate fixed patterns.

[0073] Furthermore, based on exploratory capability and convergence characteristics, a target power flow regulation generation model is selected from at least two power flow regulation generation models, including: when the policy entropy is greater than a first preset threshold, the corresponding power flow regulation generation model is determined to have high exploratory capability and is suitable for scenarios with a large amount of training data; when the policy entropy is less than a second preset threshold, the corresponding power flow regulation generation model is determined to have low exploratory capability and is suitable for scenarios with a small amount of training data or scenarios requiring fast convergence; based on the sample data size or computational resource constraints of the current training task, a power flow regulation generation model with matching policy entropy characteristics is selected as the target power flow regulation generation model.

[0074] Furthermore, in the actual experiment, three types of models were selected for comparison based on their policy entropy at the initial stage of GRPO training. The three types of models were the standard model Qwen2.5-1.5B, the hybrid reasoning model Qwen3-8B, and the pure reasoning model DeepSeek-Qwen2.5-7B. Among them, the initial entropy of the standard model was about 8.5, with low initial rewards and high exploratory power, but limited basic capabilities; the initial entropy of the hybrid reasoning model was about 5.0, with moderate initial rewards, balancing exploration and utilization; and the initial entropy of the pure reasoning model was about 3.0, with high initial rewards, enabling rapid adaptation but insufficient exploration.

[0075] Experimental comparisons show that in the early stages of a task or when the amount of data is large, a standard model (high entropy) can be selected to ensure sufficient exploration; when the task is mature or the amount of data is small, an inference model (low entropy) can be selected to ensure rapid convergence; when computing resources are limited, a standard model with small parameters (such as 1.5B-3B) can be selected; and when performance requirements are high, a standard model with large parameters (such as 7B-14B) can be selected.

[0076] In addition, tests were conducted on the IEEE 39-node system where the training data was generated, and the results are shown in Table 1. It can be seen that model size and performance are positively correlated, conforming to the scaling law of language models. Furthermore, the 7B model surpasses traditional numerical RL methods, while also possessing interpretability and generalization ability. The 14B model achieves a success rate of 83.37%, approaching practical application level.

[0077] Table 1

[0078] In addition, the model was tested on systems not seen during training, specifically the IEEE 57-bus system. The test results are shown in Table 2.

[0079] Table 2

[0080] For the IEEE 118-bus system (large-scale system), the test results are shown in Table 3.

[0081] Table 3

[0082] As can be seen, the model still maintains a high success rate on unseen systems, verifying its zero-shot generalization ability. Furthermore, large-scale systems require a longer context window to support multi-round inference. By using the YaRN method to extend the context from 32K to 128K, the success rate is significantly improved (+10% to +24%).

[0083] In addition, the effectiveness of each module was verified through ablation experiments, and the results are shown in Table 4.

[0084] Table 4

[0085] It is evident that the NoCoT model performs significantly worse than the CoT model (50% vs 74%), making supervised fine-tuning (CoT) necessary. Based on CoT, GRPO further improves performance by 16 percentage points, demonstrating its effectiveness. Direct GRPO training is prone to crashing, while SFT provides stable initialization, which is crucial.

[0086] In summary, this invention provides raw input reflecting the current power grid topology, load level, and equipment status to subsequent models by acquiring state observation data. This ensures that the control generation is based on real operating conditions. By constructing samples of forward disturbance sampling and trajectory reversal, the reverse process from fault state to normal state is simulated, thereby explicitly generating high-value samples covering the system's critical state, fault evolution, and recovery process. This enables the model to deeply perceive the physical mechanisms and dynamic boundaries of the power system, giving it a deep adaptability to the complex and ever-changing operating conditions of the power system. This prevents the model from generalizing and failing when facing high-difficulty control tasks. Furthermore, by using group relative strategy optimization (GRPO) for reinforcement learning training, the model can autonomously explore better strategies in a multi-dimensional action space, ensuring that the control generation results output by the model are physically convergent and compliant with safety regulations. This effectively addresses challenging issues such as voltage exceeding limits or power flow congestion, significantly improving the model's power flow control accuracy, robustness, generalization ability, and reliability.

[0087] The power flow regulation generation device based on supervised fine-tuning and reinforcement learning provided by the present invention will be described below. The power flow regulation generation device based on supervised fine-tuning and reinforcement learning described below can be referred to in correspondence with the power flow regulation generation method based on supervised fine-tuning and reinforcement learning described above.

[0088] Figure 2 A schematic diagram of a power flow regulation generation device based on supervised fine-tuning and reinforcement learning is shown. The device includes: Data acquisition module 21 acquires power system status observation data; The regulation generation module 22 inputs the power system state observation data into the power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model. The power flow regulation generation model is obtained by supervising the fine-tuning of the initial power flow regulation generation model after training using the first observation training sample, and then using the group relative strategy optimization algorithm to train the model obtained by reinforcement learning. The first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

[0089] It should be noted that the specific principles of the embodiments of the present invention are the same as those of the method embodiments described above. For details, please refer to the method embodiments above. More detailed explanations will not be repeated here.

[0090] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the logic instructions in the memory 330 to execute a power flow regulation generation method based on supervised fine-tuning and reinforcement learning. This method includes: acquiring power system state observation data; inputting the power system state observation data into a power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model; wherein the power flow regulation generation model is obtained by supervised fine-tuning an initial power flow regulation generation model trained using a first observation training sample, and by reinforcement learning training the model obtained by supervised fine-tuning using a population relative policy optimization algorithm; the first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

[0091] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the power flow regulation generation method based on supervised fine-tuning and reinforcement learning provided by the above methods. The method includes: acquiring power system state observation data; inputting the power system state observation data into a power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model; wherein the power flow regulation generation model is obtained by supervising fine-tuning an initial power flow regulation generation model trained using a first observation training sample, and by using a population relative policy optimization algorithm to train the model obtained by supervising fine-tuning through reinforcement learning; the first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

[0093] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the power flow regulation generation method based on supervised fine-tuning and reinforcement learning provided by the above methods. The method includes: acquiring power system state observation data; inputting the power system state observation data into a power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model; wherein the power flow regulation generation model is obtained by supervising fine-tuning an initial power flow regulation generation model trained using a first observation training sample, and by using a population relative policy optimization algorithm to train the model obtained by supervising fine-tuning through reinforcement learning; the first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A power flow regulation generation method based on supervised fine-tuning and reinforcement learning, characterized in that, include: Acquire power system status observation data; The power system state observation data is input into the power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model; wherein, the power flow regulation generation model is obtained by supervising the fine-tuning of the initial power flow regulation generation model after training using the first observation training sample, and then using the group relative strategy optimization algorithm to train the model obtained by reinforcement learning; the first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

2. The power flow regulation generation method based on supervised fine-tuning and reinforcement learning according to claim 1, characterized in that, Before inputting the power system state observation data into the power flow regulation generation model, the following steps are included: Acquire observation training data and the corresponding control labels, and construct the first observation training sample based on the observation training data using positive perturbation sampling and trajectory reversal; Using the first observation training samples and the control label, the pre-trained initial power flow control generation model is subjected to supervised fine-tuning according to a preset iterative strategy; the preset iterative strategy includes: For each iteration step, the first observation training sample is input into the initial power flow regulation generation model to obtain the regulation training result output by the initial power flow regulation generation model; Based on the training results and corresponding control labels, determine the negative log-likelihood loss function; The first gradient is determined based on the negative log-likelihood loss function, and the model parameters of the initial power flow regulation generation model are updated using the first gradient. The first observation training sample is input into the updated initial power flow regulation generation model, and the iteration is repeated until the negative log-likelihood loss function converges, at which point the training ends and the supervised fine-tuned model is obtained.

3. The power flow regulation generation method based on supervised fine-tuning and reinforcement learning according to claim 2, characterized in that, Based on the observed training data, a first observed training sample is constructed using forward perturbation sampling and trajectory reversal, including: Select initial convergent observation training data, randomly select a control variable, and apply a random perturbation to obtain observation perturbation data; the control variable is used to characterize the adjustment parameters of the adjustable device at the corresponding node in the observation training data; Based on the observed disturbance data, the convergence state of the power flow calculation is determined using the power flow model; the power flow model is constructed based on the operating state of the power system grid. When the power flow calculation converges, the corresponding successful state is recorded, the observed disturbance success data is obtained, and the control variables are randomly selected again and random disturbances are applied. The power balance model is then reused to determine the convergence state of the power flow calculation; or... When it is determined that the power flow calculation does not converge, the corresponding failure state is recorded, the observation disturbance failure data is obtained, and the disturbance is stopped; According to a preset number of perturbation steps, random perturbation is applied to the observation perturbation failure data to generate non-convergent state data, and combined with the observation perturbation success data obtained by perturbing the observation perturbation failure data before the observation perturbation failure data, positive trajectory data is obtained. The forward trajectory data is reversed, and the state transitions between adjacent data points in the reversed trajectory are combined to construct the corresponding first observation training sample.

4. The power flow regulation generation method based on supervised fine-tuning and reinforcement learning according to claim 3, characterized in that, Based on the observed disturbance data, the convergence state of the power flow calculation is determined using the power flow model, including: Based on the observed disturbance data and power flow model, the active power residual and reactive power residual are obtained using a preset iterative algorithm. When both the active power residual and the reactive power residual are less than their respective preset thresholds, the power flow calculation is considered converged; or, When the active power residual and / or the reactive power residual are determined to be greater than or equal to the corresponding preset threshold, the convergence state of the power flow calculation is non-convergence. The forward trajectory data is reversed, and the state transitions between adjacent data points in the reversed trajectory are combined to construct the corresponding first observation training sample, including: The forward trajectory data is reversed to obtain the reversed trajectory; The control command is determined based on the difference between adjacent states in the reverse trajectory, and the corresponding first observation training sample is constructed by combining the previous state in the corresponding adjacent state.

5. The power flow regulation generation method based on supervised fine-tuning and reinforcement learning according to claim 2, characterized in that, The first observation training sample is input into the pre-trained initial power flow regulation generation model to obtain the regulation training results output by the initial power flow regulation generation model, including: The first observation training sample is input into the pre-trained initial power flow regulation generation model, and the first observation training sample is subjected to feature encoding and embedding processing to map the first observation training sample into a high-dimensional vector representation. Based on the high-dimensional vector representation, and using a multi-head self-attention mechanism, the correlation and dependency between power system state features and control variables are captured to obtain a hidden state representation that incorporates contextual information. Based on the autoregressive generation method, the probability distribution of the next word is predicted step by step according to the hidden state representation, thereby generating the control training results.

6. The power flow regulation generation method based on supervised fine-tuning and reinforcement learning according to claim 5, characterized in that, Based on an autoregressive generation method, the probability distribution of the next lexical unit is predicted step-by-step according to the hidden state representation, generating a controlled training result, including: For each time step, the input words for that time step are obtained, and the input words are embedded and mapped and positionally encoded to obtain the word vector for the current step. Based on the multi-head self-attention mechanism, the hidden state representation is used to decode the current step word vector and predict the log probability of each word in the vocabulary space. Based on the logarithmic probability, the target word at the corresponding time step is determined by sampling or greedy search. The target word is concatenated into the prefix output sequence and used as the input word for the next time step. The above steps are repeated until an end marker is generated, resulting in a control training result sequence containing intermediate result thought chains and control instructions.

7. The power flow regulation generation method based on supervised fine-tuning and reinforcement learning according to claim 1, characterized in that, After obtaining the supervised fine-tuned model, the following is included: For each iteration step, based on the previously acquired observation training data, the supervised fine-tuned model is used to obtain the regulation result; the regulation result includes the prediction result of the target number. Based on the prediction results, the corresponding reward is determined, and based on the corresponding reward for each prediction result, the corresponding advantage function is obtained, and the objective function is determined. Based on the objective function, the second gradient is determined by backpropagation, and the supervised fine-tuned model is updated using the second gradient. This process is repeated until the objective function converges or the maximum number of iterations is reached, thus obtaining the power flow regulation generation model.

8. The power flow regulation generation method based on supervised fine-tuning and reinforcement learning according to claim 7, characterized in that, The prediction results include control instructions, corresponding answer tags for the control instructions, intermediate result thought chains, and corresponding thought chain tags for the intermediate result thought chains. Based on each prediction result, a corresponding reward is determined, including: For each prediction result, the integrity, correct order, data format, and parsability of the thought chain tags and result tags in the corresponding prediction results are verified. Based on the verification results and combined with the preset binary reward, the corresponding prediction results are assigned a value to obtain the format reward value corresponding to each prediction result. Based on the prediction results, perform power flow simulation to determine the power flow convergence reward value for each prediction result; Based on the prediction results, determine the out-of-bounds situation of the control variables corresponding to the control instructions in the prediction results, and based on the out-of-bounds situation of the control variables, determine the corresponding penalty or safety reward, and obtain the boundary reward value of the corresponding prediction result. Based on the format reward value, power flow convergence reward value, and boundary reward value of each prediction result, the corresponding reward for each prediction result is obtained; Based on the rewards corresponding to each prediction result, the corresponding advantage function is obtained, and the objective function is determined, including: Based on each prediction result, and combined with the log probabilities of each prediction result obtained from the observation training data under the previous model parameters, the importance sampling ratio is obtained by exponential operation. The importance sampling ratio is limited according to a preset clipping gradient threshold to obtain the clipping result; Based on the rewards corresponding to each prediction result, the mean and standard deviation of the rewards are determined. Then, combined with the preset group size, the rewards of each prediction result are normalized to determine the dominance function of each prediction result. The preset group size is used to characterize the number of prediction results generated by the model. Determine the relative entropy KL divergence between the current model and the supervised fine-tuning model, and determine the KL divergence penalty term in combination with the preset penalty coefficient; Based on the importance sampling ratio, the pruning result, and the advantage function of each prediction result, combined with the KL divergence penalty term, the objective function is obtained.

9. The power flow regulation generation method based on supervised fine-tuning and reinforcement learning according to claim 7, characterized in that, The number of initial power flow regulation generation models is at least two, and after obtaining the power flow regulation generation models, it includes: Obtain the policy entropy of each power flow regulation generation model on the validation dataset. The policy entropy is used to measure the uncertainty of the model output distribution. Based on the strategy entropy of each power flow regulation generation model, the exploratory capability and convergence characteristics of each model are analyzed. Based on the exploration capability and convergence characteristics, a target power flow regulation generation model is selected from at least two power flow regulation generation models and used as input for the power system state observation data.

10. A power flow regulation generation device based on supervised fine-tuning and reinforcement learning, characterized in that, include: The data acquisition module acquires power system status observation data; The regulation generation module inputs the power system state observation data into the power flow regulation generation model to obtain the regulation generation result output by the power flow regulation generation model. The power flow regulation generation model is obtained by supervising the fine-tuning of an initial power flow regulation generation model trained using a first observation training sample, and then using a group relative strategy optimization algorithm to train the model obtained from the supervised fine-tuning through reinforcement learning. The first observation training sample is constructed based on the observation training data using forward perturbation sampling and trajectory reversal.

Images