A power grid real-time scheduling decision method and device based on large model cooperative reinforcement learning

Abstract

The application discloses a power grid real-time scheduling decision method and device based on large model cooperative reinforcement learning, and relates to the field of power grid scheduling. The method comprises the following steps: obtaining input text information, analyzing the input text information based on a preset large language model to generate a real-time semantic feature vector; the large language model is fine-tuned based on a preset corpus; obtaining a power grid real-time state vector, fusing the power grid real-time state vector and the real-time semantic feature vector to obtain a fused state vector; inputting the fused state vector into a preset reinforcement learning intelligent agent to obtain scheduling action information; converting the scheduling action information and the power grid real-time state vector into a to-be-processed sequence based on a preset text conversion mapping table, and inputting the to-be-processed sequence into the large language model to generate a scheduling action explanation text in a natural language form.

Description

Technical Field

[0001] This invention relates to the field of power grid dispatching technology, and in particular to a method and apparatus for real-time power grid dispatching decision-making based on large-scale collaborative reinforcement learning. Background Technology

[0002] Currently, with the large-scale integration of new energy sources such as wind and solar power into the power grid and the accelerated construction of new power systems, power dispatch and control centers, as the core hubs of power grid operation, are facing unprecedented technological challenges. The significantly increased uncertainty on both the source and load sides, the increasingly prominent characteristics of power electronics, and the flexible and varied operating modes mean that real-time dispatch not only needs to address the power balance issues in traditional steady-state operation but also needs to make rapid and coordinated decisions between security constraints, economic efficiency, and low-carbon goals. Against this backdrop, power grid dispatch is evolving towards intelligence and automation, urgently requiring the introduction of artificial intelligence technology to improve the response speed and accuracy of dispatch decisions, in order to support the safe and stable operation of the power grid under large-scale new energy consumption and complex operating scenarios.

[0003] Existing real-time power grid dispatching methods based on deep reinforcement learning provide auxiliary decision support for dispatchers by constructing intelligent agents, but they still have significant limitations: First, traditional reinforcement learning models struggle to effectively process unstructured information such as dispatchers' natural language commands and accident reports, leading to low efficiency in human-computer interaction; second, the decision-making process based on deep neural networks is a typical "black box" model, making it difficult for dispatchers to understand the logical basis of the agent's output actions, greatly limiting its practical application in high-risk scenarios; furthermore, existing reinforcement learning methods cannot flexibly incorporate dispatchers' risk assessments of the system situation, operational preferences, and implicit rule constraints, resulting in frequent discrepancies between the agent's decision-making behavior and human expectations, affecting the credibility and practicality of the auxiliary decision-making system. Summary of the Invention

[0004] This invention provides a method and apparatus for real-time power grid scheduling decision-making based on large-scale collaborative reinforcement learning, in order to improve the interpretability of the decision-making process of reinforcement learning agents and enhance the credibility of intelligent scheduling decisions in practical applications.

[0005] To address the aforementioned technical problems, this invention provides a real-time power grid scheduling decision-making method based on large-model collaborative reinforcement learning, comprising: The input text information is acquired, and the input text information is parsed based on a preset large language model to generate a real-time semantic feature vector; the large language model is obtained by fine-tuning based on a preset corpus. Obtain the real-time state vector of the power grid, and fuse the real-time state vector of the power grid with the real-time semantic feature vector to obtain the fused state vector; The fused state vector is input into a preset reinforcement learning agent to obtain scheduling action information; wherein, the reinforcement learning agent is trained based on a pre-constructed state space, action space and reward function, the state space includes power grid operation state information and prediction information, the action space includes generator control commands and energy storage control commands, and the reward function includes operation adjustment targets and safety constraint penalties; Based on a preset text conversion mapping table, the scheduling action information and the real-time power grid state vector are converted into a sequence to be processed, and the sequence to be processed is input into the large language model to generate a natural language form of scheduling action explanation text.

[0006] This invention acquires input text information and parses it using a large language model fine-tuned based on a corpus, generating real-time semantic feature vectors. This achieves accurate understanding of the natural language text input by dispatchers, transforming unstructured text into machine-processable semantic features. Next, by acquiring the real-time power grid state vector and fusing it with the real-time semantic feature vector, a fused state vector is obtained. This achieves multimodal information fusion of human intent and power grid physical state, providing a more comprehensive input basis for decision-making. Furthermore, by inputting the fused state vector into a pre-defined reinforcement learning agent, dispatch action information is obtained. This reinforcement learning agent is trained based on a state space containing power grid operating status information and prediction information, an action space containing generator and energy storage control commands, and a reward function containing operating adjustment targets and safety constraint penalties, ensuring that dispatch actions satisfy both economic objectives and power grid safety constraints. Finally, based on a pre-defined text conversion mapping table, the dispatch action information and the real-time power grid state vector are converted into a sequence to be processed and input into the large language model to generate natural language decision explanation text. This visualizes and interprets the decision-making process, improving dispatchers' trust in intelligent decision-making and enhancing human-computer interaction efficiency.

[0007] Furthermore, the acquisition of input text information, based on a preset large language model, involves parsing the input text information to generate a real-time semantic feature vector, including: Obtain input text information and input the input text information into the encoder of the large language model to obtain a contextual semantic vector representation; The context semantic vector representation is subjected to intent classification, entity recognition, and relation extraction operations to obtain intent labels, entity names, and constraints. The intent labels, entity names, and constraints are respectively converted into intent-dense vectors, entity-dense vectors, and constraint-dense vectors; The intent dense vector, entity dense vector, and constraint dense vector are concatenated and subjected to nonlinear dimensionality reduction to obtain a real-time semantic feature vector.

[0008] This invention obtains contextual semantic vector representations by acquiring input text information and inputting it into the encoder of a large language model. It leverages the deep semantic understanding capabilities of the large model to capture contextual information and implicit meanings within the text. By performing intent classification, entity recognition, and relation extraction operations on the contextual semantic vector representations, intent labels, entity names, and constraints are obtained, achieving structured extraction of core elements from the input text information. Furthermore, the intent labels, entity names, and constraints are converted into intent-dense vectors, entity-dense vectors, and constraint-dense vectors, respectively, mapping discrete semantic labels to a continuous vector space representation, facilitating subsequent numerical calculations. Finally, by concatenating and performing nonlinear dimensionality reduction on the intent-dense vectors, entity-dense vectors, and constraint-dense vectors, real-time semantic feature vectors are obtained. This achieves efficient fusion and dimensionality reduction of multi-dimensional semantic information, providing semantically rich and dimensionally controllable input features for reinforcement learning agents.

[0009] Furthermore, the step of obtaining input text information and inputting the input text information into the encoder of the large language model to obtain a contextual semantic vector representation includes: Obtain input text information, preprocess the input text information, and obtain a standardized text sequence; The standardized text sequence is segmented using a preset word segmenter to obtain several words and sub-words; A token sequence is generated based on the aforementioned word segmentation and sub-words, and the token sequence is input into the encoder of the large language model to obtain a contextual semantic vector representation.

[0010] This invention obtains a standardized text sequence by acquiring and preprocessing the input text information, eliminating noise and format inconsistencies in the original text, and ensuring the data quality for subsequent processing. Furthermore, it segments the standardized text sequence using a preset word segmenter to obtain several words and sub-words, dividing the continuous natural language text into discrete semantic units, facilitating model understanding. Finally, it generates a token sequence based on the aforementioned words and sub-words, and inputs the token sequence into the encoder of a large language model to obtain a contextual semantic vector representation. This achieves a complete transformation from the original text to a deep semantic representation, laying the foundation for the large model to understand scheduling instructions.

[0011] Furthermore, the step of obtaining the real-time power grid state vector and fusing the real-time power grid state vector with the real-time semantic feature vector to obtain a fused state vector includes: Collect the current power grid operating status data and generate a first state component based on the operating status data; the operating status data includes line load rate, bus voltage amplitude, actual generator output and actual energy storage charging and discharging power; The plan and forecast data for the next moment are obtained, and a second state component is generated based on the plan and forecast data. The plan and forecast data includes the day-ahead scheduling plan or ultra-short-term forecast output of generators, the day-ahead scheduling plan of energy storage, and the ultra-short-term forecast demand of load. The first state component and the second state component are concatenated to generate the real-time state vector of the power grid; The real-time state vector of the power grid and the real-time semantic feature vector are concatenated along the feature dimension to generate the fused state vector.

[0012] This invention collects the current operating status data of the power grid and generates a first state component based on this data, comprehensively reflecting the current operating status and safety condition of the power grid. It then acquires the planning and forecasting data for the next time step and generates a second state component based on this data, introducing predictive information from the future time domain and enabling forward-looking scheduling decisions. By concatenating the first and second state components, a real-time power grid state vector is generated, achieving a fusion of the current state and future trends, providing the agent with complete state perception in the time dimension. Finally, by concatenating the real-time power grid state vector with the real-time semantic feature vector in the feature dimension, a fused state vector is generated, achieving deep coupling between physical world information and human intention information, enabling the reinforcement learning agent's decisions to simultaneously consider both objective power grid constraints and subjective human instructions.

[0013] Furthermore, the fused state vector is input into a preset reinforcement learning agent to obtain scheduling action information, including: The fused state vector is input into the reinforcement learning agent to obtain the initial scheduling instructions for each controlled device; Obtain the upper and lower limits of the adjustable range for each controlled device at the current moment; The initial scheduling instructions for each controlled device are mapped and converted into the actual scheduling power of each controlled device based on the upper limit and lower limit of the adjustable range of each controlled device. The scheduling action information is constructed based on the actual scheduling power of each controlled device.

[0014] This invention obtains initial scheduling instructions for each controlled device by inputting fused state vectors into a reinforcement learning agent. The agent's policy network then generates a basic control scheme that conforms to the current state and human intent. By acquiring the upper and lower limits of the adjustable range for each controlled device at the current moment, the physical adjustment capability boundaries of each device are monitored in real time, avoiding the generation of unexecutable scheduling instructions. The initial scheduling instructions are mapped to the actual scheduling power of each controlled device based on the upper and lower limits of its adjustable range, transforming theoretically normalized instructions into actual executable power values ​​that conform to the physical constraints of the devices, ensuring the physical feasibility of the scheduling instructions. Finally, by constructing scheduling action information based on the actual scheduling power of each controlled device, a complete power grid control scheme is formed, providing an input basis for subsequent execution and interpretation.

[0015] Furthermore, the step of inputting the scheduling action information and the real-time power grid state vector into the large language model to generate natural language form of reinforcement learning agent action explanation text includes: Based on a preset first text conversion mapping table, the key state features in the real-time state vector of the power grid are converted into a first natural language description. Based on a preset second text conversion mapping table, the control instructions in the scheduling action information are converted into a second natural language description. The first natural language description and the second natural language description are filled into a preset explanation prompt word template to generate an input sequence to be processed; The sequence to be processed is input into the large language model to generate a natural language decision explanation text.

[0016] This invention converts key state features in the real-time state vector of the power grid into first natural language descriptions based on a preset text conversion mapping table, transforming abstract numerical state data into natural language expressions that dispatchers can intuitively understand, thus lowering the barrier to understanding specialized data. Simultaneously, it converts control instructions in dispatch action information into second natural language descriptions based on the same preset text conversion mapping table, transforming numerical control schemes into semantic operational instructions, facilitating dispatchers' rapid grasp of decision-making content. Furthermore, by filling the first and second natural language descriptions into preset explanation prompt word templates, a sequence of inputs to be processed is generated. The templates guide the large-scale model to generate explanations according to a standardized structure, ensuring the professionalism and consistency of the output. Finally, by inputting the sequence of inputs into the large-scale language model, natural language decision explanation text is generated, achieving automated conversion from machine-generated decision-making to human-readable text, providing dispatchers with intuitive decision-making references.

[0017] Furthermore, the step of inputting the sequence to be processed into the large language model to generate natural language decision explanation text includes: The input sequence to be processed is input into the large language model, and the attention contribution of each state variable in the sequence to be processed is calculated using the attention mechanism of the large language model. Based on the attention contribution, the dominant state factor of the scheduling action information is identified, and a causal relationship between the state and the action is established based on the dominant state factor. Based on a preset cluster search algorithm and causal relationship, word-by-word decoding is performed to generate natural language decision explanation text, which includes working condition description text, operation logic description text, and expected effect description text.

[0018] This invention inputs the input sequence to be processed into a large language model, utilizes the attention mechanism of the large language model to calculate the attention contribution of each state variable in the sequence, and quantifies the influence of different state factors on the final decision, thus achieving transparency in decision attribution. By identifying the dominant state factors of scheduling action information based on attention contribution, and establishing a causal relationship between state and action based on the dominant state factors, the inherent logic between the decision and the real-time state of the power grid is revealed, enhancing the interpretability of the decision. Through word-by-word decoding based on a preset cluster search algorithm and causal relationship, a natural language decision explanation text containing operating condition description text, operation logic description text, and expected effect description text is generated. The three-part structure fully presents the decision background, decision reasons, and decision expectations, enabling dispatchers to fully understand and evaluate the rationality of the agent's decision.

[0019] Furthermore, after inputting the sequence to be processed into the large language model to generate natural language decision explanation text, and performing real-time scheduling of the power grid system based on the decision text, the method further includes: Obtain the environmental feedback status of the power grid system; The real-time semantic feature vector, the real-time power grid state vector, the scheduling action information, and the environmental feedback state are concatenated to construct a joint feature vector; The joint feature vector is input into the large language model to obtain reward feedback information; The original reward information is obtained based on the reward function and the environmental feedback state; The reward feedback information and the original reward information are weighted and fused to obtain fused reward information, and the reinforcement learning agent is iteratively optimized based on the fused reward information.

[0020] This invention acquires the environmental feedback state of the power grid system, enabling real-time monitoring of the grid response after scheduling actions, thus providing a data foundation for subsequent optimization iterations. By concatenating real-time semantic feature vectors, real-time grid state vectors, scheduling action information, and environmental feedback states to construct a joint feature vector, it achieves a comprehensive fusion of human intent, pre-decision state, decision action, and post-decision effect, providing multi-dimensional input for value assessment. By inputting the joint feature vector into a large language model to obtain reward feedback information, the semantic understanding capability of the large model is used to evaluate the degree of conformity between scheduling actions and human intent, achieving a quantitative expression of human-machine value alignment. Furthermore, by obtaining raw reward information based on the reward function and environmental feedback states, the performance of scheduling actions at the physical level of the power grid is evaluated, including economic and safety indicators. Finally, by weighted fusion of reward feedback information and raw reward information, a fused reward information is obtained, and the reinforcement learning agent is iteratively optimized based on the fused reward information. This achieves dual guidance from "human preferences" and physical goals, enabling the agent to continuously evolve towards the direction expected by humans during the continuous learning process, improving the long-term adaptability of the scheduling strategy and the human-machine collaborative effect.

[0021] Secondly, the present invention provides a power grid real-time scheduling decision-making device based on large-model collaborative reinforcement learning, including a module for executing the method.

[0022] Thirdly, the present invention provides a power grid real-time scheduling decision-making device based on large-model collaborative reinforcement learning, including a processor and an interface circuit. The interface circuit is used to receive signals from other external devices and transmit them to the processor or send signals from the processor to other external devices. The processor is used to implement the method through logic circuits or execution code instructions. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a real-time power grid scheduling decision-making method based on large-model collaborative reinforcement learning, provided in an embodiment of the present invention; Figure 2 This is a framework diagram for the collaboration between a large model and reinforcement learning, provided in an embodiment of the present invention. Detailed Implementation

[0024] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0025] The terms "first" and "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] Example 1 See Figure 1 , Figure 1 This is a flowchart illustrating a real-time power grid scheduling decision-making method based on large-model collaborative reinforcement learning, provided by an embodiment of the present invention. The embodiment of the present invention provides a real-time power grid scheduling decision-making method based on large-model collaborative reinforcement learning, including steps 101 to 104, as detailed below: Step 101: Obtain input text information, parse the input text information based on a preset large language model, and generate a real-time semantic feature vector; the large language model is obtained by fine-tuning based on a preset corpus; In this embodiment, the step of acquiring input text information and parsing the input text information based on a preset large language model to generate a real-time semantic feature vector includes: Obtain input text information and input the input text information into the encoder of the large language model to obtain a contextual semantic vector representation; The context semantic vector representation is subjected to intent classification, entity recognition, and relation extraction operations to obtain intent labels, entity names, and constraints. The intent labels, entity names, and constraints are respectively converted into intent-dense vectors, entity-dense vectors, and constraint-dense vectors; The intent dense vector, entity dense vector, and constraint dense vector are concatenated and subjected to nonlinear dimensionality reduction to obtain a real-time semantic feature vector.

[0028] In this embodiment, the step of obtaining input text information and inputting the input text information into the encoder of the large language model to obtain a contextual semantic vector representation includes: Obtain the input text information, preprocess the input text information to obtain a standardized text sequence; Perform word segmentation on the standardized text sequence based on a preset word segmenter to obtain a number of words and subwords; Generate a token sequence based on the number of words and subwords, and input the token sequence into the encoder of the large language model to obtain a context semantic vector representation.

[0029] In this embodiment, first, obtain the natural language instructions input by the dispatcher or the external text report from the production management system as the input text information, and perform preprocessing operations on the input text information, specifically including uniformly converting the text to the UTF-8 encoding format, removing irrelevant characters and extra spaces in the text, correcting common spelling mistakes, and converting non-standard date and time and number formats into standard formats, so as to obtain a standardized text sequence. Subsequently, perform word segmentation and subword division on the standardized text sequence based on a preset word segmenter, and cut the continuous text into basic semantic units understandable by the model to obtain a number of words and subwords; furthermore, generate a corresponding token sequence based on these words and subwords, and each word or subword is mapped to a unique index number in the vocabulary to generate a token sequence. Finally, input the token sequence into the encoder of the large language model. The encoder calculates the deep semantic representation of each token in a specific context through the self-attention mechanism and the feed-forward neural network in the multi-layer Transformer structure, and finally outputs the context semantic vector representation corresponding to each token. This vector representation contains the semantic information of each word in the text in the overall context, provides a rich feature basis for subsequent semantic parsing operations such as intent classification, entity recognition, and relationship extraction, and ensures that the large language model can accurately understand the true meaning of the dispatching instructions.

[0030] As a specific example of the embodiment of the present invention, the dispatcher inputs the original instruction text "Adjust the output of Unit 3 of xx Power Plant to 450 megawatts and complete it before 15:00". After preprocessing, the standardized text sequence "Adjust the output of Unit 3 of xx Power Plant to 450 megawatts and complete it before 15:00" is obtained. Subsequently, perform word segmentation and subword division on the standardized text sequence based on a preset word segmenter, such as "Adjust", "xx Power Plant", "Unit 3", "Output", "Adjust", "To", "450", "Megawatts", "And before", "15:00", "Before", "Complete". Next, generate a corresponding token sequence based on these words and subwords, and each word or subword is mapped to a unique index number in the vocabulary to form a token sequence in the form of [1024, 3567, 892, 451, 2378, 1562, 78, 6732, 891, 456, 1234, 567, 890], and then input the token sequence into the encoder part of the large language model.

[0031] In this embodiment, the large language model is fine-tuned based on a pre-set corpus. Specifically: First, unstructured text data such as historical dispatch logs, operating procedures, dispatcher operation records, alarm event reports, and accident handling reports from the power grid dispatching domain are collected as raw corpora. These raw corpora undergo preprocessing operations such as noise reduction, deduplication, and typo correction. Equipment names and IDs such as generators, transformers, lines, and sections are standardized to ensure corpus quality. Subsequently, three types of key semantic annotations are applied to the standardized corpus. The first type is text instruction semantic feature annotation, which annotates the key dispatching actions, operation object entities, and constraints contained in the text, constructing text instruction semantic feature training pairs. The second type is explanatory logic annotation, which annotates the reasons and expected effects of the agent's choice of actions in a specific power grid state, constructing state-action explanatory text training pairs. The third type is preference value annotation, which annotates the dispatcher's preference for two different dispatching actions, constructing a state-action-action-human feedback quadruple for value alignment. The standardized and annotated data is organized into a structured power grid dispatching domain corpus and divided into training, validation, and test sets.

[0032] As a specific example of this invention, three types of key semantic annotations are performed on the standardized corpus. First, text instruction-semantic feature annotation is performed, annotating the key scheduling actions, operational entities, and constraints contained in the text, constructing a "text instruction-semantic feature" training pair. Second, explanation-logic annotation is performed, annotating the reasons and expected effects of the agent's choice of actions in a specific power grid state, constructing a "(state-action)-explanation text" training pair. Third, preference-value annotation is performed, annotating the dispatcher's preference tendencies or corrective feedback for a specific sequence of scheduling actions. For two different scheduling actions, the dispatcher's preference is annotated. (-1: preference for action 1, 1: preference for action 2, 0: no difference), construct a (state, action 1, action 2, human feedback) quadruple for value alignment. The standardized and labeled data are organized into a structured corpus for the power grid dispatch domain and divided into training, validation, and test sets.

[0033] In this embodiment, a pre-trained general-purpose large language model is selected as the base model, and the constructed corpus is used to fine-tune it for domain adaptation, so that it incorporates power grid professional knowledge and understands professional terminology and logic.

[0034] In this embodiment, after completing the domain-adaptive fine-tuning, the parameters of the main body of the large language model are frozen to preserve the stability of its domain knowledge. Above the frozen main output layer, trainable task-specific layers are added to meet the functional requirements of different modules. The natural language perception task layer is trained using annotated text instruction semantic features, enabling it to map input instruction text into structured semantic feature vectors that the reinforcement learning agent can directly utilize. The action logic interpretation task layer is trained using annotated state-action explanation text, enabling it to receive power grid status and scheduling actions and output natural language decision explanations. The value alignment reward task layer is trained using annotated state-action-action-human feedback quadruples. Through a preference learning algorithm, it outputs a value alignment reward to evaluate the degree of alignment between scheduling actions and human preferences, guiding the reinforcement learning agent to learn strategies more aligned with expert experience.

[0035] In this embodiment, the encoder of the large language model consists of a multi-layer Transformer structure. It performs deep semantic modeling of the input text using a self-attention mechanism and a feedforward neural network, outputting a contextual semantic vector representation. This vector representation contains the semantic information of each token within a specific context. Subsequently, based on this contextual semantic vector representation, the task-specific layer of the large language model performs three types of semantic parsing operations in parallel: intent classification, entity recognition, and relation extraction. Intent classification is used to determine the dispatcher's core operational intent, such as adjusting output, switching equipment, or setting constraints. Entity recognition is used to extract specific equipment names, power plant names, or numbers mentioned in the text. Relation extraction is used to identify logical relationships between entities, such as inclusion, connection, or control relationships. Through these operations, the model outputs structured intent labels, entity names, and constraints.

[0036] In this embodiment, the intent label, entity name, and constraints are input into the corresponding embedding layers. Each embedding layer maintains a trainable embedding matrix, mapping discrete labels and names to continuous dense vectors, resulting in intent dense vectors, entity dense vectors, and constraint dense vectors, respectively. Finally, these three dense vectors are concatenated along the feature dimension to form a combined vector. This combined vector is then input into a nonlinear dimensionality reduction network consisting of fully connected layers. A nonlinear transformation is performed through an activation function, outputting a low-dimensional real-time semantic feature vector. This vector condenses the key semantic information in the input text, facilitating subsequent fusion with the power grid state vector.

[0037] Step 102: Obtain the real-time state vector of the power grid, and fuse the real-time state vector of the power grid with the real-time semantic feature vector to obtain the fused state vector; In this embodiment, obtaining the real-time state vector of the power grid and fusing the real-time state vector of the power grid with the real-time semantic feature vector to obtain a fused state vector includes: Collect the current power grid operating status data and generate a first state component based on the operating status data; the operating status data includes line load rate, bus voltage amplitude, actual generator output and actual energy storage charging and discharging power; The plan and forecast data for the next moment are obtained, and a second state component is generated based on the plan and forecast data. The plan and forecast data includes the day-ahead scheduling plan or ultra-short-term forecast output of generators, the day-ahead scheduling plan of energy storage, and the ultra-short-term forecast demand of load. The first state component and the second state component are concatenated to generate the real-time state vector of the power grid; The real-time state vector of the power grid and the real-time semantic feature vector are concatenated along the feature dimension to generate the fused state vector.

[0038] In this embodiment, the real-time state vector of the power grid is obtained and fused with the real-time semantic feature vector output by the previously generated natural language perception module to generate a fused state vector.

[0039] In this embodiment, the current operating status data of the power grid is first collected. Specifically, this includes obtaining the real-time load rate of each line from the data acquisition and monitoring control system, the voltage amplitude of each bus from the substation monitoring and control device, the actual active and reactive power output of each generator from the power plant monitoring system, and the actual charging and discharging power of each energy storage device from the energy storage management system. A first state component is generated based on this operating data. Simultaneously, the planned and predicted data for the next time step is acquired. Specifically, this includes reading the planned output values ​​of thermal power units and hydropower units for the next time step from the day-ahead dispatching plan system, reading the ultra-short-term predicted output values ​​of wind farms and photovoltaic power plants from the new energy power prediction system, reading the planned charging and discharging power of each energy storage device for the next time step from the energy storage dispatching plan, and reading the ultra-short-term predicted demand of the system load for the next time step from the load prediction system. A second state component is generated based on this planned and predicted data.

[0040] In this embodiment, the first state component and the second state component are sequentially concatenated along the feature dimension to form a comprehensive vector that includes the current operating status and the trend of change in the next moment, which is the real-time state vector of the power grid.

[0041] In this embodiment, the real-time state vector of the power grid is concatenated with the previously generated real-time semantic feature vector in the feature dimension to generate a fused state vector. This fused state vector contains both the real-time operation information of the power grid's physical world and the dispatcher's instruction intent information. As the complete input state of the reinforcement learning agent, it enables the agent to take into account both the objective constraints of the power grid and the user's subjective control intent when making scheduling decisions.

[0042] Step 103: Input the fused state vector into a preset reinforcement learning agent to obtain scheduling action information; wherein, the reinforcement learning agent is trained based on a pre-constructed state space, action space and reward function, the state space includes power grid operation state information and prediction information, the action space includes generator control commands and energy storage control commands, and the reward function includes operation adjustment targets and safety constraint penalties; In this embodiment, the construction process of the reinforcement learning agent includes multiple steps such as simulation environment construction, state space definition, action space definition, reward function design, and agent training algorithm selection.

[0043] In this embodiment, a reinforcement learning simulation environment is first constructed for the real-time power grid dispatching task. This environment uses an AC power flow model to calculate network power flow in order to accurately simulate the real operating state of the power system. This includes branch active power equation, branch reactive power equation, node active power balance equation, and node reactive power balance equation. It also includes operating state update models for various types of equipment such as generators, energy storage, and loads. Taking energy storage as an example, its state of charge is updated according to the charging and discharging power and charging and discharging efficiency according to the corresponding formulas. Among them, generators include thermal power generators, hydropower generators, wind power generators, and photovoltaic generators.

[0044] In this embodiment, the AC power flow model is specifically as follows: (1) (2) (3) (4) in, , , , These are the busbars i The connected busbar set, generator set, energy storage set, and load set; and They are t Time bus i Flow to bus j The active power and reactive power are the quantities to be determined. yes t Time bus i and busbar j The voltage phase difference between them is the quantity to be determined; yes t Time bus i The voltage amplitude is the quantity to be determined; and It is a connecting bus. i and busbar j The conductance and susceptance of the branch are the input quantities; and They are t Time of the first k The active and reactive power outputs of each generator are the input quantities. yes t Time of the first k Each energy storage unit has a charging and discharging power rating; charging is positive and discharging is negative, representing the input quantity. and They are t Time of the first k The active and reactive power demands of each load are the input quantities.

[0045] In this embodiment, within the simulation environment, the controlled device receives scheduling instructions to update its operating state, while the uncontrollable device undergoes state transitions based on scenario data. Taking energy storage as an example, its state update is as follows: (5) In the formula, yes t Time of the first i The amount of electricity stored is the amount to be demanded; yes t Time of the first i The charging and discharging power of the energy storage is the input quantity; It is the first i The charging and discharging efficiency of each energy storage unit is the input quantity; is the scheduling time interval, and is the input quantity; in this invention, 0.25 hours is selected.

[0046] In this embodiment, a state space is defined. state space It includes the system's operating status information at the current time (t) and the planning and forecasting information for the next time (t+1). The operating status information at the current time includes the line load rate. Bus voltage amplitude Actual output of generator and the actual charging and discharging power of energy storage The next moment's planning and forecasting information includes the day-ahead scheduling plan or ultra-short-term forecasted output of the generators. Day-ahead dispatch plan for energy storage and the ultra-short-term forecast demand of the load This information is then combined to form a complete real-time state vector of the power grid.

[0047] In this embodiment, the action space is then defined. The action space includes generator control commands and energy storage control commands; the agent outputs the original scheduling commands. Then, based on the upper and lower limits of the adjustable range of each controlled device at the current moment, it is mapped to the actual scheduling power to ensure the physical feasibility of the scheduling command. Specifically: (6) in, , Controlled equipment i exist t The upper and lower limits of the adjustable range at any given time.

[0048] In this embodiment, the reward function uses minimizing the adjustment amount in the real-time stage as the objective function and takes the negative value of the adjustment amount. At the same time, corresponding penalties are imposed on cases where the control command violates the power balance constraint, the upper and lower limits of branch power flow constraint, and the upper and lower limits of bus voltage constraint. Specifically: (7) in, Is the system in t The amount of adjustment at any given time; yes t Penalties for constantly violating power balance constraints; yes t Penalties for constantly violating the upper and lower limits of the branch flow; yes t Penalties for constantly violating the upper and lower limits of bus voltage constraints; , , and These are the weighting coefficients for each item.

[0049] In this embodiment, the adjustment amount includes the sum of the absolute values ​​of the differences between the actual dispatched power of generators and energy storage and the day-ahead dispatch plan. The system in... t Adjustment amount at time as follows: (8) In the formula, and These are the system's generator set and energy storage set, respectively. and The generator is at tReal-time scheduling plan and day-ahead scheduling plan. and They are energy storage t Real-time scheduling plan and day-ahead scheduling plan.

[0050] In this embodiment, the power balance constraint penalty is calculated by evaluating whether the balancing machine exceeds the limit, specifically: (9) In the formula, for t The output of the constant balancing machine, , for t The upper and lower limits of the unit's allowable output should be balanced at all times.

[0051] In this embodiment, the branch power flow constraint penalty is calculated based on the cumulative value of line load rates exceeding one, specifically: (10) In the formula, It is the set of branches of the system. yes t Time Branch i The load rate.

[0052] In this embodiment, the bus voltage constraint penalty is calculated based on the cumulative value of the voltage amplitude deviation range, which specifically ranges from 0.95 to 1.05. (11) In the formula, It is the system's bus set. It is the voltage amplitude of bus i at time t.

[0053] In this embodiment, the near-end policy optimization algorithm is selected as the basic training algorithm, and the generalized advantage estimation method is used to calculate the advantage function. A policy network and a value network are constructed. The training objective of the policy network is to maximize the pruned policy loss function, and the training objective of the value network is to minimize the mean squared error between the predicted value and the target reward. This is achieved through iterative steps such as interactive sampling with the environment, calculating fusion rewards, and updating network parameters. Finally, a reinforcement learning agent capable of outputting optimized scheduling actions is trained. During the training process, the agent also incorporates value-aligned rewards generated by a large language model, ensuring that the agent's decision-making behavior simultaneously conforms to both power grid physical constraints and human scheduling preferences.

[0054] In this embodiment, since traditional reinforcement learning rewards cannot fully reflect human preferences and implicit rules, this solution integrates human-machine value alignment into the reinforcement learning framework for agent training.

[0055] In this embodiment, a fusion reward function is first constructed to integrate the real-time semantic feature vectors. Real-time state vector of the power grid at the current moment The scheduling actions output by the intelligent agent ,Will , and After concatenation, the data is input into a pre-trained value alignment reward module. This module is trained on a corpus of four-tuples containing human preference annotations and can directly output value alignment rewards. This reward value is used to evaluate the degree to which scheduling actions conform to human instructions and scheduling procedures.

[0056] In this embodiment, the original reward for reinforcement learning is calculated simultaneously based on the reward function defined in the reinforcement learning simulation environment. The original reward includes penalties for negative operational adjustments, power balance constraints, branch power flow constraints, and bus voltage constraints, used to evaluate the economic and security performance of dispatch actions at the power grid physical level. A weighted combination of the reinforcement learning original reward and the value alignment reward is then constructed to create a fused reward: (12) in, It is a hyperparameter used to balance the original reward in reinforcement learning and the degree of alignment between human and machine value. By adjusting... The value of can control the degree to which the agent emphasizes physical goals and human preferences during the optimization process.

[0057] Subsequently, a policy network and a value network are constructed based on a fully connected neural network. The policy network is denoted as... The probability distribution used to output scheduling actions based on the current state, with network parameters θ, is denoted as the value network. The network parameters used to evaluate the value of the current state are: The reinforcement learning algorithm PPO (Proximal Policy Optimization) was selected as the basic training algorithm, and the advantage function was calculated using the generalized advantage estimation (GAE) method.

[0058] In this embodiment, during each training round, the agent interacts with the environment to collect a batch of experience trajectories, based on fusion rewards. Calculate single-step timing difference error Specifically: (13) Furthermore, the dominance function is calculated based on the single-step time-series difference error. The advantage function is equal to the weighted sum of the discounted time-series difference errors over the subsequent multiple steps, specifically: (14) in, It is the value network's response to the current state. The predicted value, It is the discount factor hyperparameter. It is a smoothing factor hyperparameter, and this advantage function is used to measure the performance of a state. Take action below The degree of superiority or inferiority relative to the average level.

[0059] In this embodiment, the training objective of the policy network is to maximize the pruned policy loss function. The mathematical expectation is in the form of a minimum value operation, specifically: (15) in The importance sampling ratio is calculated as follows: , As the current strategy, For the old strategy, the clip function will... The range is limited to 1 minus ϵ to 1 plus ϵ, where ϵ is the pruning magnitude hyperparameter used to control the step size of policy updates. By taking the minimum value of the original advantage weighting term and the pruned advantage weighting term, the policy update magnitude is avoided from being too large, which would lead to training instability.

[0060] In this embodiment, the training objective of the value network is to minimize the mean squared error between the predicted value and the target reward, and the loss function is... for: (16) in, It is the value network's prediction of the current state, calculated based on the fusion reward and the value of subsequent states.

[0061] In this embodiment, through multiple rounds of iterative training, the parameters of the policy network and the value network are continuously updated and optimized, ultimately enabling the reinforcement learning agent to output optimized scheduling actions that meet both the requirements of safe and economical operation of the power grid and human scheduling preferences, thereby achieving continuous learning and evolution of human-machine value alignment.

[0062] In this embodiment, the fused state vector is input into a preset reinforcement learning agent to obtain scheduling action information, including: The fused state vector is input into the reinforcement learning agent to obtain the initial scheduling instructions for each controlled device; Obtain the upper and lower limits of the adjustable range for each controlled device at the current moment; The initial scheduling instructions for each controlled device are mapped and converted into the actual scheduling power of each controlled device based on the upper limit and lower limit of the adjustable range of each controlled device. The scheduling action information is constructed based on the actual scheduling power of each controlled device.

[0063] In this embodiment, after obtaining a fused state vector containing information on the physical state of the power grid and human intent, the fused state vector is input into a preset reinforcement learning agent to obtain scheduling action information. First, the fused state vector is passed as input to the policy network of the reinforcement learning agent. This policy network consists of a multi-layer fully connected neural network. After forward propagation calculation, it outputs the initial scheduling instructions for each controlled device. These initial scheduling instructions are continuous value vectors normalized to [-1, 1].

[0064] As a specific example of an embodiment of the present invention, for a regional power grid containing 5 generator sets and 2 energy storage power stations, the strategy network outputs a 7-dimensional vector, where each component corresponds to the original control command value of 1 generator or 1 energy storage power station, with positive values ​​indicating increased output or charging and negative values ​​indicating decreased output or discharging.

[0065] At the same time, the system obtains the upper and lower limits of the adjustable range of each controlled device in real time from the equipment monitoring system. For example, since the No. 1 thermal power unit is in peak power generation, its adjustable upper limit is 100% of the rated capacity and its adjustable lower limit is 80%. The No. 2 hydropower unit has sufficient water supply and its adjustable range is 30% to 90%. The No. 3 energy storage power station is currently in a high state of charge and its adjustable upper limit of charging power is 100% of the rated power and its adjustable upper limit of discharging power is 0.

[0066] Subsequently, the initial dispatch command is mapped and transformed based on the upper and lower limits of the adjustable range of each controlled device. For each controlled device, its normalized initial command value is linearly mapped to the actual adjustable range. For example, if the initial command value of a generator is 0.6, the upper limit of adjustable range is 100MW, and the lower limit of adjustable range is 50MW, then after the transformation, its actual dispatch power is 50MW plus 0.6 multiplied by the difference between 100MW and 50MW, which is 80MW. As another example, if the initial command value of a certain energy storage power station is -0.3, indicating discharge, but its adjustable range has an upper limit of discharge of 30MW and a lower limit of discharge of 0MW, then after the transformation, its actual discharge power is 0MW plus the absolute value of -0.3 multiplied by the difference between 30MW and 0MW, which is 9MW. Through this mapping process, the theoretical normalized command is transformed into an actual executable power value that conforms to the current physical constraints of each device.

[0067] Finally, the calculated actual dispatch power of each controlled device is combined into a complete dispatch action information vector according to the device number order. This vector contains the planned output of all generator sets and the planned charging and discharging power of all energy storage stations.

[0068] Step 104: Based on a preset text conversion mapping table, the scheduling action information and the real-time power grid state vector are converted into a sequence to be processed, and the sequence to be processed is input into the large language model to generate a natural language form of scheduling action explanation text.

[0069] In this embodiment, the step of inputting the scheduling action information and the real-time state vector of the power grid into the large language model to generate reinforcement learning agent action explanation text in natural language form includes: Based on a preset first text conversion mapping table, the key state features in the real-time state vector of the power grid are converted into a first natural language description. Based on a preset second text conversion mapping table, the control instructions in the scheduling action information are converted into a second natural language description. The first natural language description and the second natural language description are filled into a preset explanation prompt word template to generate an input sequence to be processed; The sequence to be processed is input into the large language model to generate a natural language decision explanation text.

[0070] In this embodiment, the scheduling action information is... and the real-time state vector of the power grid Inputting a large language model to generate natural language decision explanation text, firstly, the real-time state vector of the power grid is converted based on a pre-defined text transformation mapping table. Converting key state features into first natural language descriptions The first text conversion mapping table is a predefined set of mapping relationships that maps numerical state features such as line load rate, bus voltage amplitude, and generator output to corresponding natural language phrases. For example, a load rate of 95% is mapped to "Line A load rate is too high" or a voltage amplitude of 1.05 per unit is mapped to "Bus B voltage is close to the upper limit".

[0071] In this embodiment, the scheduling action information is simultaneously based on a preset second text conversion mapping table. Control instructions in Converted into a second natural language description, the mapping table maps the power adjustment of generators and energy storage to natural language operation instructions. For example, increasing the output of a generator by 20 megawatts is mapped to "increase the output of generator No. 1 by 20 megawatts" or discharging a storage device by 10 megawatts is mapped to "discharge 10 megawatts from storage device No. 2".

[0072] In this embodiment, the first natural language description will then be... Second natural language description The input is filled into a preset explanation prompt template, which is a fixed sentence structure containing placeholders. For example, it might be designed as "The current power grid is in [State], and the agent suggests performing [Action]. Please explain the logical reason for performing this operation in conjunction with the power system operation procedures." After filling the first natural language description into the state placeholder and the second natural language description into the operation placeholder, a complete input sequence to be processed is generated. Finally, the input sequence to be processed... The input is a large language model, which leverages its expertise and reasoning abilities learned during the fine-tuning phase in the power grid dispatching domain to process the input sequence. Perform semantic understanding and logical analysis to generate natural language decision explanation text as output.

[0073] In this embodiment, the step of inputting the sequence to be processed into the large language model to generate natural language decision explanation text includes: The input sequence to be processed is input into the large language model, and the attention contribution of each state variable in the sequence to be processed is calculated using the attention mechanism of the large language model. Based on the attention contribution, the dominant state factor of the scheduling action information is identified, and a causal relationship between the state and the action is established based on the dominant state factor. Based on a preset cluster search algorithm and causal relationship, word-by-word decoding is performed to generate natural language decision explanation text, which includes working condition description text, operation logic description text, and expected effect description text.

[0074] In this embodiment, the generated input sequence to be processed is first... As input to the large language model, which has been fine-tuned using corpus data from the power grid dispatching domain, the input sequence contains power grid state features and dispatching action information described in natural language. Upon receiving the input sequence, the large language model utilizes its multi-head self-attention mechanism to calculate the attention contribution value for each token in the input sequence. This mechanism assigns attention weights to each token through matrix operations on the query keys, thereby quantifying the influence of different state variables in the input sequence on the final output. Specifically, it calculates the attention contribution value for tokens corresponding to key state features such as line load rate, bus voltage, and generator output. A higher value indicates a greater contribution of that state feature to subsequent text generation. This is based on the calculated attention contribution of each state variable. The model identifies the dominant state factor that causes the agent to take the current scheduling action. For example, if it finds that the attention weight corresponding to "the load rate of line A is too high" is significantly higher than other state variables, it determines that the factor is the dominant reason for the decision and establishes a causal relationship between the dominant state factor and the scheduling action in the model's internal representation layer, that is, "because the load rate of line A is too high, it is necessary to perform operations to increase output or adjust power flow".

[0075] In this embodiment, after establishing the causal relationship, the model enters the text decoding stage, using a preset cluster search algorithm for word-by-word generation. This algorithm retains multiple candidate word sequence paths with the highest probability in each decoding step, and combines them with the established causal relationship to guide the search direction, ensuring that the generated text can accurately reflect the logical connection between the state and the action. The final generated natural language decision explanation text is structured into three parts: the first part is the operating condition description text to explain the key operating state of the current power grid; the second part is the operation logic description text to explain the reason for taking the scheduling action; and the third part is the expected effect description text to explain the expected improvement of the power grid state after the action is executed, so that the dispatcher can fully understand the internal logic and expected impact of the agent's decision.

[0076] As a specific example of an embodiment of the present invention, the scheduling action information output by the reinforcement learning agent is "Increase the output of generator No. 1 by 20 MW and discharge the energy storage station No. 2 by 10 MW". The key state feature contained in the corresponding real-time state vector of the power grid is "The load rate of line A reaches 95% and the voltage of bus B is 1.05 per unit". The system first converts the key state feature into a first natural language description based on a preset first text conversion mapping table, resulting in "The load rate of line A is too high, reaching 95%, and the voltage of bus B is close to the upper limit of 1.05 per unit". At the same time, based on a preset second text conversion mapping table, the control instruction in the scheduling action information is converted into a second natural language description, resulting in "Increase the output of generator No. 1 by 20 MW and allow the energy storage station No. 2 to discharge 10 MW". The two natural language descriptions were then filled into a preset explanation prompt template, which was designed as "The current power grid is in a certain state. The agent suggests performing an operation. Please explain the logical reason for performing this operation in conjunction with the power system operation procedures." The generated input sequence after filling was "The current power grid is in a state where the load rate of line A is too high, reaching 95%, and the voltage of bus B is close to the upper limit of 1.05 per unit value. The agent suggests performing an operation to increase the output of generator No. 1 by 20 MW and allow energy storage station No. 2 to discharge 10 MW. Please explain the logical reason for performing this operation in conjunction with the power system operation procedures."

[0077] In this embodiment, the input sequence to be processed is input into a large language model fine-tuned from the power grid dispatch corpus. The model analyzes the correlation between state and action through an attention mechanism, identifying the dominant state factors as high load rate of line A and high voltage of bus B. Combining the provisions on active and reactive power regulation in the power system operation procedures, the final natural language decision explanation text is generated as follows: "Currently, the load rate of line A is 95%, indicating an overload risk. At the same time, the voltage of bus B is close to the upper limit, requiring a reduction in reactive power. The agent suggests increasing the output of generator No. 1 by 20 MW to increase the system's active power supply and alleviate the line load pressure, and allowing energy storage station No. 2 to discharge 10 MW to absorb reactive power through energy storage discharge to reduce the bus voltage. This operation is expected to reduce the load rate of line A to 88% and restore the voltage of bus B to the safe range of 1.02 per unit." This natural language decision explanation text fully includes three elements: operating condition description, operational logic, and expected effect, providing dispatchers with a clear and understandable decision-making basis.

[0078] In this embodiment, after inputting the sequence to be processed into the large language model to generate natural language decision explanation text, and performing real-time scheduling of the power grid system based on the decision text, the method further includes: Obtain the environmental feedback status of the power grid system; The real-time semantic feature vector, the real-time power grid state vector, the scheduling action information, and the environmental feedback state are concatenated to construct a joint feature vector; The joint feature vector is input into the large language model to obtain reward feedback information; The original reward information is obtained based on the reward function and the environmental feedback state; The reward feedback information and the original reward information are weighted and fused to obtain fused reward information, and the reinforcement learning agent is iteratively optimized based on the fused reward information.

[0079] In this embodiment, after generating natural language decision explanation text and performing real-time scheduling of the power grid system based on the text, the reinforcement learning agent is further iteratively optimized through a value alignment reward mechanism. First, the environmental feedback state of the power grid system after the scheduling action is executed is obtained. This environmental feedback state includes operational data such as the line load rate, bus voltage amplitude, actual generator output, and actual charging and discharging power of energy storage at the next moment after the action is executed, reflecting the actual impact of the scheduling action on the power grid.

[0080] In this embodiment, a joint feature vector is constructed by concatenating the real-time semantic feature vector, the real-time power grid state vector before the scheduling action, the scheduling action information executed, and the environmental feedback state obtained after execution. This vector simultaneously contains information from four dimensions: human intent, pre-decision state, execution action, and post-decision effect. This joint feature vector is then input into the value alignment reward module of the large language model. This module, trained on a corpus of four-tuples containing human preference annotations, outputs a reward feedback message. This reward value is used to evaluate the degree to which the scheduling action executed under the current human intent and power grid state conforms to human preferences and scheduling procedures.

[0081] In this embodiment, raw reward information is calculated simultaneously based on a predefined reward function and environmental feedback state in the reinforcement learning simulation environment. This raw reward includes negative values ​​for operational adjustments and penalty terms for power balance constraints, branch power flow constraints, and bus voltage constraints, used to evaluate the economic and security performance of scheduling actions at the power grid physical level. The reward feedback information output by the large language model is weighted and fused with the raw reward information calculated based on physical constraints to obtain fused reward information. This fused reward simultaneously reflects the degree of conformity to human preferences and the degree of achievement of power grid physical goals.

[0082] In this embodiment, the policy network and value network of the reinforcement learning agent are iteratively optimized based on the fused reward information. The advantage function and loss function are calculated and the network parameters are updated through the PPO algorithm, so that the agent can continuously evolve in a direction that conforms to both human preferences and the requirements of safe and economical operation of the power grid during the continuous learning process, thereby achieving closed-loop optimization of human-machine value alignment.

[0083] Please refer to Figure 2 , Figure 2 This invention provides a framework diagram for the collaboration between a large language model and reinforcement learning. The framework consists of two parts: a large language model side and a reinforcement learning agent side. The large language model side includes three functional modules: a natural language perception module, an action logic interpretation module, and a value alignment reward module. These three modules are built based on the same large language model fine-tuned from a corpus in the power grid scheduling domain, sharing underlying power grid expertise. The reinforcement learning agent side includes a policy network and a value network, used to generate scheduling actions and evaluate state values.

[0084] In this embodiment, firstly, the natural language commands or external text reports from the human dispatcher are input into the natural language perception module, which outputs a real-time semantic feature vector. This real-time semantic feature vector is then fused with the real-time state vector of the power grid and input into the policy network of the reinforcement learning agent. The policy network outputs a dispatching action. This dispatching action is executed in the power grid simulation environment and, along with the real-time state vector of the power grid, is input into the action logic interpretation module. This module generates a natural language decision interpretation text to feed back to the dispatcher. Simultaneously, the environmental feedback state after the dispatching action is executed, along with the real-time semantic feature vector and the real-time state vector of the power grid, is input into the value alignment reward module, which outputs a value alignment reward. This value alignment reward is then fused with the original reward calculated based on the environmental feedback to obtain a fused reward used for iterative optimization of the policy network of the reinforcement learning agent. The entire framework forms a complete closed loop from human command input to dispatching action output, and then to decision interpretation and reward feedback, achieving deep collaboration between a large language model and reinforcement learning.

[0085] This invention provides a power grid real-time scheduling decision-making device based on large-model collaborative reinforcement learning, including a module for executing the method.

[0086] This invention provides a real-time power grid scheduling decision-making device based on large-model collaborative reinforcement learning, including a processor and an interface circuit. The interface circuit is used to receive signals from other external devices and transmit them to the processor, or to send signals from the processor to other external devices. The processor is used to implement the method through logic circuits or execution code instructions.

[0087] The present invention provides a computer program product, including a computer program or instructions, which, when executed by a communication device, implement the method described therein.

[0088] In this embodiment of the invention, a processing device is also provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described large-model collaborative reinforcement learning method for real-time power grid dispatching decision-making.

[0089] In this embodiment of the invention, a computer-readable storage medium is also provided, which includes a stored computer program, wherein the computer program controls the device where the computer-readable storage medium is located to execute the above-described large-model collaborative reinforcement learning power grid real-time dispatch decision-making method when it is running.

[0090] For example, a computer program can be divided into one or more modules, one or more of which are stored in memory and executed by a processor to perform the present invention. The one or more modules can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a processing device.

[0091] The processing device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processing device may include, but is not limited to, a processor, memory, and a display. Those skilled in the art will understand that the above components are merely examples of the processing device and do not constitute a limitation on the processing device. It may include more or fewer components than the specified components, or a combination of certain components, or different components. For example, the processing device may also include input / output devices, network access devices, buses, etc.

[0092] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the processing device, connecting all parts of the processing device through various interfaces and lines.

[0093] Memory can be used to store computer programs and / or modules. The processor performs various functions of the processing device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function (such as sound playback, text conversion, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as audio data, text message data, etc.). In addition, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0094] The module for real-time power grid dispatching decisions based on collaborative reinforcement learning of large models, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. Those skilled in the art can understand and implement this without any inventive effort.

[0095] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A real-time power grid dispatching decision-making method based on large-scale collaborative reinforcement learning, characterized in that, include: The input text information is obtained, and the input text information is parsed based on a preset large language model to generate a real-time semantic feature vector; The large language model is obtained by fine-tuning based on a pre-set corpus; Obtain the real-time state vector of the power grid, and fuse the real-time state vector of the power grid with the real-time semantic feature vector to obtain the fused state vector; The fused state vector is input into a preset reinforcement learning agent to obtain scheduling action information; wherein, the reinforcement learning agent is trained based on a pre-constructed state space, action space and reward function, the state space includes power grid operation state information and prediction information, the action space includes generator control commands and energy storage control commands, and the reward function includes operation adjustment targets and safety constraint penalties; Based on a preset text conversion mapping table, the scheduling action information and the real-time power grid state vector are converted into a sequence to be processed, and the sequence to be processed is input into the large language model to generate a natural language form of scheduling action explanation text.

2. The power grid real-time dispatch decision-making method based on large-model collaborative reinforcement learning as described in claim 1, characterized in that, The process of acquiring input text information involves parsing the input text information based on a preset large language model to generate real-time semantic feature vectors, including: Obtain input text information and input the input text information into the encoder of the large language model to obtain a contextual semantic vector representation; The context semantic vector representation is subjected to intent classification, entity recognition, and relation extraction operations to obtain intent labels, entity names, and constraints. The intent labels, entity names, and constraints are respectively converted into intent-dense vectors, entity-dense vectors, and constraint-dense vectors; The intent dense vector, entity dense vector, and constraint dense vector are concatenated and subjected to nonlinear dimensionality reduction to obtain a real-time semantic feature vector.

3. The power grid real-time dispatch decision-making method based on large-model collaborative reinforcement learning as described in claim 2, characterized in that, The step of obtaining input text information and inputting the input text information into the encoder of the large language model to obtain a contextual semantic vector representation includes: Obtain input text information, preprocess the input text information, and obtain a standardized text sequence; The standardized text sequence is segmented using a preset word segmenter to obtain several words and sub-words; A token sequence is generated based on the aforementioned word segmentation and sub-words, and the token sequence is input into the encoder of the large language model to obtain a contextual semantic vector representation.

4. The power grid real-time dispatch decision-making method based on large-model collaborative reinforcement learning as described in claim 1, characterized in that, The step of obtaining the real-time power grid state vector and fusing the real-time power grid state vector with the real-time semantic feature vector to obtain a fused state vector includes: Collect the current power grid operating status data and generate a first state component based on the operating status data; the operating status data includes line load rate, bus voltage amplitude, actual generator output and actual energy storage charging and discharging power; The plan and forecast data for the next moment are obtained, and a second state component is generated based on the plan and forecast data. The plan and forecast data includes the day-ahead scheduling plan or ultra-short-term forecast output of generators, the day-ahead scheduling plan of energy storage, and the ultra-short-term forecast demand of load. The first state component and the second state component are concatenated to generate the real-time state vector of the power grid; The real-time state vector of the power grid and the real-time semantic feature vector are concatenated along the feature dimension to generate the fused state vector.

5. The power grid real-time dispatch decision-making method based on large-model collaborative reinforcement learning as described in claim 4, characterized in that, The fused state vector is input into a preset reinforcement learning agent to obtain scheduling action information, including: The fused state vector is input into the reinforcement learning agent to obtain the initial scheduling instructions for each controlled device; Obtain the upper and lower limits of the adjustable range for each controlled device at the current moment; The initial scheduling instructions for each controlled device are mapped and converted into the actual scheduling power of each controlled device based on the upper limit and lower limit of the adjustable range of each controlled device. The scheduling action information is constructed based on the actual scheduling power of each controlled device.

6. The power grid real-time dispatch decision-making method based on large-model collaborative reinforcement learning as described in claim 5, characterized in that, The step of inputting the scheduling action information and the real-time state vector of the power grid into the large language model to generate natural language form of reinforcement learning agent action explanation text includes: Based on a preset first text conversion mapping table, the key state features in the real-time state vector of the power grid are converted into a first natural language description. Based on a preset second text conversion mapping table, the control instructions in the scheduling action information are converted into a second natural language description. The first natural language description and the second natural language description are filled into a preset explanation prompt word template to generate an input sequence to be processed; The sequence to be processed is input into the large language model to generate a natural language decision explanation text.

7. The power grid real-time dispatch decision-making method based on large-model collaborative reinforcement learning as described in claim 6, characterized in that, The step of inputting the sequence to be processed into the large language model to generate natural language decision explanation text includes: The input sequence to be processed is input into the large language model, and the attention contribution of each state variable in the sequence to be processed is calculated using the attention mechanism of the large language model. Based on the attention contribution, the dominant state factor of the scheduling action information is identified, and a causal relationship between the state and the action is established based on the dominant state factor. Based on a preset cluster search algorithm and causal relationship, word-by-word decoding is performed to generate natural language decision explanation text, which includes working condition description text, operation logic description text, and expected effect description text.

8. A real-time power grid dispatching decision-making method based on large-scale collaborative reinforcement learning as described in any one of claims 1 to 7, characterized in that, After inputting the sequence to be processed into the large language model to generate natural language decision explanation text, and performing real-time scheduling of the power grid system based on the natural language decision explanation text, the method further includes: Obtain the environmental feedback status of the power grid system; The real-time semantic feature vector, the real-time power grid state vector, the scheduling action information, and the environmental feedback state are concatenated to construct a joint feature vector; The joint feature vector is input into the large language model to obtain reward feedback information; The original reward information is obtained based on the reward function and the environmental feedback state; The reward feedback information and the original reward information are weighted and fused to obtain fused reward information, and the reinforcement learning agent is iteratively optimized based on the fused reward information.

9. A real-time power grid dispatching decision-making device based on large-scale collaborative reinforcement learning, characterized in that, Includes a module for performing the method as described in any one of claims 1 to 8.

10. A real-time power grid dispatching decision-making device based on large-scale collaborative reinforcement learning, characterized in that, The device includes a processor and an interface circuit, wherein the interface circuit is used to receive signals from other external devices and transmit them to the processor or to send signals from the processor to other external devices, and the processor is used to implement the method as described in any one of claims 1 to 8 through logic circuits or execution code instructions.

Images