Power grid dispatching decision assistance method and system based on multi-modal large model

Abstract

The application provides a power grid dispatching decision auxiliary method and system based on a multi-modal large model, relates to the technical field of intelligent power grid dispatching assistance, and first acquires multi-modal dispatching reference data streams from a power grid dispatching center, then respectively performs adaptive feature coding on three kinds of modal data through three special coding branches of a pre-constructed dispatching scene multi-modal large model to generate corresponding hidden state sequences; subsequently generates a multi-modal joint hidden state that fuses a dispatching scene context through cross-modal collaborative reasoning, finally decodes to generate a target dispatching decision scheme containing complete dispatching operation steps and execution time points, and sends the target dispatching decision scheme to a power grid dispatching execution terminal for use by dispatching personnel. The application can fully tap the correlation value of multi-modal dispatching information, improve the accuracy and generation efficiency of power grid dispatching auxiliary decision, and adapt to the dispatching decision needs of complex large power grids.

Description

Technical Field

[0001] This invention relates to the field of intelligent power grid dispatching support technology, and more specifically, to a power grid dispatching decision support method and system based on a multimodal large model. Background Technology

[0002] Power grid dispatching decisions require the integration of multi-source information, including real-time power grid operating status, dispatcher intentions, and historical dispatching experience, to support decision-making. This multi-source information encompasses various modalities such as images, voice, and text. Traditional power grid dispatching decision support methods mostly process different modalities separately before manually integrating the information to generate a decision plan. This approach is not only inefficient but also susceptible to biases due to the subjective experience of dispatchers. Existing methods often only process single-modal information or simply concatenate different modalities before inputting them into the model. They lack dedicated coding branches for the feature attributes of different modalities, failing to fully extract effective features. Some multimodal fusion methods only perform simple feature concatenation, failing to achieve true cross-modal collaborative reasoning and thus unable to uncover implicit relationships between different modalities. Consequently, they struggle to generate accurate dispatching decision plans that meet the actual needs of power grid operation. Furthermore, existing methods do not integrate and utilize the three core dispatching data types—power grid topology images, dispatching voice commands, and historical dispatching text—failing to cover all core information dimensions in the dispatching scenario. This results in decision plans with missing information, failing to meet the dispatching decision support needs of complex power grid operation scenarios. Summary of the Invention

[0003] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a power grid dispatching decision support method based on a multimodal large model, the method comprising:

[0004] The system acquires a multimodal dispatch reference data stream collected by the power grid dispatch center. The multimodal dispatch reference data stream includes a real-time electrical topology map image stream characterizing the power grid topology operation status, a dispatch voice command stream recording dispatcher dispatch instructions, and a dispatch text report stream recording historical dispatch operation records.

[0005] The multimodal scheduling reference data stream is input into a pre-constructed multimodal scheduling scenario model. The topology graph encoding branch in the multimodal scheduling scenario model performs topology operation status encoding on the real-time electrical topology graph image stream to generate a topology operation status hidden state sequence. The speech encoding branch in the multimodal scheduling scenario model performs instruction intent encoding on the scheduling speech command stream to generate a scheduling instruction intent hidden state sequence. The text encoding branch in the multimodal scheduling scenario model performs historical scheduling experience encoding on the scheduling text report stream to generate a historical scheduling experience hidden state sequence.

[0006] The multimodal fusion reasoning branch in the multimodal large model of the scheduling scenario performs cross-modal collaborative reasoning processing on the hidden state sequence of the topology operation status, the hidden state sequence of the scheduling instruction intent, and the hidden state sequence of the historical scheduling experience, generating a multimodal joint reasoning hidden state sequence of the fusion scheduling scenario context.

[0007] The scheduling decision decoding branch in the multimodal large model of the scheduling scenario performs scheduling decision generation processing on the hidden state sequence of the multimodal joint inference to generate a target scheduling decision scheme. The target scheduling decision scheme includes a sequence of scheduling operation steps and a set of scheduling operation execution time points.

[0008] The target scheduling decision scheme is sent to the power grid scheduling execution terminal.

[0009] Furthermore, embodiments of the present invention also provide a power grid dispatching decision support system based on a multimodal large model, comprising:

[0010] A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to execute the aforementioned power grid dispatch decision support method based on a multimodal large model by executing the machine-executable instructions.

[0011] Based on the above, and targeting the multimodal core data types in power grid dispatching scenarios, this paper acquires multimodal dispatching reference data streams containing real-time electrical topology image streams, dispatching voice command streams, and dispatching text report streams. By setting dedicated encoding branches for the three modalities in a pre-constructed multimodal model of the dispatching scenario, feature encoding processing is performed on different modalities. This adapts to the feature attributes of different modalities, fully extracting the effective dispatching information contained in each modality. Cross-modal collaborative reasoning is completed through a multimodal fusion reasoning branch, generating a multimodal joint reasoning hidden state sequence for the fused dispatching scenario context. This fully utilizes multi-source... The complementarity of information, compared to simple feature splicing, can obtain fused features that are more in line with the scheduling scenario; by directly generating a target scheduling decision scheme containing a sequence of scheduling operation steps and a set of execution time points through the scheduling decision decoding branch, and sending it to the scheduling execution terminal, it realizes end-to-end processing from multimodal information input to directly executable scheduling decision output, without the need for manual information integration, which greatly improves the efficiency of scheduling decision assistance, while ensuring the integrity and accuracy of the decision scheme. It makes full use of multimodal scheduling information, improves the accuracy and generation efficiency of power grid scheduling assistance decision, and can adapt to the scheduling decision needs under complex power grid operation scenarios. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the execution flow of the power grid dispatching decision support method based on a multimodal large model provided in an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the multimodal data stream processing and feature fusion architecture provided in the embodiments of this application.

[0014] Figure 3 This is a schematic diagram of the evolution curves of KL divergence and asymmetric exponent provided in the embodiments of this application.

[0015] Figure 4 This is a schematic diagram of the evolution curve of the continuous homology Betti number provided in the embodiments of this application.

[0016] Figure 5 This is a schematic diagram of the convergence characteristic curve of the Lyapunov function provided in the embodiments of this application. Detailed Implementation

[0017] Figure 1 This is a flowchart illustrating a power grid dispatching decision support method based on a multimodal large model, provided by an embodiment of the present invention. The method runs on a computing server in a power grid dispatching center. This computing server is equipped with a graphics processor and memory. The graphics processor loads pre-built network weights of a multimodal large model of the dispatching scenario. This method generates a target dispatching decision scheme by performing multimodal joint encoding and inference on real-time electrical topology image streams, dispatching voice command streams, and dispatching text report streams, thereby assisting dispatchers in making decisions.

[0018] Step S110: Obtain the multimodal dispatch reference data stream collected by the power grid dispatch center. The multimodal dispatch reference data stream includes a real-time electrical topology map image stream that characterizes the power grid topology operation status, a dispatch voice command stream that records dispatcher dispatch instructions, and a dispatch text report stream that records historical dispatch operation records.

[0019] In this embodiment, the data bus of the power grid dispatch center continuously receives real-time electrical topology image streams pushed by the data acquisition and monitoring system. These real-time electrical topology image streams are time-series images, one frame per second. Each frame is a digital snapshot of the power grid's main wiring diagram, on which devices such as buses, circuit breakers, disconnectors, and transformers are drawn using standard graphic symbols, and the energized status coloring and power flow direction arrows of the devices are updated in real time. The dispatch voice command stream is acquired through a microphone array on the dispatch console, consisting of audio signals of the dispatcher's spoken operation instructions, with a sampling rate of 16 kHz, a bit depth of 16 bits, and mono audio. The dispatch text report stream comes from the dispatch operation ticket system and the historical event record database, and is a formatted text document containing the background, steps, and effect summary of the dispatch operation; the file format is plain text. After timestamp alignment, the above three modal data streams are packaged into a multimodal dispatch reference data stream for subsequent processing. All data acquisition is conducted within the dispatch dedicated network in accordance with the power monitoring system security protection regulations, and has been anonymized, concealing specific plant / station names and dispatcher identity information.

[0020] Step S120: Input the above-mentioned multimodal scheduling reference data stream into the pre-constructed multimodal scheduling scenario model. The topology graph encoding branch in the multimodal scheduling scenario model performs topology operation status encoding processing on the above-mentioned real-time electrical topology graph image stream to generate a topology operation status hidden state sequence. The speech encoding branch in the multimodal scheduling scenario model performs instruction intent encoding processing on the above-mentioned scheduling speech instruction stream to generate a scheduling instruction intent hidden state sequence. The text encoding branch in the multimodal scheduling scenario model performs historical scheduling experience encoding processing on the above-mentioned scheduling text report stream to generate a historical scheduling experience hidden state sequence.

[0021] In this embodiment, the pre-built multimodal large model of the scheduling scenario is a heterogeneous neural network consisting of multiple encoding branches, a fusion inference branch, and a decoding branch. The network weights of this heterogeneous neural network have been obtained through pre-training and domain fine-tuning. During the inference phase, the multimodal scheduling reference data stream is split into three parallel encoding branches. The computation between branches is independent and executed in parallel.

[0022] Step S121: Extract the real-time electrical topology image stream from the multimodal scheduling reference data stream, input each frame of the real-time electrical topology image stream into the graphic element parsing layer of the topology encoding branch, and perform graphic element recognition processing on the bus connection lines, circuit breaker symbols, disconnector symbols, transformer symbols and load label text in the real-time electrical topology image to obtain the set of graphic elements contained in the real-time electrical topology image.

[0023] The graphic element parsing layer internally includes an anchor-bound object detection network. The backbone of this network is a residual convolutional network, and the neck uses a feature pyramid structure. The detection head outputs the bounding box coordinates and class confidence score for each graphic element. The input is a two-dimensional digital image matrix with a shape of height x width x three channels. The output is a set of graphic elements, where each graphic element contains a graphic element type identifier, the coordinates of its top-left corner, and the coordinates of its bottom-right corner. For load label text, the graphic element parsing layer also integrates an optical character recognition (OCR) module to convert the text region image into character encoding. For example, in a frame of a real-time electrical topology diagram image, the graphic element parsing layer detects a circuit breaker symbol numbered "QF101," whose graphic element type identifier is "circuit breaker," and whose top-left corner coordinates are pixel positions (120, 300) and bottom-right corner coordinates are pixel positions (180, 360). Simultaneously, it detects the text label "QF101," and the OCR module outputs a character encoding sequence.

[0024] Step S122: Extract graphic element attributes for each graphic element in the above graphic element set to obtain the graphic element type identifier, the planar coordinate position in the above real-time electrical topology diagram image, and the list of adjacent graphic elements that are connected to each graphic element. Combine the graphic element type identifier, planar coordinate position, and list of adjacent graphic elements of all graphic elements in the above graphic element set to form a topology diagram structured description record.

[0025] For each identified graphic element, the geometric center coordinates of its bounding box are taken as its planar coordinate position in the real-time electrical topology diagram image. Next, a line segment detection algorithm is used to extract all connecting line segments in the real-time electrical topology diagram image. For each connecting line segment, the nearest distance between its two endpoints and the bounding boxes of each graphic element is calculated. If this nearest distance is less than a preset distance threshold, the connecting line segment is determined to be associated with the graphic element. Two graphic elements sharing the same connecting line segment are identified as having a connection relationship. All graphic elements are traversed, and the identifiers of graphic elements with a connection relationship are added to the list of adjacent graphic elements. For example, the planar coordinate position of graphic element "QF101" is pixel (150, 330). Through connecting line segment tracing, it is determined that it has a connection relationship with "Isolating Switch QS11" on the left and "Bus W1" on the right. Therefore, its adjacent graphic element list is "Isolating Switch QS11" and "Bus W1". Combine the graphic element type identifiers, planar coordinate positions, and lists of adjacent graphic elements of all graphic elements to form a structured description record of the topology graph organized in key-value pairs.

[0026] Step S123: Arrange the topology structured description records of multiple consecutive frames of real-time electrical topology images in chronological order to form a topology structured description record time series. Perform graphic element state change comparison processing on two adjacent frames of topology structured description records in the above topology structured description record time series. Identify the first type of changed element set where the graphic element type identifier changes, the second type of changed element set where the planar coordinate position of the graphic element changes, and the third type of changed element set where the list of adjacent graphic elements of the graphic element changes.

[0027] The topological graph structured description records at time t and t+1 are taken. Using the graphic element identifier as the key, the attributes of the same graphic element identifier corresponding to the same graphic element identifier are compared in the records of the two time points. If a graphic element identifier exists at time t but not at time t+1, or vice versa, it is classified into the first type of changing element set. For graphic element identifiers that exist at both time points, the Euclidean distance of their planar coordinate positions is calculated. If the Euclidean distance is greater than the position change threshold, it is classified into the second type of changing element set. The differences between the sets of adjacent graphic element lists are compared. If there are added or disappeared connections, they are classified into the third type of changing element set.

[0028] Step S124: Merge the first set of changing elements, the second set of changing elements, and the third set of changing elements to obtain a set of electrical topology state change events between adjacent frames. Assign an event type label to each electrical topology state change event in the set of electrical topology state change events between adjacent frames. The event type label includes a device switching event label, a topology reconfiguration event label, and a power flow migration event label.

[0029] The three sets of changing elements are deduplicated and merged, as the same graphic element may appear in multiple sets simultaneously. After merging, each independent instance of graphic element change constitutes an electrical topology state change event. The event is classified according to the type and change attributes of the graphic elements in the event. If the graphic element type identifier changes from none to present or from present to none, and the graphic element is a circuit breaker or disconnector, a device switching event is assigned. If the list of adjacent graphic elements of a graphic element changes, involving a change in bus connection relationships, a topology reconfiguration event is assigned. If the graphic element type identifier and the list of adjacent graphic elements remain unchanged, but only the planar coordinate position changes, a power flow migration event is assigned, indicating that the electrical topology state change event is caused by a movement of the power flow direction or the position of the load label.

[0030] Step S125: The set of electrical topology state change events between adjacent frames with the event type label is sequentially input into the topology state evolution recursive network of the topology graph encoding branch according to the time sequence of the time sequence recorded by the structured description of the topology graph. The topology state evolution recursive network encodes the causal dependency of the electrical topology state change events in the time dimension to generate the hidden state of the topology operation status at each time step.

[0031] The topology state evolution recurrent network is a sequence encoder based on gated recurrent units. At each time step, the input contains a feature vector representation of the electrical topology state change events of the current time step. This feature vector is composed of an embedding vector of the graphical element identifiers involved in the event, a one-hot encoded vector of the event type label, and a vector of the changed graphical element attributes. The gated recurrent unit of the topology state evolution recurrent network maintains a hidden state vector, which is updated at each step based on the current input and the hidden state vector of the previous step. The update process is controlled by update gates and reset gates, allowing the hidden state vector to selectively memorize long-term event dependencies. For example, after a device switching event occurs, the cascading topology reconfiguration events it triggers will be progressively encoded into the hidden state vector in subsequent time steps.

[0032] Step S126: Combine the hidden states of the topology operation status at all time steps output by the above topology state evolution recursive network into a hidden state sequence of topology operation status according to time order, and output the hidden state sequence of topology operation status to the output interface of the above topology graph encoding branch.

[0033] The hidden state vector at each time step serves as the hidden state of the topological operation at that time step. All the hidden state vectors of the topological operation at time steps 1 to T are stacked sequentially to form a matrix of shape T multiplied by the dimension of the hidden state vectors. This matrix is ​​the sequence of hidden states of the topological operation and is passed to the subsequent multimodal fusion inference branch.

[0034] Step S131: Extract the scheduling voice command stream from the multimodal scheduling reference data stream, input the scheduling voice command stream into the speech signal preprocessing layer of the speech coding branch, perform silence segment removal processing and speech activity endpoint detection processing on the scheduling voice command stream, and obtain a set of effective scheduling voice segments with silence segments removed.

[0035] The speech signal preprocessing layer first segments the input mono audio signal into 20-millisecond frames with a frame shift of 10 milliseconds. The short-time energy and zero-crossing rate of each frame are calculated. Continuous frame sequences with short-time energy below an energy threshold and a zero-crossing rate below a zero-crossing rate threshold are identified as silent segments and removed. For non-silent regions, a speech activity endpoint detector based on a deep neural network is used for precise start and end point localization, segmenting multiple continuous, effective scheduling speech segments containing valid speech content.

[0036] Step S132: Perform acoustic feature extraction processing on each effective scheduled speech segment in the above set of effective scheduled speech segments, extract the Mel frequency cepstral coefficient feature sequence and the fundamental frequency change trajectory feature sequence of each effective scheduled speech segment, and concatenate the above Mel frequency cepstral coefficient feature sequence and the above fundamental frequency change trajectory feature sequence along the feature dimension to generate the acoustic feature matrix of each effective scheduled speech segment.

[0037] For each valid scheduled speech segment, 40-dimensional Mel-frequency cepstral coefficient features are extracted with a window length of 25 milliseconds and a frame shift of 10 milliseconds, forming a Mel-frequency cepstral coefficient feature sequence with a shape equal to the number of frames multiplied by 40. Simultaneously, the fundamental frequency value is extracted frame-by-frame using an autocorrelation algorithm, forming a fundamental frequency change trajectory feature sequence with a shape equal to the number of frames multiplied by 1. Each frame vector of the Mel-frequency cepstral coefficient feature sequence is concatenated with its corresponding fundamental frequency value in the last dimension, forming a 41-dimensional frame-level feature vector. The 41-dimensional vectors from all frames are combined to form the acoustic feature matrix of the valid scheduled speech segment, with a shape equal to the number of frames multiplied by 41.

[0038] Step S133: Input the acoustic feature matrix of each effective scheduled speech segment into the scheduling term acoustic encoder of the speech coding branch, perform convolutional downsampling and self-attention coding on the acoustic feature matrix, and generate a scheduling term acoustic embedding vector sequence for each effective scheduled speech segment. Each scheduling term acoustic embedding vector in the scheduling term acoustic embedding vector sequence corresponds to a pronunciation unit in the effective scheduled speech segment.

[0039] The scheduling terminology acoustic encoder consists of two cascaded temporal convolutional layers and a multi-layer Transformer encoder. The input acoustic feature matrix first passes through two temporal convolutional layers, each with a kernel size of 3 and a stride of 2, downsampling the temporal dimension to one-quarter of the original frame number. Subsequently, the downsampled acoustic feature sequence, after being encoded with positional parameters, is fed into a 6-layer Transformer encoder. Each layer of the Transformer encoder contains a multi-head self-attention mechanism and a feedforward neural network. The self-attention mechanism calculates the attention weights between each time-step feature and features from all other time-steps, and aggregates contextual information accordingly. The encoder outputs a sequence of scheduling terminology acoustic embedding vectors with a shape equal to one-quarter of the frame number multiplied by the hidden dimension. Each embedding vector at each time step corresponds to approximately 40 milliseconds of articulation units in the original speech.

[0040] Step S134: Input the above-mentioned scheduling term acoustic embedding vector sequence into the scheduling domain adaptation connection temporal classifier of the above-mentioned speech coding branch, and perform scheduling domain term sequence decoding processing on the above-mentioned scheduling term acoustic embedding vector sequence to obtain the scheduling term text sequence corresponding to each valid scheduling speech segment. The scheduling term text sequence includes scheduling device name words, scheduling operation action words, and scheduling magnitude description words.

[0041] The scheduling domain-adaptive connection-time classifier is a structure consisting of a linear classification layer and a connection-time classification decoding algorithm. The weight matrix of the linear classification layer is fine-tuned by constraints from a scheduling domain terminology dictionary, and the output number of categories is the size of the scheduling domain terminology dictionary plus a blank marker. The input sequence of scheduling terminology acoustic embedding vectors is processed by the linear classification layer to obtain the posterior probability distribution of each frame on the terminology vocabulary. Using the beam search strategy of the connection-time classification decoding algorithm, the path of the word sequence with the highest probability is searched, blank markers and duplicate words are removed, and the final scheduling terminology text sequence is output. For example, the output text sequence might be "pull open", "Wang Jinxian 101 circuit breaker", or "from operation to maintenance".

[0042] Step S135: Perform part-of-speech tagging and scheduling semantic role tagging on each word in the above scheduling terminology text sequence, identify the device name words that act as scheduling device name semantic roles, the operation action words that act as scheduling operation action semantic roles, and the magnitude description words that act as scheduling magnitude description semantic roles in the above scheduling terminology text sequence, and construct the scheduling semantic structure triplet for each valid scheduling speech segment based on the recognition results.

[0043] A pre-trained part-of-speech tagging model for the power industry is used to tag text sequences with parts of speech. Then, a scheduling semantic role labeling is performed based on a sequence labeling model. This model uses a bidirectional long short-term memory network plus a conditional random field structure to assign a semantic role label to each word in the sequence. The label set includes "equipment name words," "operation action words," "magnitude description words," and "others." Based on the labels, the corresponding words are extracted from the text sequence. The extracted equipment name words, operation action words, and magnitude description words are combined into a triplet data structure, namely the scheduling semantic structure triplet. For example, for the text "open the Wangjinxian 101 circuit breaker," the triplet is constructed as "equipment name word: Wangjinxian 101 circuit breaker, operation action word: open, magnitude description word: none."

[0044] Step S136: Input the above scheduling semantic structure triplet into the instruction intent encoder of the above speech coding branch, perform semantic combination encoding on the device name word, operation action word and magnitude description word in the above scheduling semantic structure triplet, generate the scheduling instruction intent semantic vector of each valid scheduling speech segment, arrange the above scheduling instruction intent semantic vector along the time order of the above valid scheduling speech segments, generate the scheduling instruction intent hidden state sequence, and output the above scheduling instruction intent hidden state sequence to the output interface of the above speech coding branch.

[0045] Internally, the instruction intent encoder first maps device name words, action words, and magnitude description words to pre-trained word embedding vectors. Then, these vectors are concatenated in the order of "device name word embedding vector, action word embedding vector, magnitude description word embedding vector" to obtain a combined vector. This combined vector is semantically encoded using a multilayer perceptron (MLP), which contains two fully connected layers with a linear rectified activation function in between. The output vector of the MLP serves as the scheduling instruction intent semantic vector. All scheduling instruction intent semantic vectors in the temporal order of a valid scheduling speech segment are arranged to form a scheduling instruction intent hidden state sequence, with a shape equal to the scalar product of the dimensions of the valid scheduling speech segments and the scheduling instruction intent semantic vectors.

[0046] Step S141: Extract the scheduling text report stream from the multimodal scheduling reference data stream, input each scheduling text report in the scheduling text report stream into the report chapter parsing layer of the text encoding branch, perform chapter boundary detection processing on the scheduling text report, identify the scheduling background description chapter, scheduling operation record chapter, and scheduling effect summary chapter of the scheduling text report, and extract the plain text content of the scheduling operation record chapter as the scheduling operation record text.

[0047] The report parsing layer operates on a rule-based matching method. It sequentially scans the text content, using predefined regular expression patterns to match chapter titles, such as keywords like "Item X," "Operation Task," and "Operation Steps." After identifying a chapter title, it divides subsequent content into corresponding chapters based on the title type until the next chapter title appears. It then extracts all text from the identified "Schedule Operation Record Chapter," removing any possible table lines and serial numbers to form a plain text version of the schedule operation record.

[0048] Step S142: Perform sentence segmentation and scheduling entity word recognition processing on the above scheduling operation record text, extract the scheduling equipment name entity words, scheduling operation action entity words, scheduling operation time entity words, and electrical quantity change description entity words before and after the scheduling operation that appear in the above scheduling operation record text, and construct a scheduling operation event entity association graph according to the co-occurrence relationship of the extracted entity words in the above scheduling operation record text.

[0049] The scheduling operation record text is segmented into sentences using punctuation marks such as periods and semicolons. Within each sentence, a named entity recognition model is applied to extract entities. This model employs a bidirectional long short-term memory network combined with a conditional random field structure. Its training corpus is labeled with entity categories such as "equipment name entity words," "operation action entity words," "operation time entity words," and "electrical quantity change description entity words." Entities appearing in the same extracted sentence are considered to have a co-occurrence relationship. A graph is constructed, with each entity word as a node. If two entity words co-occur in the same sentence, an edge is established between them. The attributes of a node include the entity word text and entity word category, while the attribute of an edge is the co-occurrence relationship.

[0050] Step S143: Perform node attribute encoding processing on each scheduling device name entity node in the above scheduling operation event entity association diagram, convert the topology location number and voltage level type of the above scheduling device name entity node in the power grid topology into a node location attribute vector, and attach the node location attribute vector of the above scheduling device name entity node to the corresponding node in the above scheduling operation event entity association diagram.

[0051] For each dispatchable device name entity node, the static topology database of the power grid is queried to obtain its unique topology location number in the power grid topology structure, such as the node number within the substation, and its voltage level type, such as 500 kV or 220 kV. The topology location number is converted into a location embedding vector through an embedding layer. The voltage level type is converted into a voltage level vector through one-hot encoding. The location embedding vector and the voltage level vector are concatenated to obtain the node location attribute vector. This node location attribute vector is then assigned to the "location attribute" field of the corresponding node in the dispatch operation event entity association graph.

[0052] Step S144: Input the scheduling operation event entity association graph with the additional node location attribute vector into the historical event graph encoder of the above text encoding branch, perform multi-hop neighbor information aggregation processing on the scheduling operation event entity association graph, obtain the neighbor context aggregation vector of each scheduling device name entity word node in the scheduling operation event entity association graph, and combine the neighbor context aggregation vectors of all scheduling device name entity word nodes in the scheduling operation event entity association graph into a historical scheduling event graph-level representation vector.

[0053] The historical event graph encoder is a graph isomorphic network. This graph isomorphic network contains K layers of graph convolutional layers. In each layer, for each node in the graph, message passing is performed: the feature vectors of all its neighboring nodes in the previous layer are aggregated, added to its own feature vector, and then updated through a multilayer perceptron and a non-linear activation function. After K iterations, the feature vector of each node aggregates the context information within its K-hop neighbor range. The feature vectors of all scheduling device name entity word nodes in the graph after K iterations are taken as their neighbor context aggregation vectors. The neighbor context aggregation vectors of all scheduling device name entity word nodes are stacked along a new dimension to form the historical scheduling event graph-level representation vector.

[0054] Step S145: Arrange the historical scheduling event graph-level representation vectors corresponding to multiple consecutive scheduling operation record texts in the same scheduling text report according to the scheduling operation time order, and input them into the historical experience sequence modeler of the above text encoding branch. The historical experience sequence modeler encodes the success and failure mark transfer rules and operation interval duration rules between adjacent scheduling operation events in the historical scheduling event graph-level representation vector sequence to generate the historical scheduling experience hidden state of each scheduling operation event.

[0055] The historical experience sequence modeler employs a bidirectional long short-term memory network. The input sequence is a sequence of graph-level representation vectors of historical scheduling events ordered chronologically within the same report. For each time step in the sequence, its input vector is composed of the historical scheduling event graph-level representation vector of the current scheduling operation event and two auxiliary feature vectors. Auxiliary feature vector one is the embedding vector of the "success" or "failure" flag of the previous operation result, extracted from the scheduling effect summary section. Auxiliary feature vector two is the log-normalized value of the interval between the current operation and the previous operation. The bidirectional long short-term memory network reads the entire sequence and outputs a historical scheduling experience hidden state at each time step, concatenating the forward and backward hidden states.

[0056] Step S146: Combine the historical scheduling experience hidden states of all step scheduling operation events output by the above historical experience sequence modeler into a historical scheduling experience hidden state sequence according to the scheduling operation time order, and output the above historical scheduling experience hidden state sequence to the output interface of the above text encoding branch.

[0057] The historical scheduling experience hidden state vectors output at each time step of the bidirectional long short-term memory network are arranged in chronological order of operation time to form a sequence of historical scheduling experience hidden states, the shape of which is the sequence length multiplied by the dimension of the historical scheduling experience hidden state vectors.

[0058] Step S150: The multimodal fusion reasoning branch in the multimodal large model of the above scheduling scenario performs cross-modal collaborative reasoning processing on the hidden state sequence of the above topological operation status, the hidden state sequence of the above scheduling instruction intent, and the hidden state sequence of the above historical scheduling experience, to generate the multimodal joint reasoning hidden state sequence of the fusion scheduling scenario context.

[0059] The multimodal fusion inference branch is the core of the multimodal large model for scheduling scenarios, responsible for the deep fusion of the outputs of the three single-modal coding branches. This step adopts a phased, multi-round attention interaction mechanism to sequentially achieve bidirectional alignment of situation and instructions, experience enhancement, and causal inference.

[0060] Step S151: Input each hidden state of the topology operation status in the above-mentioned hidden state sequence of topology operation status and the time-aligned hidden state of the scheduling instruction intent in the above-mentioned hidden state sequence of scheduling instruction intent into the topology intent cross-attention layer of the above-mentioned multimodal fusion inference branch, calculate the first cross-attention weight of each hidden state of topology operation status to each hidden state of scheduling instruction intent, and use the first cross-attention weight to perform a weighted summation on the above-mentioned hidden state sequence of scheduling instruction intent to generate an intent-aware topology operation status correction hidden state sequence.

[0061] Before input, a time alignment operation ensures that the hidden state of topological operational status at each time step is paired with the hidden state of scheduling instruction intent at the same or nearest time step. In the topological intent cross-attention layer, the hidden state of topological operational status serves as the query vector, and the hidden state of scheduling instruction intent serves as the key vector and value vector. The dot product similarity between the query vector and all key vectors is calculated, and after softmax normalization, the first cross-attention weight is obtained. This first cross-attention weight is used to perform a weighted summation on all value vectors, and the result is added to the original query vector to form the intent-aware corrected hidden state of topological operational status. The corrected hidden states of all time steps are combined into an intent-aware corrected hidden state sequence of topological operational status.

[0062] Step S152: Input each hidden state of scheduling instruction intent in the above-mentioned hidden state sequence of hidden state ...

[0063] The operation of the intention-topology cross-attention layer is symmetrical to step S151. The hidden state of the scheduling instruction intention is used as the query vector, and the hidden state of the topology operation status is used as the key vector and value vector. The second cross-attention weight is calculated, and the hidden state sequence of the topology operation status is weighted and summed. The result is added to the original hidden state of the scheduling instruction intention to generate the topology-aware modified hidden state of the scheduling instruction intention. The modified hidden states of all time steps are combined into the topology-aware sequence of modified hidden states of the scheduling instruction intention.

[0064] Step S153: Input the above-mentioned intention-aware topology operation status correction hidden state sequence and the above-mentioned topology-aware scheduling instruction intention correction hidden state sequence into the fusion gating unit of the above-mentioned multimodal fusion inference branch, and perform gating fusion processing on the intention-aware topology operation status correction hidden state and the topology-aware scheduling instruction intention correction hidden state at each time step to generate the topology instruction dual-modal fusion hidden state at each time step.

[0065] The fusion gating unit operates in parallel at each time step. For time step t, it concatenates the intention-aware topology operational status correction hidden state and the topology-aware scheduling instruction intention correction hidden state, passes them through a fully connected layer and a sigmoid activation function, and outputs a gating coefficient vector with the same dimension as both. Using this gating coefficient vector, the correction hidden states of the two modalities are weighted and combined through element-wise multiplication to generate the topology instruction bimodal fusion hidden state. The gating mechanism dynamically determines whether to rely more on topology status information or instruction intention information at each time step.

[0066] Step S154: Input each topology instruction bimodal fusion hidden state in the above topology instruction bimodal fusion hidden state sequence and the historical scheduling experience hidden state corresponding to the scheduling device name entity node in the above historical scheduling experience hidden state sequence into the experience-enhanced cross-attention layer of the above multimodal fusion inference branch. Calculate the third cross-attention weight of each topology instruction bimodal fusion hidden state to each historical scheduling experience hidden state. Use the third cross-attention weight to perform a weighted summation on the above historical scheduling experience hidden state sequence to generate an experience-enhanced topology instruction fusion hidden state sequence.

[0067] This step incorporates historical scheduling experience information related to the target device of the current instruction by querying the entity words of the scheduling device name. For each bimodal fusion hidden state of a topology instruction, the associated entity words of the scheduling device name are extracted, and all historical scheduling experience hidden states of that entity word node are retrieved from the historical scheduling experience hidden state sequence. Using the bimodal fusion hidden state of the topology instruction as the query vector, and the retrieved set of historical scheduling experience hidden states as the key vector and value vector, the third cross-attention weight is calculated and weighted, resulting in an experience enhancement vector. The experience enhancement vector is then residually joined with the original bimodal fusion hidden state of the topology instruction to generate an experience-enhanced topology instruction fusion hidden state.

[0068] Step S155: Input the above-mentioned experience-enhanced topology instruction fusion hidden state sequence into the causal Transformer inference layer of the above-mentioned multimodal fusion inference branch. Through the causal self-attention masking mechanism of the causal Transformer inference layer, encode the causal dependency relationship of the scheduling operation between the fusion hidden state of the preceding time step and the fusion hidden state of the following time step in the above-mentioned experience-enhanced topology instruction fusion hidden state sequence to generate the multimodal joint inference hidden state of each time step.

[0069] The causal Transformer inference layer consists of multiple stacked Transformer decoder layers with causal masks. In each layer, during the self-attention mechanism computation, information from future time steps is masked using an upper triangular matrix, ensuring that each time step only focuses on the current and past time steps. This forces the model to learn causal dependencies between operation steps, meaning subsequent operations depend on previous operations and states. The empirically enhanced topological instruction fusion hidden state sequence, after processing through multiple layers of causal self-attention and feedforward networks, outputs a feature sequence rich in context and causal inference information.

[0070] Step S156: Combine the hidden states of the multimodal joint inference of all time steps output by the above causal Transformer inference layer into a multimodal joint inference hidden state sequence in chronological order, and output the above multimodal joint inference hidden state sequence to the output interface of the above multimodal fusion inference branch.

[0071] The output sequence of the last layer of the causal Transformer inference layer is the multimodal joint inference hidden state sequence. Each vector in this sequence integrates the current and historical topological situation, instruction intent, and historical experience, and encodes the causal dependencies between operations.

[0072] Step S160: The scheduling decision decoding branch in the multimodal large model of the above scheduling scenario performs scheduling decision generation processing on the hidden state sequence of the above multimodal joint inference to generate a target scheduling decision scheme. The target scheduling decision scheme includes a sequence of scheduling operation steps and a set of scheduling operation execution time points.

[0073] The scheduling decision decoding branch receives the hidden state sequence of multimodal joint inference and decodes three core decision elements in parallel: operation step sequence, operation target device, and operation execution time.

[0074] Step S161: Input the above multimodal joint reasoning hidden state sequence into the scheduling step sequence decoder of the above scheduling decision decoding branch. The scheduling step sequence decoder performs scheduling operation step label classification processing on each multimodal joint reasoning hidden state in the above multimodal joint reasoning hidden state sequence, generates the scheduling operation step label corresponding to each time step, and combines the scheduling operation step labels of consecutive time steps into a scheduling operation step sequence.

[0075] The scheduling step sequence decoder is a linear classifier. The input multimodal joint inference latent state vector for each time step is mapped to a scheduling operation step label space through a fully connected layer. This space is equal in size to the predefined set of scheduling operation step labels. The probability of each label is calculated using the softmax function, and the label with the highest probability is selected as the scheduling operation step label for that time step. A sequence of consecutive, non-empty scheduling operation step labels constitutes the scheduling operation step sequence.

[0076] Step S162: Extract the multimodal joint inference hidden states from the multimodal joint inference hidden state sequence that are time-aligned with the labels of each scheduling operation step in the scheduling operation step sequence, and input them into the scheduling equipment location decoder of the scheduling decision decoding branch. The scheduling equipment location decoder performs position prediction processing on the corresponding scheduling equipment node in the power grid topology for each multimodal joint inference hidden state, and generates the target scheduling equipment node number corresponding to each scheduling operation step label.

[0077] The scheduling device location decoder is a multi-classifier. It takes a multimodal joint inference hidden state vector as input and maps it through a fully connected layer to a vector with a dimension equal to the total number of operable device nodes in the power grid topology. The softmax function is used to obtain the probability distribution of each device node. The node number with the highest probability is selected as the target scheduling device node number for that scheduling operation step.

[0078] Step S163: Input the multimodal joint inference hidden state in the above multimodal joint inference hidden state sequence that is time-aligned with the label of each scheduling operation step in the above scheduling operation step sequence into the scheduling operation execution timing decoder of the above scheduling decision decoding branch. The scheduling operation execution timing decoder performs operation execution time point offset prediction processing on each multimodal joint inference hidden state to generate the operation execution time offset corresponding to each scheduling operation step label.

[0079] The scheduling operation execution timing decoder is a regressor. It takes a multimodal joint inference hidden state vector as input, passes through a fully connected layer, and outputs a scalar value representing the offset of the operation execution time relative to the current reference time, in seconds.

[0080] Step S164: Obtain the current time base value of the power grid dispatch center, sum the operation execution time offset corresponding to each dispatch operation step label with the current time base value of the power grid dispatch center to obtain the absolute execution time point corresponding to each dispatch operation step label, and combine the absolute execution time points corresponding to all dispatch operation step labels into a set of dispatch operation execution time points.

[0081] The current Coordinated Universal Time (UTC) timestamp is obtained from the time synchronization system of the power grid dispatch center and used as the time base value. The execution time offset of each operation output in step S163 is added to this time base value to obtain the absolute execution time point.

[0082] Step S165: Combine the above-mentioned scheduling operation step sequence, the target scheduling device node number corresponding to each scheduling operation step label, and the above-mentioned scheduling operation execution time point set to generate a target scheduling decision scheme. The target scheduling decision scheme includes the scheduling operation step sequence and the scheduling operation execution time point set.

[0083] Create a data structure containing three main fields: field 1 is the sequence of scheduling operation steps generated in step S161; field 2 is a list of all target scheduling device node numbers generated in step S162; and field 3 is the set of scheduling operation execution time points generated in step S164. This data structure constitutes the target scheduling decision scheme.

[0084] Step S170: Send the above target scheduling decision scheme to the power grid scheduling execution terminal.

[0085] The computing server encapsulates the generated target scheduling decision scheme into an IEC60870-5-104 protocol message via the scheduling data network and sends it to the designated power grid dispatch execution terminal, such as the remote terminal equipment of a substation or the monitoring workstation of the dispatch console. Encrypted transmission is used during the transmission process to ensure command security.

[0086] Step S210: Before inputting the above multimodal scheduling reference data stream into the pre-built multimodal scheduling scenario model, obtain a set of historical power grid scheduling scenarios. This set of historical power grid scheduling scenarios includes historical real-time electrical topology image stream samples, historical scheduling voice command stream samples, historical scheduling text report stream samples, and corresponding historical scheduling decision scheme real annotations.

[0087] Training samples are formed by extracting correctly verified dispatch operation events from the historical database of the power grid dispatch center over the past year. Each sample includes a historical real-time electrical topology image stream sample of 30 frames before and after the event, a historical dispatch voice command stream sample recording the dispatch instruction, a historical dispatch text report stream sample of the corresponding electronic operation ticket or dispatch log text, and a standard operation sequence and operation time manually compiled as a true annotation of the historical dispatch decision scheme.

[0088] Step S220: Input the above-mentioned historical real-time electrical topology image stream samples into the topology graph encoding branch to be trained, perform power grid topology connection relationship parsing processing on each frame of the historical real-time electrical topology image stream samples, generate a historical topology connection relationship matrix sequence, input the above-mentioned historical topology connection relationship matrix sequence into the graph convolutional topology feature extraction network, perform topology evolution encoding processing on the changes in topology connection relationships between adjacent frames, and obtain the historical topology operation status hidden state sequence samples.

[0089] Each frame of the historical real-time electrical topology image is parsed into a graph structure, generating a corresponding adjacency matrix as the historical topology connection matrix. The sequence of historical topology connection matrices is then input into a spatiotemporal graph convolutional network. This network consists of two spatial graph convolutional layers and one temporally gated convolutional layer. The spatial graph convolutional layers perform graph convolution on the adjacency matrix and node features of each frame, aggregating neighbor node information. The temporally gated convolutional layer performs convolution along the time dimension, capturing the topology change patterns between frames and outputting a hidden state sequence sample of the historical topology operating status.

[0090] Step S230: Input the above-mentioned historical scheduling voice command stream samples into the speech coding branch to be trained, perform scheduling command voice endpoint detection processing and scheduling domain adaptive speech recognition processing on the above-mentioned historical scheduling voice command stream samples to obtain historical scheduling terminology text sequence samples, perform scheduling semantic role labeling processing and command intent encoding processing on the above-mentioned historical scheduling terminology text sequence samples to generate historical scheduling command intent hidden state sequence samples.

[0091] The data processing flow for this step is the same as steps S131 to S136 in the inference phase, but the weights of all encoders are randomly initialized. The generated historical scheduling instruction intent hidden state sequence samples are then processed.

[0092] Step S240: Input the above-mentioned historical scheduling text report stream sample into the text encoding branch to be trained, perform scheduling operation event extraction processing on the above-mentioned historical scheduling text report stream sample, extract the historical scheduling operation step record text and the historical scheduling operation effect description text, perform joint encoding processing on the above-mentioned historical scheduling operation step record text and the above-mentioned historical scheduling operation effect description text, and generate historical scheduling experience hidden state sequence sample.

[0093] The data processing flow for this step is the same as steps S141 to S146 in the inference phase. The difference is that the input of the bidirectional long short-term memory network not only includes graph-level representation vectors of historical scheduling events, but also concatenates sentence vectors encoded from the text describing the effects of historical scheduling operations at each time step to introduce operation effect information.

[0094] Step S250: Input the above-mentioned historical topology operation status hidden state sequence samples, the above-mentioned historical scheduling instruction intent hidden state sequence samples, and the above-mentioned historical scheduling experience hidden state sequence samples into the multimodal fusion reasoning branch to be trained, perform cross-modal collaborative reasoning processing, generate historical multimodal joint reasoning hidden state sequence samples, input the above-mentioned historical multimodal joint reasoning hidden state sequence samples into the scheduling decision decoding branch to be trained, and generate a predictive scheduling decision scheme.

[0095] During training, the forward computation process of the multimodal fusion inference branch and the scheduling decision decoding branch is completely consistent with the inference phase steps S150 to S165.

[0096] Step S260: Calculate the difference loss value of the scheduling operation step sequence and the difference loss value of the scheduling operation execution time point between the above predicted scheduling decision scheme and the above historical scheduling decision scheme. Perform weighted merging of the above scheduling operation step sequence difference loss value and the above scheduling operation execution time point difference loss value to obtain the multi-task joint training loss value.

[0097] The loss value for the sequence difference of scheduling operation steps is calculated using the cross-entropy loss function, comparing the predicted distribution of scheduling operation step labels at each time step with the true labels. The loss value for the difference of scheduling operation execution time points is calculated using the smoothing L1 loss function, comparing the predicted operation execution time offset with the true time offset. These two loss values ​​are then weighted and summed using a preset weight coefficient alpha, where alpha is greater than 0 and less than 1, to obtain the multi-task joint training loss value.

[0098] Step S270: Using the multi-task joint training loss value, perform synchronous backpropagation update processing on the network weights of the topology graph encoding branch, the speech encoding branch, the text encoding branch, the multimodal fusion inference branch, and the scheduling decision decoding branch to complete one round of iterative training of the multimodal large model of the scheduling scenario.

[0099] Using the Adam optimizer, the joint training loss value for multiple tasks is backpropagated to calculate the gradient of the weights of all branches of the network, and the weights are updated according to the gradient. Steps S210 to S270 are repeated to traverse all training samples until the loss value converges, resulting in the final multimodal large model of the scheduling scenario.

[0100] Step S310: After obtaining the multi-modal dispatch reference data stream collected by the power grid dispatch center, obtain the current real-time electrical topology image and the previous historical electrical topology image from the above real-time electrical topology image stream, and perform pixel-level differential operation on the current real-time electrical topology image and the previous historical electrical topology image to generate an electrical topology differential image.

[0101] A real-time electrical topology image frame at time t and a historical electrical topology image frame at time t minus 1 are taken, both of which are grayscale images. A pixel-by-pixel interpolation operation is performed, and the absolute value is taken to obtain a differential electrical topology image. In this differential electrical topology image, regions with pixel values ​​close to zero represent unchanged regions, while regions with pixel values ​​significantly greater than zero represent locations where the electrical topology state has changed.

[0102] Step S320: Perform connected component detection processing on the above electrical topology differential image, extract the set of connected regions in the above electrical topology differential image where the pixel change value exceeds the preset change threshold, perform geometric center coordinate calculation processing and connected region area calculation processing on each connected region in the above connected region set, and obtain the geometric center coordinates and connected region area of ​​each connected region.

[0103] Set a change threshold. In the electrical topology differential image, set pixels with values ​​greater than the change threshold to 1, and the rest to 0, resulting in a binary image. Use a two-step connected component labeling algorithm to extract all connected regions consisting of pixels with a value of 1. For each connected region, calculate the mean of the x-coordinates and y-coordinates of all its pixels, using these as the geometric center coordinates of the region. Count the total number of pixels contained within the connected region, which is the area of ​​the connected region.

[0104] Step S330: Locate the corresponding electrical equipment graphic symbol in the real-time electrical topology map image at the current moment according to the geometric center coordinates of each connected region, identify the equipment type mark and voltage level mark of the electrical equipment graphic symbol, and combine the equipment type mark, the voltage level mark, the geometric center coordinates of the connected region and the area of ​​the connected region into an electrical topology change event record.

[0105] A search box is defined on the real-time electrical topology map at the current moment, centered on the geometric center coordinates of each connected region. Within this search box, the target detection network based on anchor points, as described in step S121, identifies the electrical equipment graphic symbol closest to the geometric center coordinates and obtains its equipment type and voltage level labels. These four attributes are then packaged into an electrical topology change event record.

[0106] Step S340: Arrange the electrical topology change event records corresponding to multiple consecutive frames of real-time electrical topology images in the above real-time electrical topology image stream in chronological order to generate an electrical topology change event sequence. Perform time interval calculation and spatial distance calculation on two adjacent electrical topology change event records in the above electrical topology change event sequence to obtain event interval duration parameters and event spatial offset parameters.

[0107] Sort all electrical topology change event records by timestamp. For two adjacent event records, calculate the difference between their timestamps to obtain the event interval parameter. Simultaneously, calculate the Euclidean distance between their geometric center coordinates to obtain the event space offset parameter.

[0108] Step S350: Based on the above event interval duration parameter and the above event spatial offset parameter, perform event aggregation processing on the above electrical topology change event sequence, and merge the continuous electrical topology change event records with event interval duration parameter less than the interval duration threshold and event spatial offset parameter less than the spatial offset threshold into the same power grid disturbance event candidate, and generate a power grid disturbance event candidate set.

[0109] Set interval duration thresholds and spatial offset thresholds. Using a greedy clustering algorithm, starting from the head of the sequence, group subsequent event records that meet the conditions of an interval duration less than the interval duration threshold and a spatial offset less than the spatial offset threshold into a single cluster. Each cluster constitutes a candidate power grid disturbance event.

[0110] Step S360: Perform disturbance intensity assessment processing on each power grid disturbance event candidate in the above power grid disturbance event candidate set, count the number of electrical topology change event records included in each power grid disturbance event candidate, and the number of different voltage level types involved in the electrical equipment graphic symbols in each power grid disturbance event candidate, to obtain the disturbance spread parameter and the disturbance cross-voltage level parameter.

[0111] For each candidate power grid disturbance event, the total number of its member event records is counted as a disturbance breadth parameter. The electrical equipment graphic symbols associated with its member event records are traversed, and all voltage level types that have appeared are collected. After deduplication, their number is counted as a disturbance cross-voltage level parameter.

[0112] Step S370: Mark the grid disturbance event candidates whose disturbance reach parameter exceeds the reach threshold or whose disturbance cross-voltage level parameter exceeds the cross-voltage level threshold as valid grid disturbance events. Extract the real-time electrical topology image frame interval corresponding to the grid disturbance event candidates marked as valid grid disturbance events from the real-time electrical topology image stream to generate a real-time electrical topology image stream that has been pre-screened by grid disturbance events.

[0113] Set thresholds for the breadth of impact and the cross-voltage level. Determine if both parameters of each grid disturbance event candidate exceed the thresholds. If either exceeds, mark the grid disturbance event candidate as a valid grid disturbance event. Based on the earliest and latest event record timestamps contained in the valid grid disturbance event, extract image frames corresponding to the time interval from the original real-time electrical topology image stream. This filtered image stream removes steady-state and minimally changing frames, highlighting segments containing significant disturbance events as high-value input for subsequent models.

[0114] Step S410: After sending the target scheduling decision scheme to the power grid scheduling execution terminal, receive the scheduling operation execution feedback data stream transmitted in real time by the power grid scheduling execution terminal during the process of executing the scheduling operation step sequence in the target scheduling decision scheme. The scheduling operation execution feedback data stream includes the measured change value sequence of power grid electrical quantities after each scheduling operation step is executed and the sequence of substation status image snapshots when the scheduling operation step is executed.

[0115] After completing each operation step in the target dispatch decision scheme, the power grid dispatch execution terminal sends a data packet back to the dispatch center's computing server. This data packet contains a sequence of measured changes in power grid electrical quantities, such as bus voltage, active power flow, and reactive power flow, taken at a sampling rate of 10 times per second for 5 seconds starting from the moment the operation is completed. Simultaneously, a sequence of high-definition image snapshots of corresponding electrical equipment captured by industrial cameras installed in the substation is also sent back.

[0116] Step S420: Perform deviation direction analysis and deviation magnitude analysis on the above-mentioned measured change value sequence of power grid electrical quantities and the reference interval of expected electrical quantity change in the corresponding scheduling operation step in the above-mentioned target scheduling decision scheme, and generate electrical quantity execution deviation direction mark and electrical quantity execution deviation magnitude level mark for each scheduling operation step. The electrical quantity execution deviation direction mark includes a positive deviation mark and a negative deviation mark, and the electrical quantity execution deviation magnitude level mark is obtained by mapping the electrical quantity deviation percentage.

[0117] In the target dispatching decision-making scheme, each operational step is accompanied by a reference range for expected changes in electrical quantities. The deviation between the final stable value of the measured change sequence of electrical quantities in the power grid and the upper and lower limits of the expected reference range is calculated. If the final stable value is greater than the upper limit of the reference range, a positive deviation marker is generated. If it is less than the lower limit of the reference range, a negative deviation marker is generated. The percentage of the absolute value of the deviation divided by the width of the reference range is calculated, and this percentage is mapped to a deviation magnitude level marker. For example, a deviation percentage less than 10% is level 1, 10% to 20% is level 2, and so on.

[0118] Step S430: Perform image semantic comparison processing on each substation status image snapshot in the above substation status image snapshot sequence and the expected substation target status description of the corresponding scheduling operation step in the above target scheduling decision scheme. Extract the opening and closing indicator status features and the appearance status features of the substation in the substation status image snapshot. Match the opening and closing indicator status features with the expected opening and closing indicator status description to generate a confirmation result of equipment operation in place.

[0119] The expected target status description for the substation equipment is text, such as "operation in place". For a snapshot of the substation equipment status image, a pre-trained convolutional neural network for substation equipment status recognition is used. This network contains a classification head and two attribute regression heads. The classification head identifies the opening and closing status of the equipment, while the attribute regression heads extract the angular features of the opening and closing indicators and the features indicating whether there are any abnormalities in their appearance, respectively. The extracted opening and closing status text is compared with the expected target status text. If they match, a "operation in place" confirmation result is generated; otherwise, a "operation not in place" confirmation result is generated.

[0120] Step S440: Combine the electrical quantity execution deviation direction mark, electrical quantity execution deviation magnitude level mark, and equipment operation in place confirmation result of each scheduling operation step into an execution effect evaluation record for that scheduling operation step. Arrange the execution effect evaluation records of all scheduling operation steps in the above scheduling operation step sequence in the execution order to generate a scheduling operation execution effect evaluation sequence.

[0121] Create a structure with fields for the operation step number, the electrical quantity execution deviation direction flag, the electrical quantity execution deviation magnitude level flag, and the equipment operation completion confirmation result. Generate one instance of such a structure for each operation step and arrange them in the order of the operation step sequence to obtain the scheduling operation execution effect evaluation sequence.

[0122] Step S450: Input the above-mentioned scheduling operation execution effect evaluation sequence into the pre-constructed scheduling effect attribution analysis network. The scheduling effect attribution analysis network performs attribution analysis on the causal relationship between the electrical quantity execution deviation direction marker and electrical quantity execution deviation magnitude level marker appearing in the above-mentioned scheduling operation execution effect evaluation sequence and the scheduling operation action type and target scheduling equipment node number of the corresponding scheduling operation steps in the above-mentioned scheduling operation step sequence, and generates deviation attribution analysis results.

[0123] The pre-built scheduling effect attribution analysis network is a Transformer-based sequence-to-sequence model. Its encoder reads the embedding representation of the scheduling operation execution effect evaluation sequence, and the decoder reads the embedding representation of the original scheduling operation step sequence. Through a cross-attention mechanism, the model learns which operation steps and devices are most strongly associated with the deviation markers in the evaluation sequence. The decoder outputs an attribution vector for each deviation event. After passing through a classifier, this vector outputs one or more scheduling operation step numbers that caused the deviation and their contribution scores, serving as the deviation attribution analysis result.

[0124] Step S460: Based on the above deviation attribution analysis results, adjust the operation execution time point or target scheduling equipment node number of the scheduling operation steps that are causally related to the deviation in the above target scheduling decision scheme, generate a corrected scheduling decision feedback scheme, and send it to the above power grid scheduling execution terminal.

[0125] If the deviation attribution analysis shows that a deviation is caused by improper execution time of a certain operation step, a time correction is calculated to adjust the operation execution time offset of that operation step. If the deviation is caused by incorrect target device selection, a new target scheduling device node number is recommended. After adjustment, the corrected scheduling decision feedback scheme is reassembled and sent again through step S170 to form closed-loop feedback control.

[0126] For example, in step S510: before inputting the multimodal scheduling reference data stream into the pre-built multimodal scheduling scenario model, a set of knowledge alignment samples in the power grid control domain is constructed. Each set of alignment samples in the power grid control domain knowledge alignment sample set includes an electrical main wiring diagram image, a voice message of a scheduling operation instruction, a text fragment of a scheduling operation procedure, and corresponding scheduling operation decision step sequence annotations and scheduling operation execution condition annotations.

[0127] Data was collected from the power grid regulations database and training simulation system. The electrical main wiring diagrams are standardized drawings of various typical electrical main wiring configurations. Dispatch operation commands were recorded by experienced dispatchers based on the drawings and tasks. Dispatch operation procedure text snippets are clauses from the procedure documents related to the specific operation task. Annotation information was compiled by experts based on the procedures.

[0128] Step S520: Input the above electrical main wiring diagram image into the topology graph encoding branch of the above multimodal large model of the scheduling scenario, perform electrical equipment symbol detection processing and electrical connection line tracing processing on the above electrical main wiring diagram image, generate an electrical equipment symbol location list and an electrical connection line vector set, convert the above electrical equipment symbol location list and the above electrical connection line vector set into a structured topology description graph of the electrical main wiring diagram, perform graph convolutional encoding processing on the above structured topology description graph, and generate the semantic encoding vector of the electrical main wiring diagram.

[0129] This step is similar to steps S121 and S122, but the output is a graph-level representation of a complete graph structure. After processing by a historical event graph encoder similar to S144, a readout function is used to aggregate the features of all nodes to obtain the graph-level embedding vector of the entire electrical main wiring diagram, i.e., the semantic encoding vector of the electrical main wiring diagram.

[0130] Step S530: Input the above-mentioned scheduling operation command speech into the speech coding branch of the above-mentioned scheduling scenario multimodal large model, perform scheduling command word speech recognition processing and scheduling command sentence parsing processing on the above-mentioned scheduling operation command speech to obtain the scheduling command structured field. The scheduling command structured field includes the operation object device name field, the operation action type field, and the operation condition limitation field. Perform field semantic embedding processing on the above-mentioned scheduling command structured field to generate the scheduling operation command speech semantic coding vector.

[0131] This step is the same as steps S131 to S136, but the scheduling semantic structure triplet is expanded to a quadruple containing the "operation condition constraint" field. The input to the instruction intent encoder is also expanded accordingly. The output vector of the instruction intent encoder serves as the speech semantic encoding vector for the scheduling operation instruction.

[0132] Step S540: Input the above-mentioned scheduling operation procedure text fragment into the text encoding branch of the above-mentioned scheduling scenario multimodal large model, perform procedure clause number recognition processing and procedure clause content semantic encoding processing on the above-mentioned scheduling operation procedure text fragment, generate procedure clause semantic encoding vector sequence, perform logical relationship encoding processing between procedure clauses on the above-mentioned procedure clause semantic encoding vector sequence, and generate scheduling operation procedure text semantic encoding vector.

[0133] A rule-based recognizer is used to extract clause numbers. The text content of each clause is converted into a semantic encoding vector of the procedure clause by a pre-trained domain text encoder, such as a fine-tuned BERT model for the power domain. The vector sequence of all clauses is input into a Transformer encoder to encode the logical relationships between clauses, such as "preconditions" and "operational consequences". The vector at position zero of the encoder output is taken as the semantic encoding vector of the scheduling operation procedure text.

[0134] Step S550: Input the above electrical main wiring diagram semantic encoding vector, the above scheduling operation instruction voice semantic encoding vector, and the above scheduling operation procedure text semantic encoding vector into the multimodal fusion inference branch of the above scheduling scenario multimodal big model, perform cross-modal semantic alignment fusion processing, generate cross-modal domain knowledge alignment fusion vector, input the above cross-modal domain knowledge alignment fusion vector into the scheduling decision decoding branch of the above scheduling scenario multimodal big model, and generate the alignment prediction scheduling decision step sequence and alignment prediction scheduling operation execution conditions.

[0135] The fusion branch simplifies to a cross-modal fusion layer, which concatenates the semantic encoding vectors of the three modalities and feeds them into a multilayer perceptron, outputting a cross-modal domain knowledge-aligned fusion vector. A fully connected layer in the scheduling decision decoding branch uses this fusion vector as input to predict the sequence of scheduling decision steps and the execution conditions of scheduling operations in parallel.

[0136] Step S560: Calculate the step sequence alignment loss value between the above-mentioned alignment prediction scheduling decision step sequence and the above-mentioned scheduling operation decision step sequence label, calculate the execution condition alignment loss value between the above-mentioned alignment prediction scheduling operation execution condition and the above-mentioned scheduling operation execution condition label, and sum the above-mentioned step sequence alignment loss value and the above-mentioned execution condition alignment loss value to obtain the total domain knowledge alignment loss value.

[0137] The step sequence alignment loss is calculated using cross-entropy loss, while the execution condition alignment loss is calculated using binary classification cross-entropy loss. The two are directly added together to obtain the total domain knowledge alignment loss.

[0138] Step S570: Based on the total loss value aligned with the domain knowledge mentioned above, perform gradient backpropagation update processing on the network parameters of the topology graph encoding branch, speech encoding branch, text encoding branch, multimodal fusion inference branch and scheduling decision decoding branch in the multimodal large model of the scheduling scenario mentioned above, and complete a round of continuous alignment and fine-tuning processing for the domain knowledge of power grid regulation.

[0139] Using a relatively small learning rate, the weights of the entire multimodal large model for the scheduling scenario are updated based on the total loss value aligned with domain knowledge. This process enables the model not only to learn the ability to imitate operation sequences, but more importantly, its internal representation remains consistent with the operating procedures, conceptual definitions, and operational logic in the field of power grid regulation.

[0140] Step S610: During the process of cross-modal collaborative reasoning processing of the hidden state sequence of the topology operation status, the hidden state sequence of the scheduling instruction intent, and the hidden state sequence of the historical scheduling experience by the multimodal fusion reasoning branch in the multimodal large model of the scheduling scenario, the power grid topology adjacency matrix corresponding to the real-time electrical topology graph image stream is obtained. The power grid topology adjacency matrix is ​​used to describe whether there is a direct electrical connection relationship between any two power grid equipment nodes in the power grid topology structure. The elements of each row in the power grid topology adjacency matrix are combined to form a set of single-hop neighbor nodes of the corresponding power grid equipment node.

[0141] From the parsing results of the real-time electrical topology graph image stream, an adjacency matrix describing the current power grid topology connections can be directly extracted. This adjacency matrix is ​​an N-by-N binary matrix, where N is the total number of power grid nodes. If node i and node j have a direct electrical connection, then the element in the i-th row and j-th column of the matrix is ​​1. Taking the column indices of all nodes with a value of 1 in the i-th row of the matrix forms the set of single-hop neighbor nodes for node i.

[0142] Step S620: Associate and bind each hidden state of the topology operation status in the above-mentioned hidden state sequence with the set of single-hop neighbor nodes of the corresponding power grid equipment node in the above-mentioned power grid topology structure, and establish a topology adjacency association pair between each hidden state of the topology operation status and the single-hop neighbor hidden state of the single-hop neighbor power grid equipment node.

[0143] For each hidden state of the topology operation status, assume it corresponds to node i. Retrieve the set of single-hop neighbor nodes of node i, and index the hidden states of the topology operation status corresponding to these neighbor nodes from the sequence of hidden states of the topology operation status. Pair the hidden state of the topology operation status of node i with the hidden state of the topology operation status of each neighbor to form multiple topology adjacency pairs.

[0144] Step S630: Perform electrical conduction coupling calculation on the hidden state of the topology operation status and the hidden state of the single-hop neighbor topology operation status in each topology adjacency association pair, and input the element-by-element difference vector between the hidden state of the topology operation status and the hidden state of the single-hop neighbor topology operation status into the electrical conduction strength encoder to generate the topology conduction coupling strength vector.

[0145] Calculate the element-wise difference between the hidden states of two topological operational states in a topological adjacency pair to obtain a difference vector. This difference vector is then input into an electrical conduction strength encoder. The electrical conduction strength encoder is a two-layer fully connected network that outputs a topological conduction coupling strength vector with the same dimension as the difference vector. Each element in this topological conduction coupling strength vector represents the conduction influence strength of the neighbor node on the central node in the corresponding feature dimension.

[0146] Step S640: Use the above-mentioned topology conduction coupling strength vector to perform weighted transformation on the hidden state of the single-hop neighbor topology operation status to generate the hidden state of single-hop neighbor conduction influence. Then, perform element-wise summation and aggregation on the hidden states of single-hop neighbor power grid equipment nodes corresponding to the hidden state of the above-mentioned topology operation status to generate the hidden state of topology conduction aggregation influence.

[0147] The element-wise product of the topological propagation coupling strength vector and the hidden state of the single-hop neighbor topological operation status is used to obtain the hidden state of the single-hop neighbor propagation influence. The element-wise summation of the hidden states of the single-hop neighbor propagation influence of all neighbors belonging to the same central node is then performed to obtain an aggregated vector, which is the hidden state of the topological propagation aggregate influence.

[0148] Step S650: Input the above-mentioned hidden state of topology operation status and the above-mentioned hidden state of topology transmission and aggregation influence into the transmission gating fusion unit. The transmission gating fusion unit generates a transmission fusion gating coefficient based on the matching degree of the above-mentioned hidden state of topology operation status and the above-mentioned hidden state of topology transmission and aggregation influence. The transmission fusion gating coefficient is used to perform weighted fusion processing on the above-mentioned hidden state of topology operation status and the above-mentioned hidden state of topology transmission and aggregation influence to generate a modified hidden state of topology operation status with injected topology transmission constraints.

[0149] The conduction-gated fusion unit concatenates the hidden states of the topology's operational status and the hidden states of topology's conduction-aggregated influence. Through a fully connected layer and sigmoid activation, it outputs a conduction-fusion gating coefficient vector with the same dimension as the hidden states. Using this vector, the element-wise multiplication of the conduction-fusion gating coefficient vector by the hidden states of topology's conduction-aggregated influence is calculated. Adding this to the element-wise multiplication of the conduction-fusion gating coefficient vector by the hidden states of the topology's operational status yields the corrected hidden states of the topology's operational status, infused with topology conduction constraints.

[0150] Step S660: Replace the original hidden state of the topology operation status in the above-mentioned hidden state sequence with the modified hidden state of the topology operation status injected with the above-mentioned topology transmission constraints, generate the hidden state sequence of the topology operation status injected with the topology transmission constraints, and input it into the above-mentioned multimodal fusion reasoning branch for subsequent cross-modal collaborative reasoning processing.

[0151] This replacement operation enables the subsequent multimodal fusion reasoning process to explicitly consider the connection and conduction relationships of the power grid physical topology, improving the physical rationality of the state representation.

[0152] Step S710: During the cross-modal collaborative reasoning process of the hidden state sequence of the topology operation status, the hidden state sequence of the scheduling instruction intent, and the hidden state sequence of the historical scheduling experience by the multimodal fusion reasoning branch in the multimodal large model of the scheduling scenario, multiple hidden states of scheduling instruction intent that overlap in time in the hidden state sequence of the scheduling instruction intent are extracted to generate a set of concurrent hidden states of scheduling instruction intent. Any two concurrent hidden states of scheduling instruction intent in the set of concurrent hidden states of scheduling instruction intent are compared with the target devices of the instruction operation to identify the first type of conflict pair of scheduling instruction intents with the same target device and the second type of conflict pair of scheduling instruction intents with the same target device and the topological connection relationship between the target devices.

[0153] Examine the timestamps of all hidden states of scheduling instructions, and consider instructions with time window overlap exceeding a threshold as concurrent. Extract the concurrent groups. For any two instructions within a group, compare their resolved target scheduling device node numbers. If they are the same, mark them as a first-type scheduling instruction intention conflict pair. If they are different, but the two nodes have a direct or secondary connection relationship in the power grid topology adjacency matrix, mark them as a second-type scheduling instruction intention conflict pair.

[0154] Step S720: Perform instruction intent semantic decoding on the two concurrent scheduling instruction intent hidden states in the first type of scheduling instruction intent conflict pair to obtain the operation action type of the first scheduling instruction and the operation action type of the second scheduling instruction. Input the operation action type of the first scheduling instruction and the operation action type of the second scheduling instruction into the scheduling operation procedure conflict rule base for rule matching processing to determine whether the first type of scheduling instruction intent conflict pair constitutes an operation action conflict prohibited by the operation procedure.

[0155] The semantic vector of the instruction intent in the first type of scheduling instruction intent conflict pair is decoded to obtain the operation action type text, such as "close" or "open". The scheduling operation procedure conflict rule base is retrieved. This rule base is in key-value pair format, where the key is the action type pair and the value is a conflict Boolean value. For example, if the key is "close, close" and the target device is the same, the value is "conflict". A precise match is performed to obtain the conflict determination result.

[0156] Step S730: Extract the associated topology operation status hidden state of the two concurrent scheduling instruction intent hidden states in the above-mentioned second type of scheduling instruction intent conflict pair, obtain the topology path electrical connection strength parameter between the target device node of the first scheduling instruction and the target device node of the second scheduling instruction in the above-mentioned second type of scheduling instruction intent conflict pair, and mark the second type of scheduling instruction intent conflict pair whose topology path electrical connection strength parameter exceeds the electrical connection threshold as topology transmission conflict pair.

[0157] From the sequence of hidden states of the topology operating status injected with topology transmission constraints, the hidden states of the topology operating status corresponding to the two instruction target device nodes are retrieved. The cosine similarity of these two hidden states of the topology operating status is calculated as a parameter of the electrical connection strength of the topology path. If the cosine similarity is greater than the set electrical connection threshold, it indicates that the two nodes are electrically coupled closely, and the operation of one device will seriously affect the other device. Therefore, this conflict pair is marked as a topology transmission conflict pair.

[0158] Step S740: Merge the conflict pairs in the first type of scheduling instruction intent conflict pairs that are determined to constitute operational action conflicts prohibited by the operating procedures and the conflict pairs in the second type of scheduling instruction intent conflict pairs marked as topological transmission conflict pairs to form a set of scheduling instruction conflicts that need to be resolved. Perform conflict source labeling on the two hidden states of scheduling instruction intents involved in each scheduling instruction conflict pair in the set of scheduling instruction conflicts that need to be resolved, and add a conflict type label and a conflicting scheduling instruction identifier.

[0159] Take the union of the conflict pairs from the two sources. For each conflict pair in the conflict set, append a conflict tag vector to the hidden state vectors of the two involved instruction intentions. This conflict tag vector is formed by concatenating the embedding vector of the conflict type label with the embedding vector of the scheduling instruction identifier of the conflicting counterpart.

[0160] Step S750: The hidden state of scheduling instruction intent with added conflict type label and conflicting opponent scheduling instruction identifier is input into the instruction conflict resolution attention module. The instruction conflict resolution attention module generates the conflict-aware scheduling instruction intent hidden state correction amount by calculating the conflict intensity attention weight between each hidden state of scheduling instruction intent and its conflicting opponent scheduling instruction intent hidden state.

[0161] The instruction conflict resolution attention module uses its own instruction intent hidden state as the query vector and the hidden states of all conflicting instruction intents as the key and value vectors to calculate a multi-head attention output. This output represents the direction and magnitude of the adjustment that the self-instruction intent should make after considering the influence of all conflicting instructions, serving as the adjustment amount for the hidden state of the scheduling instruction intent in conflict perception.

[0162] Step S760: Use the above-mentioned conflict-aware scheduling instruction intention hidden state correction amount to perform residual correction processing on the original scheduling instruction intention hidden state, generate the conflict-resolved scheduling instruction intention hidden state, and replace the corresponding original scheduling instruction intention hidden state in the above-mentioned scheduling instruction intention hidden state sequence to generate the conflict-resolved scheduling instruction intention hidden state sequence.

[0163] Adding the original hidden state of the scheduling instruction intent to the correction value yields the hidden state of the scheduling instruction intent after conflict resolution. This new hidden state reflects a rational adjustment to the instruction intent after considering concurrent instruction conflicts, such as reducing its enforceability or modifying its target magnitude. Replace the corresponding item in the original sequence with this new hidden state.

[0164] Step S810: In the process of generating scheduling decisions for the hidden state sequence of the multimodal joint inference by the scheduling decision decoding branch in the multimodal large model of the scheduling scenario, the set of safety boundary constraint parameters under the current operation mode of the power grid is obtained. The set of safety boundary constraint parameters includes the power flow transmission limit value of each transmission line, the voltage allowable offset range of each voltage level bus, and the operation number limit value of each substation equipment. The set of safety boundary constraint parameters is transformed into a set of safety boundary constraint embedding vectors.

[0165] Real-time safety boundary constraint data is obtained from the power grid energy management system. For example, the power flow transmission limit of line "L1" is P_limit_L1. These numerical constraints are then normalized. For each constraint, a constraint encoder is used to map the constraint type, constraint limit, and associated device identifier into a fixed-dimensional safety boundary constraint embedding vector.

[0166] Step S820: Input each multimodal joint reasoning hidden state in the above multimodal joint reasoning hidden state sequence into the security boundary constraint encoder. The security constraint type self-attention layer in the security boundary constraint encoder performs similarity calculation on the different dimension segments in the above multimodal joint reasoning hidden state and the corresponding security boundary constraint embedding vectors in the above security boundary constraint embedding vector set, and generates the security constraint association strength distribution vector of each multimodal joint reasoning hidden state.

[0167] The hidden state of the multimodal joint inference is divided into multiple segments, each corresponding to a different power grid device. The security constraint type self-attention layer uses each device segment as the query vector and all security boundary constraint embedding vectors as the key vectors to calculate the scaled dot product similarity between them. This results in a similarity matrix with the shape of the number of devices multiplied by the number of constraints, which is the distribution vector of security constraint association strength.

[0168] Step S830: Using the aforementioned security constraint association strength distribution vector, retrieve strongly correlated security boundary constraint embedding vectors from the aforementioned security boundary constraint embedding vector set whose correlation with the current multimodal joint inference hidden state exceeds the association threshold. Input the aforementioned strongly correlated security boundary constraint embedding vectors and the aforementioned multimodal joint inference hidden state into the security constraint cross-attention layer, and calculate the security constraint cross-attention weight of the aforementioned multimodal joint inference hidden state on the aforementioned strongly correlated security boundary constraint embedding vectors.

[0169] Constraints with similarity greater than the association threshold are selected from the security constraint association strength distribution vector. These strongly correlated security boundary constraints are embedded into vectors as key and value vectors, and cross-attention is calculated using the original multimodal joint inference hidden state as the query vector.

[0170] Step S840: Based on the above-mentioned security constraint cross-attention weights, perform weighted aggregation processing on the above-mentioned strongly correlated security boundary constraint embedding vector to generate a security boundary constraint context vector. Perform residual connection fusion processing on the above-mentioned security boundary constraint context vector and the above-mentioned multimodal joint inference hidden state to generate a modified multimodal joint inference hidden state embedded with security boundary constraints.

[0171] The value vector is weighted and summed using attention weights to obtain the safety boundary constraint context vector. This safety boundary constraint context vector is then directly added to the original multimodal joint inference hidden state, thus incorporating the contextual information of the most relevant safety constraint into the original representation.

[0172] Step S850: Input the modified multimodal joint inference hidden state with embedded security boundary constraints into the security boundary violation determination decoder for scheduling operation security boundary violation prediction processing, and generate a security boundary violation prediction mark sequence. This security boundary violation prediction mark sequence is used to indicate whether subsequent scheduling operation steps will trigger a security boundary violation.

[0173] The security boundary violation determination decoder is a binary classifier that follows the corrected multimodal joint inference hidden state. For each device, it predicts whether its corresponding security boundary constraint will be triggered in the following operation steps. It outputs a binary label sequence: "safe" or "violation warning".

[0174] Step S860: Feed back the above-mentioned security boundary violation prediction mark sequence to the generation process of the above-mentioned multimodal joint reasoning hidden state sequence, perform a replacement search process for the scheduling operation steps of the multimodal joint reasoning hidden state with security boundary violation prediction, replace the original multimodal joint reasoning hidden state with the above-mentioned modified multimodal joint reasoning hidden state with embedded security boundary constraints, and generate a multimodal joint reasoning hidden state sequence with embedded security boundary constraints.

[0175] If a "violation warning" appears in the safety boundary violation prediction tag sequence, the process backtracks to the scheduling step sequence decoder. The scheduling step sequence decoder searches the predefined safety operation step template library for an alternative operation step that can eliminate the warning, based on the "violation warning" tag and the current hidden state. The original prediction is replaced with the label of this alternative step, and the original hidden state is replaced with the modified multimodal joint inference hidden state embedded with safety boundary constraints. The process is then repeated with forward propagation for verification until no warnings are detected or the maximum number of iterations is reached.

[0176] Step S910: In the process of generating scheduling decisions for the hidden state sequence of the multimodal joint inference by the scheduling decision decoding branch in the multimodal large model of the scheduling scenario, the scheduling operation step sequence in the target scheduling decision scheme is obtained, the target scheduling equipment node number corresponding to each scheduling operation step label in the scheduling operation step sequence is extracted, and the node position of each target scheduling equipment node number in the power grid topology is mapped to the initial disturbance injection node on the power grid topology transmission diagram.

[0177] Based on the generated initial target scheduling decision scheme, disturbance propagation simulation is performed. A power grid topology transmission graph is established with power grid equipment as nodes and line impedance as edge weights. The target equipment node of each step operation is marked as the initial disturbance injection node.

[0178] Step S920: Starting from each initial disturbance injection node, perform electrical state change propagation simulation to adjacent grid equipment nodes according to the electrical connection path defined by the grid topology adjacency matrix. In each step of the propagation simulation, calculate the propagation attenuation coefficient based on the equipment impedance parameters of the grid equipment nodes on the propagation path and the disturbance change of the initial disturbance injection node. Use the propagation attenuation coefficient to perform attenuation correction processing on the electrical state change propagated to the current grid equipment node.

[0179] Starting from the initial disturbance injection node, breadth-first propagation is performed. The initial disturbance change is defined as a normalized vector. When the disturbance propagates to node v, the path impedance magnitude from the disturbance source to node v is calculated, and the reciprocal of this magnitude is taken as the propagation attenuation coefficient. The electrical state change propagating to node v is equal to the initial disturbance change multiplied by the propagation attenuation coefficient.

[0180] Step S930: Continue to propagate the electrical state change amount after attenuation correction to the adjacent power grid equipment nodes of the current power grid equipment node until the electrical state change amount of the power grid equipment node at the end of the propagation path is attenuated to below the preset propagation cutoff ratio of the disturbance change amount of the initial disturbance injection node, and stop the propagation. Record the cumulative electrical state change amount of each power grid equipment node on the propagation path.

[0181] If the magnitude of the electrical state change propagating along a path is lower than the magnitude of the initial disturbance change multiplied by a preset propagation cutoff ratio, propagation stops along that path. Each node in the network may receive propagation quantities from multiple paths; these quantities are vector-summed to obtain the cumulative electrical state change for each grid device node.

[0182] Step S940: Compare the cumulative electrical state changes of all power grid equipment nodes along the propagation path with the safe operating allowable change range of each power grid equipment node node node by node, identify the over-limit power grid equipment nodes whose cumulative electrical state changes exceed the safe operating allowable change range, and generate a set of over-limit power grid equipment nodes.

[0183] Query the permissible range of safe operating variations for each node, such as the safe range of node voltage amplitude. Compare the projected value of the cumulative electrical state variation in the corresponding dimension to see if it exceeds this safe range. If it does, add the node to the set of over-limit grid equipment nodes.

[0184] Step S950: Perform over-limit assessment and over-limit type determination processing on each over-limit power grid equipment node in the above set of over-limit power grid equipment nodes, and generate over-limit level label and over-limit type label. The over-limit type label includes voltage over-limit type label, current over-limit type label and equipment operation number over-limit type label.

[0185] Calculate the proportion of cumulative electrical state changes exceeding the safe range for out-of-limit equipment nodes, and map this proportion to an out-of-limit severity level label from 1 to 5. Determine the out-of-limit type label based on the physical quantity dimension of the out-of-limit activity.

[0186] Step S960: Combine the above-mentioned set of over-limit power grid equipment nodes, the over-limit level label and over-limit type label of each over-limit power grid equipment node into the topology propagation impact pre-simulation result of the above-mentioned target scheduling decision scheme, and add it to the above-mentioned target scheduling decision scheme as additional security assessment information.

[0187] Create a dictionary structure where the key is the node number of the over-limit grid equipment, and the value is a tuple containing the over-limit severity level and over-limit type flags. Embed this dictionary as a field of the safety simulation result into the data structure of the target dispatch decision scheme, and send it together to the power grid dispatch execution terminal to provide dispatchers with a safety assessment reference for decision-making.

[0188] In the specific implementation process, combined with Figure 2 As shown in the processing flow, the solution of this application can receive and process input data of various modalities. Figure 2The left side displays a sequence of real-time electrical topology image frames containing symbols for elements such as substations, circuit breakers, and lines, as well as continuous waveforms of the dispatch voice command stream and document stacks of the dispatch text report stream. This embodiment generates these through state parsing and feature encoding processing of the input signal stream in step S120. Figure 2 The central region displays the hidden state sequence of topological operation status, the hidden state sequence of scheduling instruction intent, and the hidden state sequence of historical scheduling experience. During the cross-modal feature fusion and joint inference processes in steps S130 and S140, Figure 2 The central region utilizes a cross-attention mechanism to spatiotemporally align these different hidden state sequences, ultimately outputting the result through a fusion network. Figure 2 The sequence of scheduling operation steps, the target scheduling device node number, and the set of scheduling operation execution time points shown on the right are combined into a target scheduling decision scheme. Figure 2 The complete link demonstrated, from multi-source heterogeneous signal streams to unified structured decision command output, directly reflects that this application can effectively overcome the shortcomings of incomplete information from a single data source. By fusing unstructured images, voice, and structured text, it enhances the comprehensive perception capability of the power grid dispatching system for complex operating conditions.

[0189] exist Figure 3 In the KL divergence and asymmetric index evolution diagram shown, this method identifies key causal breakpoints by calculating the differences between different feature sequences. Figure 3 The discrete data points in the dataset traced the evolution of the forward KL(Q||K) divergence, the backward KL(K||Q) divergence, and the asymmetry exponent Δ resulting from their difference as a function of scheduling time steps (T=1 to T=40). The forward KL divergence curve exhibits distinct local and global peaks at T=15 and T=22, respectively, which are related to... Figure 3 The bottom labels "causal breakpoints T=15, 22" are completely consistent. Conversely, the reverse KL divergence curve, while also exhibiting fluctuations, shows significant differences in values ​​at the two breakpoints compared to the forward divergence. Consequently, the asymmetry exponent Δ also reaches a large value at these breakpoints. Based on the generation process of the hidden state sequence for multimodal joint inference in step S150 of the specification, this curve indicates that the moments when significant asymmetry occurs between the forward and reverse divergences (i.e., causal breakpoints T=15 and T=22) are precisely the points where significant changes occur in the convergence of multimodal information during the system's inference process. Figure 3 The causal breakpoint localization results based on KL divergence asymmetry demonstrate that this application can accurately pinpoint the time position of the critical transition between scheduling instructions and topology state using statistical distance measurement methods, effectively reducing the risk of decision delay caused by feature misalignment.

[0190] exist Figure 4The evolution curve of the continuous cohomology Betti number shown demonstrates the quantitative verification of the power grid's operating status using topological data analysis. Figure 4 The evolution of two topological invariant curves, β0 (number of connected branches) and β1 (number of rings), was depicted based on the filtering parameter ε (varying from 0.05 to 1.50). The figure clearly compares the operating states under three conditions: in the normal state and the fault state, the β1 curve shows a significant difference at ε=0.35, specifically, β1=5 in the fault state and β1=8 in the normal state. Figure 4 The text label at the bottom clearly records this difference: "Fault state β1=5 vs. Normal β1=8". Meanwhile, Figure 4 It also shows that the "Solution β1" curve reaches a relatively high value at ε=0.65, exceeding the normal state baseline. Combined with the decision optimization based on the execution feedback loop in step S160 of the instruction manual, Figure 4 The observed differences in Betti number evolution demonstrate that this application can quantify the distinction between healthy and abnormal system states by calculating topological connectivity indices (β0 and β1). The significant reduction in the number of loops under fault conditions directly indicates a decrease in system topological redundancy, while the recovery and surpassing of loops after the implementation of the solution directly proves that the decision-making scheme not only repairs the fault but also effectively improves the redundancy capability of the power grid topology.

[0191] exist Figure 5 The convergence characteristic curve of the Lyapunov function shown verifies the stability of the control loop under different scheduling scenarios. Figure 5 Five typical operating scenarios were recorded, including "summer peak λ=0.28", "winter trough λ=0.32", "post-typhoon λ=0.18", "maintenance period λ=0.26", and "new energy λ=0.24". According to... Figure 5 The data description shows that these five Lyapunov function V(t) curves exhibit a smooth, monotonically decreasing trend throughout the horizontal axis feedback rounds (from round 1 to round 20), without any local oscillations or divergence. Combined with... Figure 5 The data statistics in the bottom annotation box show that the convergence rate parameter λ for all five scenarios is within the effective range of 0.18 to 0.32, and the V value of all curves remains within the range of no more than 0.08 when the feedback loop reaches 20. Based on the adaptive adjustment logic for executing the feedback loop in step S160 of the instruction manual, Figure 5The multi-scenario, fully convergent Lyapunov curves shown objectively demonstrate that the execution feedback process proposed in this application can ensure that the system state variables converge smoothly and stably to the target range when facing various different operating environment parameters. This ensures the long-term stable operation of the power grid dispatch decision system and reduces the risk of repeated adjustments to dispatch instructions due to system oscillations or divergences.

[0192] In an exemplary embodiment, a power grid dispatching decision support system based on a multimodal large model is provided. This system can be a terminal, server, etc., and its internal structure includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, near-field communication, or other technologies. When the computer program is executed by the processor, it implements a power grid dispatching decision support method based on a multimodal large model. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the shell of the power grid dispatch decision support system based on a multimodal large model, or an external keyboard, touchpad, or mouse, etc.

[0193] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A power grid dispatching decision support method based on a multimodal large model, characterized in that, The method includes: The system acquires a multimodal dispatch reference data stream collected by the power grid dispatch center. The multimodal dispatch reference data stream includes a real-time electrical topology map image stream characterizing the power grid topology operation status, a dispatch voice command stream recording dispatcher dispatch instructions, and a dispatch text report stream recording historical dispatch operation records. The multimodal scheduling reference data stream is input into a pre-constructed multimodal scheduling scenario model. The topology graph encoding branch in the multimodal scheduling scenario model performs topology operation status encoding on the real-time electrical topology graph image stream to generate a topology operation status hidden state sequence. The speech encoding branch in the multimodal scheduling scenario model performs instruction intent encoding on the scheduling speech command stream to generate a scheduling instruction intent hidden state sequence. The text encoding branch in the multimodal scheduling scenario model performs historical scheduling experience encoding on the scheduling text report stream to generate a historical scheduling experience hidden state sequence. The multimodal fusion reasoning branch in the multimodal large model of the scheduling scenario performs cross-modal collaborative reasoning processing on the hidden state sequence of the topology operation status, the hidden state sequence of the scheduling instruction intent, and the hidden state sequence of the historical scheduling experience, generating a multimodal joint reasoning hidden state sequence of the fusion scheduling scenario context. The scheduling decision decoding branch in the multimodal large model of the scheduling scenario performs scheduling decision generation processing on the hidden state sequence of the multimodal joint inference to generate a target scheduling decision scheme. The target scheduling decision scheme includes a sequence of scheduling operation steps and a set of scheduling operation execution time points. The target scheduling decision scheme is sent to the power grid scheduling execution terminal.

2. The power grid dispatching decision support method based on a multimodal large model according to claim 1, characterized in that, The process of encoding the real-time electrical topology image stream using the topology graph encoding branch in the multimodal large model of the scheduling scenario to generate a hidden state sequence of topology operation status includes: The real-time electrical topology image stream is extracted from the multimodal scheduling reference data stream. Each frame of the real-time electrical topology image stream is input into the graphic element parsing layer of the topology encoding branch. Graphic element recognition processing is performed on the bus connection lines, circuit breaker symbols, disconnector symbols, transformer symbols and load label text in the real-time electrical topology image to obtain the set of graphic elements contained in the real-time electrical topology image. For each graphic element in the graphic element set, graphic element attribute extraction processing is performed to obtain the graphic element type identifier, the planar coordinate position in the real-time electrical topology diagram image, and the list of adjacent graphic elements that are connected to each graphic element. The graphic element type identifier, planar coordinate position, and list of adjacent graphic elements of all graphic elements in the graphic element set are combined to form a topology diagram structured description record. The topology structured description records of multiple consecutive frames of real-time electrical topology images are arranged in chronological order to form a topology structured description record time series. The topology structured description record time series of two adjacent frames of topology structured description records are compared to identify changes in the state of graphic elements. The first set of changing elements whose graphic element type identifier changes, the second set of changing elements whose planar coordinate position changes, and the third set of changing elements whose adjacent graphic element lists change. The first set of changing elements, the second set of changing elements, and the third set of changing elements are merged to obtain a set of electrical topology state change events between adjacent frames. Each electrical topology state change event in the set of electrical topology state change events between adjacent frames is assigned an event type label, which includes a device switching event label, a topology reconfiguration event label, and a power flow migration event label. The set of electrical topology state change events between adjacent frames with the event type label is sequentially input into the topology state evolution recursive network of the topology graph encoding branch according to the time sequence of the topology graph structured description recording time series. The topology state evolution recursive network encodes the causal dependency of the electrical topology state change events in the time dimension to generate the hidden state of the topology operation status at each time step. The hidden states of the topology operation status at all time steps output by the topology state evolution recursive network are combined in chronological order to form a hidden state sequence of the topology operation status, and the hidden state sequence of the topology operation status is output to the output interface of the topology graph encoding branch.

3. The power grid dispatching decision support method based on a multimodal large model according to claim 1, characterized in that, The step of encoding the scheduling speech command stream using the speech coding branch in the multimodal large model of the scheduling scenario to generate a hidden state sequence of scheduling command intent includes: The scheduling voice command stream is extracted from the multimodal scheduling reference data stream, and the scheduling voice command stream is input into the speech signal preprocessing layer of the speech coding branch. The scheduling voice command stream is subjected to silence segment removal processing and speech activity endpoint detection processing to obtain a set of effective scheduling voice segments with silence segments removed. Acoustic feature extraction processing is performed on each effective scheduled speech segment in the set of effective scheduled speech segments to extract the Mel frequency cepstral coefficient feature sequence and the fundamental frequency change trajectory feature sequence of each effective scheduled speech segment. The Mel frequency cepstral coefficient feature sequence and the fundamental frequency change trajectory feature sequence are concatenated along the feature dimension to generate the acoustic feature matrix of each effective scheduled speech segment. The acoustic feature matrix of each effective scheduled speech segment is input into the scheduling term acoustic encoder of the speech coding branch. The acoustic feature matrix is ​​subjected to convolutional downsampling and self-attention encoding to generate a scheduling term acoustic embedding vector sequence for each effective scheduled speech segment. Each scheduling term acoustic embedding vector in the scheduling term acoustic embedding vector sequence corresponds to a phonetic unit in the effective scheduled speech segment. The scheduling term acoustic embedding vector sequence is input into the scheduling domain adaptation connection temporal classifier of the speech coding branch. The scheduling term acoustic embedding vector sequence is then processed by scheduling domain term sequence decoding to obtain the scheduling term text sequence corresponding to each valid scheduling speech segment. The scheduling term text sequence includes scheduling device name words, scheduling operation action words, and scheduling magnitude description words. Each word in the scheduling terminology text sequence is subjected to part-of-speech tagging and scheduling semantic role tagging. Device name words that act as scheduling device name semantic roles, operation action words that act as scheduling operation action semantic roles, and magnitude description words that act as scheduling magnitude description semantic roles are identified in the scheduling terminology text sequence. Based on the identification results, a scheduling semantic structure triplet is constructed for each valid scheduling speech segment. The scheduling semantic structure triple is input into the instruction intent encoder of the speech coding branch. The device name word, operation action word and magnitude description word in the scheduling semantic structure triple are semantically combined and encoded to generate a scheduling instruction intent semantic vector for each valid scheduling speech segment. The scheduling instruction intent semantic vector is arranged in the time order of the valid scheduling speech segments to generate a scheduling instruction intent hidden state sequence. The scheduling instruction intent hidden state sequence is output to the output interface of the speech coding branch.

4. The power grid dispatching decision support method based on a multimodal large model according to claim 1, characterized in that, The step of encoding the scheduling text report stream with historical scheduling experience by the text encoding branch in the multimodal large model of the scheduling scenario to generate a historical scheduling experience hidden state sequence includes: The scheduling text report stream is extracted from the multimodal scheduling reference data stream. Each scheduling text report in the scheduling text report stream is input into the report chapter parsing layer of the text encoding branch. Chapter boundary detection processing is performed on the scheduling text report to identify the scheduling background description chapter, scheduling operation record chapter, and scheduling effect summary chapter of the scheduling text report. The plain text content of the scheduling operation record chapter is extracted as the scheduling operation record text. The scheduling operation record text is processed by sentence segmentation and scheduling entity word recognition. Entity words for scheduling equipment name, scheduling operation action, scheduling operation time, and electrical quantity change description before and after scheduling operation are extracted from the scheduling operation record text. The extracted entity words are then used to construct a scheduling operation event entity association graph based on their co-occurrence relationship in the scheduling operation record text. Each scheduling device name entity node in the scheduling operation event entity association diagram is processed by node attribute encoding. The topology location number and voltage level type of the scheduling device name entity node in the power grid topology are converted into a node location attribute vector. The node location attribute vector of the scheduling device name entity node is then appended to the corresponding node in the scheduling operation event entity association diagram. The scheduling operation event entity association graph with the additional node location attribute vector is input into the historical event graph encoder of the text encoding branch. Multi-hop neighbor information aggregation processing is performed on the scheduling operation event entity association graph to obtain the neighbor context aggregation vector of each scheduling device name entity word node in the scheduling operation event entity association graph. The neighbor context aggregation vectors of all scheduling device name entity word nodes in the scheduling operation event entity association graph are combined into a historical scheduling event graph-level representation vector. The graph-level representation vectors of historical scheduling events corresponding to multiple consecutive scheduling operation records in the same scheduling text report are arranged in the order of scheduling operation time and input into the historical experience sequence modeler of the text encoding branch. The historical experience sequence modeler encodes the success and failure mark transfer rules and operation interval duration rules between adjacent scheduling operation events in the graph-level representation vector sequence of historical scheduling events to generate the historical scheduling experience hidden state of each scheduling operation event. The historical scheduling experience hidden states of all step scheduling operation events output by the historical experience sequence modeler are combined into a historical scheduling experience hidden state sequence according to the scheduling operation time order, and the historical scheduling experience hidden state sequence is output to the output interface of the text encoding branch.

5. The power grid dispatching decision support method based on a multimodal large model according to claim 1, characterized in that, The process involves cross-modal collaborative reasoning of the hidden state sequence of the topology operation status, the hidden state sequence of the scheduling instruction intent, and the hidden state sequence of the historical scheduling experience by the multimodal fusion reasoning branch in the multimodal large model of the scheduling scenario, generating a multimodal joint reasoning hidden state sequence of the fused scheduling scenario context, including: Each topology operation status hidden state in the topology operation status hidden state sequence and the time-aligned scheduling instruction intention hidden state in the scheduling instruction intention hidden state sequence are input into the topology intention cross-attention layer of the multimodal fusion inference branch. The first cross-attention weight of each topology operation status hidden state to each scheduling instruction intention hidden state is calculated. The scheduling instruction intention hidden state sequence is weighted and summed using the first cross-attention weight to generate an intention-aware topology operation status correction hidden state sequence. Each hidden state of scheduling instruction intent in the hidden state sequence and the time-aligned hidden state of topology operation status in the hidden state sequence of topology operation status are input into the intent topology cross-attention layer of the multimodal fusion inference branch. The second cross-attention weight of each hidden state of scheduling instruction intent to each hidden state of topology operation status is calculated. The hidden state sequence of topology operation status is weighted and summed using the second cross-attention weight to generate a topology-aware hidden state sequence of corrected hidden states of scheduling instruction intent. The intention-aware topology operation status correction hidden state sequence and the topology-aware scheduling instruction intention correction hidden state sequence are input into the fusion gating unit of the multimodal fusion inference branch. The intention-aware topology operation status correction hidden state and the topology-aware scheduling instruction intention correction hidden state at each time step are subjected to gating fusion processing to generate the topology instruction bimodal fusion hidden state at each time step. Each topology instruction bimodal fusion hidden state in the topology instruction bimodal fusion hidden state sequence and the historical scheduling experience hidden state corresponding to the scheduling device name entity node in the historical scheduling experience hidden state sequence are input into the experience-enhanced cross-attention layer of the multimodal fusion inference branch. The third cross-attention weight of each topology instruction bimodal fusion hidden state to each historical scheduling experience hidden state is calculated. The historical scheduling experience hidden state sequence is weighted and summed using the third cross-attention weight to generate the experience-enhanced topology instruction fusion hidden state sequence. The empirically enhanced topology instruction fusion hidden state sequence is input into the causal Transformer inference layer of the multimodal fusion inference branch. The causal self-attention masking mechanism of the causal Transformer inference layer encodes the causal dependency of scheduling operations between the fusion hidden state of the preceding time step and the fusion hidden state of the following time step in the empirically enhanced topology instruction fusion hidden state sequence, generating the multimodal joint inference hidden state for each time step. The multimodal joint inference hidden states of all time steps output by the causal Transformer inference layer are combined in chronological order into a multimodal joint inference hidden state sequence, and the multimodal joint inference hidden state sequence is output to the output interface of the multimodal fusion inference branch.

6. The power grid dispatching decision support method based on a multimodal large model according to claim 1, characterized in that, The step of generating a target scheduling decision scheme by generating a scheduling decision from the scheduling decision decoding branch in the multimodal large model of the scheduling scenario, based on the hidden state sequence of the multimodal joint inference, includes: The multimodal joint reasoning hidden state sequence is input into the scheduling step sequence decoder of the scheduling decision decoding branch. The scheduling step sequence decoder performs scheduling operation step label classification processing on each multimodal joint reasoning hidden state in the multimodal joint reasoning hidden state sequence, generates scheduling operation step label corresponding to each time step, and combines the scheduling operation step labels of consecutive time steps into a scheduling operation step sequence. The multimodal joint inference hidden states that are time-aligned with the labels of each scheduling operation step in the scheduling operation step sequence are extracted from the multimodal joint inference hidden state sequence and input into the scheduling equipment location decoder of the scheduling decision decoding branch. The scheduling equipment location decoder performs position prediction processing on the corresponding scheduling equipment node in the power grid topology for each multimodal joint inference hidden state, and generates the target scheduling equipment node number corresponding to each scheduling operation step label. The multimodal joint inference hidden state in the multimodal joint inference hidden state sequence, which is time-aligned with the label of each scheduling operation step in the scheduling operation step sequence, is input into the scheduling operation execution timing decoder of the scheduling decision decoding branch. The scheduling operation execution timing decoder performs operation execution time point offset prediction processing on each multimodal joint inference hidden state to generate the operation execution time offset corresponding to each scheduling operation step label. Obtain the current time reference value of the power grid dispatch center, sum the operation execution time offset corresponding to each dispatch operation step label with the current time reference value of the power grid dispatch center to obtain the absolute execution time point corresponding to each dispatch operation step label, and combine the absolute execution time points corresponding to all dispatch operation step labels into a set of dispatch operation execution time points. The target scheduling decision scheme is generated by combining the sequence of scheduling operation steps, the target scheduling device node number corresponding to each scheduling operation step label, and the set of scheduling operation execution time points. The target scheduling decision scheme includes the sequence of scheduling operation steps and the set of scheduling operation execution time points. The target scheduling decision scheme is output from the output interface of the scheduling decision decoding branch, and then sent to the power grid scheduling execution terminal through the output layer of the multimodal large model of the scheduling scenario.

7. The power grid dispatching decision support method based on a multimodal large model according to claim 1, characterized in that, The method further includes: Before inputting the multimodal scheduling reference data stream into the pre-constructed multimodal scheduling scenario model, a set of historical power grid scheduling scenarios is obtained. The set of historical power grid scheduling scenarios includes historical real-time electrical topology image stream samples, historical scheduling voice command stream samples, historical scheduling text report stream samples, and corresponding historical scheduling decision scheme real annotations. The historical real-time electrical topology image stream samples are input into the topology graph encoding branch to be trained. The power grid topology connection relationship parsing processing is performed on each frame of the historical real-time electrical topology image stream samples to generate a historical topology connection relationship matrix sequence. The historical topology connection relationship matrix sequence is input into the graph convolutional topology feature extraction network. The topology evolution encoding processing is performed on the topology connection relationship changes between adjacent frames to obtain the historical topology operation status hidden state sequence samples. The historical scheduling voice command stream samples are input into the speech coding branch to be trained. The historical scheduling voice command stream samples are processed by scheduling command voice endpoint detection and scheduling domain adaptive speech recognition to obtain historical scheduling terminology text sequence samples. The historical scheduling terminology text sequence samples are processed by scheduling semantic role labeling and command intent encoding to generate historical scheduling command intent hidden state sequence samples. The historical scheduling text report stream sample is input into the text encoding branch to be trained. The historical scheduling text report stream sample is processed to extract scheduling operation events, extracting historical scheduling operation step record text and historical scheduling operation effect description text. The historical scheduling operation step record text and historical scheduling operation effect description text are jointly encoded to generate historical scheduling experience hidden state sequence sample. The historical topology operation status hidden state sequence sample, the historical scheduling instruction intent hidden state sequence sample, and the historical scheduling experience hidden state sequence sample are input into the multimodal fusion reasoning branch to be trained for cross-modal collaborative reasoning processing to generate historical multimodal joint reasoning hidden state sequence sample. The historical multimodal joint reasoning hidden state sequence sample is then input into the scheduling decision decoding branch to be trained to generate a predictive scheduling decision scheme. Calculate the sequence difference loss value of scheduling operation steps and the actual annotation of the historical scheduling decision scheme and the sequence difference loss value of scheduling operation execution time point. Then, perform weighted merging of the sequence difference loss value of scheduling operation steps and the sequence difference loss value of scheduling operation execution time point to obtain the multi-task joint training loss value. The network weights of the topology graph encoding branch, the speech encoding branch, the text encoding branch, the multimodal fusion inference branch, and the scheduling decision decoding branch are synchronously updated using the multi-task joint training loss value, thereby completing one round of iterative training of the multimodal large model of the scheduling scenario.

8. The power grid dispatching decision support method based on a multimodal large model according to claim 1, characterized in that, The method further includes: After obtaining the multimodal scheduling reference data stream collected by the power grid dispatch center, the real-time electrical topology image at the current moment and the historical electrical topology image at the previous moment are obtained from the real-time electrical topology image stream. The real-time electrical topology image at the current moment and the historical electrical topology image at the previous moment are subjected to pixel-level differential operation to generate an electrical topology differential image. The electrical topology differential image is subjected to connected component detection processing to extract the set of connected regions in the electrical topology differential image where the pixel change value exceeds a preset change threshold. The geometric center coordinates and area of ​​each connected region in the set of connected regions are calculated to obtain the geometric center coordinates and area of ​​each connected region. Based on the geometric center coordinates of each connected region, locate the corresponding electrical equipment graphic symbol in the real-time electrical topology map image at the current moment, identify the equipment type mark and voltage level mark of the electrical equipment graphic symbol, and combine the equipment type mark, the voltage level mark, the geometric center coordinates of the connected region and the area of ​​the connected region into an electrical topology change event record; The electrical topology change event records corresponding to multiple consecutive frames of real-time electrical topology images in the real-time electrical topology image stream are arranged in chronological order to generate an electrical topology change event sequence. The time interval and spatial distance of two adjacent electrical topology change event records in the electrical topology change event sequence are calculated to obtain the event interval duration parameter and the event spatial offset parameter. Based on the event interval duration parameter and the event spatial offset parameter, the electrical topology change event sequence is subjected to event aggregation processing. Continuous electrical topology change event records with event interval duration parameters less than the interval duration threshold and event spatial offset parameters less than the spatial offset threshold are merged into the same power grid disturbance event candidate, generating a power grid disturbance event candidate set. For each power grid disturbance event candidate in the power grid disturbance event candidate set, disturbance intensity assessment is performed. The number of electrical topology change event records included in each power grid disturbance event candidate and the number of different voltage level types involved in the electrical equipment graphic symbols in each power grid disturbance event candidate are counted to obtain the disturbance spread parameter and the disturbance cross-voltage level parameter. Candidate power grid disturbance events whose disturbance reach parameter exceeds the reach threshold or whose disturbance cross-voltage level parameter exceeds the cross-voltage level threshold are marked as valid power grid disturbance events. The real-time electrical topology image frame intervals corresponding to the power grid disturbance event candidates marked as valid power grid disturbance events are extracted from the real-time electrical topology image stream to generate a real-time electrical topology image stream that has been pre-screened for power grid disturbance events.

9. The power grid dispatching decision support method based on a multimodal large model according to claim 1, characterized in that, The method further includes: After the target scheduling decision scheme is sent to the power grid scheduling execution terminal, the power grid scheduling execution terminal receives a real-time scheduling operation execution feedback data stream during the process of executing the scheduling operation step sequence in the target scheduling decision scheme. The scheduling operation execution feedback data stream includes a sequence of measured changes in power grid electrical quantities after each scheduling operation step is executed and a sequence of snapshot images of substation status images when the scheduling operation step is executed. The measured change value sequence of the power grid electrical quantity is compared with the reference interval of the expected change of electrical quantity in the corresponding scheduling operation step in the target scheduling decision scheme by performing deviation direction analysis and deviation magnitude analysis. This generates an electrical quantity execution deviation direction mark and an electrical quantity execution deviation magnitude level mark for each scheduling operation step. The electrical quantity execution deviation direction mark includes a positive deviation mark and a negative deviation mark. The electrical quantity execution deviation magnitude level mark is obtained by mapping the electrical quantity deviation percentage. Image semantic comparison processing is performed on each substation status image snapshot in the substation status image snapshot sequence and the expected substation target status description of the corresponding scheduling operation step in the target scheduling decision scheme. The status features of the circuit breaker indicator and the appearance status features of the substation are extracted from the substation status image snapshot. The status features of the circuit breaker indicator are matched with the expected status description of the circuit breaker indicator to generate a confirmation result of the equipment operation in place. The electrical quantity execution deviation direction mark, electrical quantity execution deviation magnitude level mark, and equipment operation in place confirmation result of each scheduling operation step are combined into the execution effect evaluation record of that scheduling operation step. The execution effect evaluation records of all scheduling operation steps in the scheduling operation step sequence are arranged in the execution order to generate the scheduling operation execution effect evaluation sequence. The scheduling operation execution effect evaluation sequence is input into a pre-constructed scheduling effect attribution analysis network. The scheduling effect attribution analysis network performs attribution analysis on the causal relationship between the electrical quantity execution deviation direction marker and electrical quantity execution deviation magnitude level marker appearing in the scheduling operation execution effect evaluation sequence and the scheduling operation action type and target scheduling equipment node number of the corresponding scheduling operation steps in the scheduling operation step sequence, and generates deviation attribution analysis results. Based on the deviation attribution analysis results, the operation execution time points or target scheduling equipment node numbers of the scheduling operation steps in the target scheduling decision scheme that are causally related to the deviation are adjusted, and a revised scheduling decision feedback scheme is generated and sent to the power grid scheduling execution terminal.

10. A power grid dispatching decision support system based on a multimodal large model, characterized in that, include: processor; A machine-readable storage medium for storing machine-executable instructions of the processor; The processor is configured to execute the power grid dispatch decision support method based on a multimodal large model as described in any one of claims 1 to 9 by executing the machine-executable instructions.

Images