A fine-tuning method, system and medium based on a multimodal model of power failure

Abstract

The present invention relates to the field of smart grid technology, and more particularly to a fine-tuning method, system, and medium based on a multimodal model of power faults. The method comprises generating a knowledge-preserving mask based on a pre-trained weight matrix of a large multimodal language model; generating a sparse low-rank matrix based on the spatiotemporal sparsity of a structured multimodal dataset; generating a conflict mitigation regularization term based on the knowledge-preserving mask and the sparse low-rank matrix, and constructing a power task joint loss function based on the conflict mitigation regularization term, fine-tuning the large multimodal language model to obtain a fine-tuned large multimodal language model; inputting real-time power grid operation data into the fine-tuned large multimodal language model for inference to obtain a power grid fault diagnosis result, and isolating and controlling the power grid fault section based on the power grid fault diagnosis result. The present invention utilizes the spatiotemporal sparsity modeling of power grid data and the conflict mitigation regularization strategy to efficiently fine-tune the large multimodal language model, thereby achieving rapid and accurate diagnosis of power grid faults.

Description

Technical Field

[0001] The present invention relates to the field of smart grid technology, and in particular to a fine-tuning method, system and medium based on a multimodal model of power failure. Background Art

[0002] With the development of smart grid systems, the integration of heterogeneous elements such as renewable energy, a high proportion of power electronic equipment, and distributed intelligent terminals has made the grid operating environment increasingly complex. To meet this challenge, intelligent technology based on the Multimodal Large Language Model (MLLM) has been introduced into the power grid field. It aims to process multimodal information including text data such as voltage, current or circuit breaker status data, grid topology data, grid equipment status description, and image and video data to support key tasks such as fault diagnosis, load forecasting, and scheduling optimization.

[0003] However, in actual deployment, efficient migration of large multimodal language models in power scenarios is difficult. Traditional fine-tuning methods usually rely on full parameter optimization. Although they can improve the performance of the model on specific tasks, the computing resource requirements are too high, making it difficult to meet the real-time and stability requirements of the power grid system. Although parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) reduce training overhead by introducing low-rank matrices, such methods still have significant parameter redundancy problems in power grid scenarios, resulting in inefficient model deployment and unable to meet the lightweight and fast response requirements of new power grids. In addition, existing methods are not sufficiently adaptable to the characteristics of power data. Power grid multimodal data has strong spatiotemporal correlation, high noise tolerance requirements, and dynamic evolution characteristics. Traditional low-rank adaptive methods are not optimized for such structures, resulting in low efficiency of the model in extracting key features (such as power grid transient processes and equipment abnormal modes), limiting its performance in complex power tasks.

[0004] In summary, existing fine-tuning methods are difficult to meet the requirements of new power grid systems for efficient deployment, long-term stability, and multimodal collaborative reasoning. Therefore, there is an urgent need for an optimization method for power scenarios to improve the model's perception and reasoning capabilities for multimodal data of the power grid and meet the high requirements of the new power grid system for intelligence level and operational efficiency. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a fine-tuning method, system and medium based on a multi-modal model of power faults.

[0006] In a first aspect, the present invention provides a fine-tuning method based on a multimodal model of power faults, the method comprising:

[0007] Collect multi-source heterogeneous data generated during power grid operation to obtain structured multimodal data sets;

[0008] Selecting a large multimodal language model, and generating an amplitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the large multimodal language model;

[0009] Performing power feature-oriented sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal dataset to generate a sparse low-rank matrix;

[0010] generating a conflict mitigation regularization term based on the knowledge-preserving mask and the sparsified low-rank matrix, and constructing a power task joint loss function based on the conflict mitigation regularization term;

[0011] According to the structured multimodal dataset, fine-tuning the multimodal large language model using the power task joint loss function to obtain a fine-tuned multimodal large language model;

[0012] The real-time operation data of the power grid is input into the fine-tuned multimodal large language model for reasoning to obtain a power grid fault diagnosis result, and the power grid fault section is isolated and controlled according to the power grid fault diagnosis result.

[0013] In a further embodiment, generating a magnitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the multimodal large language model comprises:

[0014] Calculating a ratio of an absolute value of the pre-trained weight matrix to a Euclidean norm of the pre-trained weight matrix to obtain a normalized weight;

[0015] Calculating the reciprocal absolute value of the logarithm of the normalized weight with the natural logarithm as the base to obtain an importance score;

[0016] A preset steepness control parameter is introduced, and the importance score is smoothed by a hyperbolic tangent function to generate a knowledge-preserving mask.

[0017] In a further embodiment, the power feature-oriented sparsity modeling is performed based on the spatiotemporal sparse characteristics of the structured multimodal dataset to generate a sparse low-rank matrix, comprising:

[0018] quantifying the spatiotemporal sparsity of the power grid according to the spatial uneven distribution characteristics and temporal fluctuation characteristics of the structured multimodal dataset to obtain the spatiotemporal sparsity of the structured multimodal dataset;

[0019] Performing low-rank decomposition on the model parameters of the multimodal large language model to obtain two initialized low-rank matrices, and performing knowledge warm-up training on the initialized low-rank matrices to obtain the original low-rank matrix;

[0020] Performing feature extraction on the structured multimodal data set to obtain power grid abnormality features;

[0021] Using the absolute value of the elements of the original low-rank matrix as a parameter importance measurement indicator, generating a power grid anomaly feature mask matrix of the same size as the original low-rank matrix according to the spatiotemporal sparsity and the power grid anomaly feature;

[0022] The original low-rank matrix is ​​pruned to obtain a pruned low-rank matrix, and the power grid anomaly feature mask matrix is ​​element-wise multiplied by the pruned low-rank matrix to obtain a sparse low-rank matrix.

[0023] In a further embodiment, generating a conflict mitigation regularization term based on the knowledge preserving mask and the sparsified low-rank matrix comprises:

[0024] By calculating the product of different sparsified low-rank matrices, a low-rank incremental matrix is ​​obtained;

[0025] Multiplying the knowledge-preserving mask by the low-rank increment matrix element-by-element to obtain a mask increment matrix;

[0026] Compute the Frobenius norm of the masked delta matrix to obtain the conflict mitigation regularization term.

[0027] In a further embodiment, the power task joint loss function includes a power fault classification task loss and a conflict mitigation regularization term loss.

[0028] In a further embodiment, constructing a power task joint loss function based on a conflict mitigation regularization term includes:

[0029] Performing multimodal encoding on the structured multimodal dataset to obtain a multimodal feature vector;

[0030] Inputting the multimodal feature vector into the multimodal large language model to perform fault classification prediction to obtain a fault classification prediction result;

[0031] Calculating the cross entropy loss between the fault classification prediction result and the true label to obtain the power fault classification task loss;

[0032] Performing soft sparsity regularization on the knowledge-preserving mask and the sparsified low-rank matrix to generate a conflict mitigation regularization term;

[0033] According to the power fault classification task loss and the conflict mitigation regularization term, a power task joint loss function is constructed using weighted summation.

[0034] In a further embodiment, the fine-tuning training of the multimodal large language model using the power task joint loss function based on the structured multimodal dataset to obtain a fine-tuned multimodal large language model includes:

[0035] Comparing the current number of training steps with a preset hot step threshold, and entering the knowledge warm-up phase when the current number of training steps is less than the preset hot step threshold;

[0036] In the knowledge preheating stage, the total loss value of the power task is calculated using the power task joint loss function according to the structured multimodal data set;

[0037] Back-propagating the total power task loss value to update the sparse low-rank matrix, generating a preliminary optimized low-rank matrix at the end of the knowledge warm-up phase;

[0038] Generate a sparse mask matrix according to the preliminary optimized low-rank matrix and the preset fault feature sparsity parameter, and enter the knowledge integration stage when the current training step number is not less than the preset warm-up step number threshold;

[0039] In the knowledge integration stage, the preliminary optimized low-rank matrix is ​​multiplied element-by-element by the sparsified mask matrix to obtain a key sparsified low-rank matrix;

[0040] Forward propagating the structured multimodal dataset, and backpropagating the power task joint loss function to update the key sparse low-rank matrix to obtain a fine-tuned sparse low-rank matrix;

[0041] Fine-tune the multimodal large language model according to the fine-tuned sparse low-rank matrix to obtain a fine-tuned multimodal large language model.

[0042] In a further embodiment, fine-tuning the multimodal large language model according to the fine-tuned sparsified low-rank matrix to obtain a fine-tuned multimodal large language model includes:

[0043] Multiplying the fine-tuned sparsified low-rank matrix to obtain a fine-tuning increment matrix, and linearly superimposing the fine-tuning increment matrix with the pre-training weight matrix to obtain a pre-training update weight value;

[0044] The multimodal large language model is fine-tuned using the pre-trained updated weight values ​​to obtain a fine-tuned multimodal large language model.

[0045] In a second aspect, the present invention provides a fine-tuning system based on a multimodal model of power faults, the system comprising:

[0046] The data acquisition module is used to collect multi-source heterogeneous data generated during the operation of the power grid and obtain a structured multimodal data set;

[0047] a mask generation module, configured to select a large multimodal language model and generate an amplitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the large multimodal language model;

[0048] A low-rank sparse module, configured to perform power feature-oriented sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal dataset and generate a sparse low-rank matrix;

[0049] a loss construction module, configured to generate a conflict mitigation regularization term based on the knowledge-preserving mask and the sparsified low-rank matrix, and construct a power task joint loss function based on the conflict mitigation regularization term;

[0050] A model construction module, configured to fine-tune the multimodal large language model using the power task joint loss function according to the structured multimodal dataset to obtain a fine-tuned multimodal large language model;

[0051] The fault isolation module is used to input the real-time operation data of the power grid into the fine-tuned multimodal large language model for inference, obtain the power grid fault diagnosis result, and isolate and control the power grid fault section according to the power grid fault diagnosis result.

[0052] In a further embodiment, generating a magnitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the multimodal large language model comprises:

[0053] Calculating a ratio of an absolute value of the pre-trained weight matrix to a Euclidean norm of the pre-trained weight matrix to obtain a normalized weight;

[0054] Calculating the reciprocal absolute value of the logarithm of the normalized weight with the natural logarithm as the base to obtain an importance score;

[0055] A preset steepness control parameter is introduced, and the importance score is smoothed by a hyperbolic tangent function to generate a knowledge-preserving mask.

[0056] In a further embodiment, the power feature-oriented sparsity modeling is performed based on the spatiotemporal sparse characteristics of the structured multimodal dataset to generate a sparse low-rank matrix, comprising:

[0057] quantifying the spatiotemporal sparsity of the power grid according to the spatial uneven distribution characteristics and temporal fluctuation characteristics of the structured multimodal dataset to obtain the spatiotemporal sparsity of the structured multimodal dataset;

[0058] Performing low-rank decomposition on the model parameters of the multimodal large language model to obtain two initialized low-rank matrices, and performing knowledge warm-up training on the initialized low-rank matrices to obtain the original low-rank matrix;

[0059] Performing feature extraction on the structured multimodal data set to obtain power grid abnormality features;

[0060] Using the absolute value of the elements of the original low-rank matrix as a parameter importance measurement indicator, generating a power grid anomaly feature mask matrix of the same size as the original low-rank matrix according to the spatiotemporal sparsity and the power grid anomaly feature;

[0061] The original low-rank matrix is ​​pruned to obtain a pruned low-rank matrix, and the power grid anomaly feature mask matrix is ​​element-wise multiplied by the pruned low-rank matrix to obtain a sparse low-rank matrix.

[0062] In a further embodiment, generating a conflict mitigation regularization term based on the knowledge preserving mask and the sparsified low-rank matrix comprises:

[0063] By calculating the product of different sparsified low-rank matrices, a low-rank incremental matrix is ​​obtained;

[0064] Multiplying the knowledge-preserving mask by the low-rank increment matrix element-by-element to obtain a mask increment matrix;

[0065] Compute the Frobenius norm of the masked delta matrix to obtain the conflict mitigation regularization term.

[0066] In a further embodiment, the power task joint loss function includes a power fault classification task loss and a conflict mitigation regularization term loss.

[0067] In a further embodiment, constructing a power task joint loss function based on a conflict mitigation regularization term includes:

[0068] Performing multimodal encoding on the structured multimodal dataset to obtain a multimodal feature vector;

[0069] Inputting the multimodal feature vector into the multimodal large language model to perform fault classification prediction to obtain a fault classification prediction result;

[0070] Calculating the cross entropy loss between the fault classification prediction result and the true label to obtain the power fault classification task loss;

[0071] Performing soft sparsity regularization on the knowledge-preserving mask and the sparsified low-rank matrix to generate a conflict mitigation regularization term;

[0072] According to the power fault classification task loss and the conflict mitigation regularization term, a power task joint loss function is constructed using weighted summation.

[0073] In a further embodiment, the fine-tuning training of the multimodal large language model using the power task joint loss function based on the structured multimodal dataset to obtain a fine-tuned multimodal large language model includes:

[0074] Comparing the current number of training steps with a preset warm-up step threshold, and entering the knowledge warm-up phase when the current number of training steps is less than the preset warm-up step threshold;

[0075] In the knowledge preheating stage, the total loss value of the power task is calculated using the power task joint loss function according to the structured multimodal data set;

[0076] Back-propagating the total power task loss value to update the sparse low-rank matrix, generating a preliminary optimized low-rank matrix at the end of the knowledge warm-up phase;

[0077] Generate a sparse mask matrix according to the preliminary optimized low-rank matrix and the preset fault feature sparsity parameter, and enter the knowledge integration stage when the current training step number is not less than the preset warm-up step number threshold;

[0078] In the knowledge integration stage, the preliminary optimized low-rank matrix is ​​multiplied element-by-element by the sparsified mask matrix to obtain a key sparsified low-rank matrix;

[0079] Forward propagating the structured multimodal dataset, and backpropagating the power task joint loss function to update the key sparse low-rank matrix to obtain a fine-tuned sparse low-rank matrix;

[0080] Fine-tune the multimodal large language model according to the fine-tuned sparse low-rank matrix to obtain a fine-tuned multimodal large language model.

[0081] In a further embodiment, fine-tuning the multimodal large language model according to the fine-tuned sparsified low-rank matrix to obtain a fine-tuned multimodal large language model includes:

[0082] Multiplying the fine-tuned sparsified low-rank matrix to obtain a fine-tuning increment matrix, and linearly superimposing the fine-tuning increment matrix with the pre-training weight matrix to obtain a pre-training update weight value;

[0083] The multimodal large language model is fine-tuned using the pre-trained updated weight values ​​to obtain a fine-tuned multimodal large language model.

[0084] In a third aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0085] The present invention provides a fine-tuning method, system and medium based on a multimodal model of power faults. The method obtains a structured multimodal dataset by collecting multi-source heterogeneous data generated during the operation of a power grid; selects a large multimodal language model, and generates an amplitude-guided knowledge-retention mask based on the pre-trained weight matrix of the large multimodal language model; performs power feature-guided sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal dataset to generate a sparse low-rank matrix; generates a conflict mitigation regularization term based on the knowledge-retention mask and the sparse low-rank matrix, and constructs a power task joint loss function based on the conflict mitigation regularization term; fine-tunes the multimodal large language model based on the structured multimodal dataset using the power task joint loss function to obtain a fine-tuned multimodal large language model; inputs real-time power grid operation data into the fine-tuned multimodal large language model for inference to obtain a power grid fault diagnosis result, and isolates and controls the power grid fault section based on the power grid fault diagnosis result. Compared with the existing technology, this method uses multi-source heterogeneous data of the power grid, and utilizes the spatiotemporal sparsity modeling and conflict mitigation regularization strategy to efficiently fine-tune the multimodal large language model, thereby achieving rapid and accurate diagnosis of power grid faults and automatic isolation and control of fault sections, improving the intelligent, safe operation and stability of the power grid system. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 1 is a flow chart of a fine-tuning method based on a multimodal model of power faults provided by an embodiment of the present invention;

[0087] Figure 2 Schematic diagram of a fine-tuning process based on a multimodal model of power failures provided by an embodiment of the present invention;

[0088] Figure 3 This is a block diagram of a fine-tuning system based on a power fault multimodal model provided by an embodiment of the present invention.

[0089] Explanation of the accompanying drawings: 101, data acquisition module; 102, mask generation module; 103, low-rank sparse module; 104, loss construction module; 105, model construction module; 106, fault isolation module. DETAILED DESCRIPTION

[0090] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings. The embodiments are provided for illustrative purposes only and are not to be construed as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of protection of the present invention. Many changes may be made to the present invention without departing from the spirit and scope of the present invention.

[0091] refer to Figure 1 ,The embodiment of the present invention provides a fine-tuning method based on a multi-modal model of power faults, such as Figure 1 As shown, the method includes the following steps:

[0092] S1. Collect multi-source heterogeneous data generated during power grid operation to obtain a structured multimodal dataset.

[0093] The current deployment of sensors in power grids has problems of spatial unevenness and insufficient density, especially at key locations such as the end of the distribution network and distributed energy access points. There are blind spots in monitoring methods, which leads to significant structural sparsity of data collection at the spatial level. In the task of power grid anomaly detection, when the power grid is in normal operation, the operating parameters of various sensors and devices (such as voltage, current and frequency, etc.) tend to be stable and have low fluctuation characteristics, with obvious data sparsity, while abnormal events (such as overload or single-phase grounding, etc.) show significant and local mutation characteristics, which makes the data show significant sparsity in spatial and temporal dimensions. Taking single-phase grounding fault as an example, the fault is manifested as a small mutation amplitude of zero-sequence current, a short duration, and is easily submerged by background noise. The abnormal signal in this type of abnormal event is not good. The signal has natural physical sparsity, which requires the model to have a stronger ability to perceive weak abnormal signals. To solve this problem, the present invention combines the data structure of the power grid system and the task characteristics, introduces a sparsity modeling strategy in the training stage, and directly performs sparsity constraints on the low-rank matrices B and A during the LoRA fine-tuning process to ensure that the BA matrix reaches the desired sparsity. By retaining key parameters that are strongly correlated with local mutation signals, the model's ability to perceive weak abnormal signals is improved. At the same time, through the sparsity modeling strategy, the fusion conflict between the pre-trained model parameters and the specific task parameters is reduced to avoid the forgetting of general knowledge. In addition, the sparsity characteristics of the power grid data are utilized to retain key parameters that are strongly correlated with local mutation signals, thereby enhancing the sensitivity and robustness of the model in the power grid anomaly detection task.

[0094] Specifically, the parameter fluctuations of power grid sensors under normal operating conditions are relatively stable and have small amplitudes, while abnormal events (such as single-phase grounding) will show local mutations and weak signals, which are easily masked by background noise. This characteristic gives the data physical sparsity, so the model must have the ability to keenly capture abnormal signals. Based on this, this embodiment introduces a sparsity modeling strategy in the model training phase, using parameter amplitude as a proxy indicator of parameter importance. After the warm-up phase is completed, the parameters associated with the low-rank matrix and steady-state data are pruned, retaining only key parameters with strong correlation with local mutation signals. In this way, the sensitivity and robustness of the model in the power grid anomaly detection task are improved. From a theoretical perspective, it can be seen that the sparsity of the product of two sparse low-rank matrices in LoRA has an upper bound, and high-probability sparsity can be obtained while maintaining the low-rank property, thereby theoretically ensuring that the model can continuously remember key safety rules and steady-state modes during long-term operation, and has fault tolerance in the case of structural disturbances and sensor fluctuations.

[0095] Considering the dual dependence of power grid tasks on general physical knowledge and electrical scenario knowledge, the model not only needs to master basic physical laws such as Kirchhoff's law, power flow equation, and energy conservation, but also needs to understand scenario characteristics such as regional topology and equipment aging to adapt to various power grid system tasks such as fault detection, load forecasting, and multi-source heterogeneous data processing. This dual dependence is particularly significant in fault detection tasks. The model needs to accurately identify the transient response characteristics of the power grid under non-steady-state disturbances (such as voltage sag, current distortion, frequency offset, etc.), and combine the power grid operation background knowledge such as relay protection devices, switch topology status and wiring methods to achieve real-time judgment and traceability of abnormal modes. In order to effectively alleviate the fusion conflict between pre-trained knowledge and task-specific knowledge, this embodiment proposes a knowledge-guided conflict mitigation regularizer. The conflict mitigation regularizer introduces pre-trained knowledge as a guiding signal during the training process. Through the regularization mechanism, the parameter update of LoRA in the key general knowledge area of ​​the pre-trained model is suppressed, and the newly introduced relevant knowledge in the power task is guided to migrate to the relatively minor parameter area of ​​the pre-trained model. When using LoRA to fine-tune the model parameters, the parameters of the pre-trained model are The numerical gradient cannot be accessed directly. This embodiment uses the parameter size as a metric for the importance of pre-trained weights and generates a size-guided retention mask to protect the pre-trained general physical knowledge. The mask can dynamically identify and protect the key general knowledge in the pre-trained model to prevent it from being over-modified during the fine-tuning process, thereby ensuring that the model retains its ability to understand general physical laws while adapting to new tasks, while guiding the electrical scene-related knowledge to migrate to the relatively minor parameter areas in the pre-trained model. In addition, in combination with the conflict mitigation regularization mechanism, the expected sparsity of LoRA is The property and its upper bound still hold. The theoretical completeness and robustness of this embodiment are fully proved through mathematical derivation. In terms of efficient parameter fine-tuning of multimodal large model connectors, after adjusting the hard pruning scheme to a soft sparsity regularization method, this method can also be used to solve the knowledge forgetting phenomenon caused by parameter redundancy in the connector module, so that it can efficiently align multimodal power information including graph structures, numerical streams, text descriptions, and graphic images, realize efficient parameter fine-tuning of multimodal large model connectors, and improve the performance and robustness of the model in tasks such as power grid anomaly detection.

[0096] This embodiment collects multi-source heterogeneous data generated during the operation of the power grid through various sensors installed in the power grid. The multi-source heterogeneous data includes electrical operation data, equipment visual data and text description data. The electrical operation data includes time series data of voltage, current and frequency obtained from the SCADA system, PMU device and fault recorder; the equipment visual data includes infrared images, visible light images and real-time monitoring video streams of the equipment collected by inspection robots and monitoring cameras; the text description data includes text description information extracted from scheduling logs, equipment maintenance records and protection action reports. In view of the data sparsity caused by the uneven spatial distribution of sensors and blind spots in key node monitoring, the multi-source heterogeneous data are pre-processed by spatiotemporal alignment, noise suppression and anomaly enhancement. The spatiotemporal alignment operation is based on a unified timestamp and spatial coordinate reference system. The electrical operation data and the equipment visual data are synchronized in time and space to solve the monitoring blind spot problem caused by the uneven spatial distribution of sensors; the noise suppression and anomaly enhancement operation uses wavelet transform to filter out background noise from the electrical operation data, and enhances local abnormal signals based on the zero-sequence current mutation characteristics, performs background difference and adaptive threshold segmentation on the equipment visual data, and extracts local hot spots and mechanical deformation areas of the equipment; then, this embodiment performs modal structured encoding on the preprocessed multi-source heterogeneous data, specifically converting the electrical operation data into a standardized numerical matrix, extracting feature vectors from the equipment visual data through a pre-trained visual encoder (such as ResNet), converting the text description data into a word embedding sequence through a Tokenizer, and organizing the preprocessed data into a structured data set according to sample-modality pairs to obtain a structured multimodal data set.

[0097] S2. Select a multimodal large language model, and generate an amplitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the multimodal large language model.

[0098] In some embodiments, the step of generating a magnitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the multimodal large language model includes:

[0099] Calculating a ratio of an absolute value of the pre-trained weight matrix to a Euclidean norm of the pre-trained weight matrix to obtain a normalized weight;

[0100] Calculating the reciprocal absolute value of the logarithm of the normalized weight with the natural logarithm as the base to obtain an importance score;

[0101] A preset steepness control parameter is introduced, and the importance score is smoothed by a hyperbolic tangent function to generate a knowledge-preserving mask.

[0102] This embodiment selects a multimodal large language model suitable for a structured multimodal data set according to the specific needs of the power grid task, and uses the selected multimodal large language model as a power fault multimodal model. The power grid task can be fault detection and load forecasting, etc. For the pre-trained weight matrix of the multimodal large language model, this embodiment first calculates the absolute value and Euclidean norm of the pre-trained weight, and calculates the normalized weight matrix based on the ratio of the absolute value and Euclidean norm of the pre-trained weight, thereby eliminating the weight scale difference and ensuring the comparability of cross-layer parameters. Then, this embodiment performs logarithmic compression on the normalized weight to generate an importance score matrix, solves the long-tail distribution problem of the weight amplitude, and highlights the key parameters. Finally, a steepness control parameter is introduced, and the importance score is mapped to the (0, 1) interval through the hyperbolic tangent function to generate an amplitude-guided knowledge retention mask. The high-score area approaches 1 to strongly protect the general knowledge. In this embodiment, the calculated parameter mask M is used as the knowledge retention mask. The knowledge retention mask is used to guide the parameter update during the model fine-tuning process. During the fine-tuning process, parameters with high importance scores (parameters with mask values ​​close to 1) will be retained and updated, while parameters with low importance scores (parameters with mask values ​​close to 0) will be suppressed from being updated, thereby protecting the core part of the general physical knowledge in the pre-trained model and allowing the model to learn specific knowledge in the power grid scenario. The calculation formula of the knowledge retention mask is:

[0103]

[0104]

[0105] Where W is the pre-trained weight matrix of the multimodal large language model; is the absolute value of the pre-trained weight matrix; is the Euclidean norm of the pre-trained weight matrix, which is used to normalize the weights to ensure the stability of the calculation; log is the natural logarithm function, which is used to enhance the discrimination of weight differences; is a very small positive number used to prevent overflow caused by zero input to the logarithmic function, ensuring the stability and computability of the formula; S is the normalized weight. The larger the normalized weight value, the more critical the parameter is to general knowledge. is a preset steepness control parameter, which is used to control the steepness of mask generation; It is an element-level multiplication operation; is the hyperbolic tangent function, which is used to map the importance score to the (0, 1) interval; M is the knowledge preservation mask, which is used to protect the common physical knowledge in pre-training, such as Kirchhoff's law and power flow equation.

[0106] S3. Perform power feature-guided sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal dataset to generate a sparse low-rank matrix.

[0107] In some embodiments, the step of performing power feature-guided sparsity modeling based on the spatiotemporal sparsity characteristics of the structured multimodal dataset to generate a sparse low-rank matrix includes:

[0108] quantifying the spatiotemporal sparsity of the power grid according to the spatial uneven distribution characteristics and temporal fluctuation characteristics of the structured multimodal dataset to obtain the spatiotemporal sparsity of the structured multimodal dataset;

[0109] Performing low-rank decomposition on the model parameters of the multimodal large language model to obtain two initialized low-rank matrices, and performing knowledge warm-up training on the initialized low-rank matrices to obtain the original low-rank matrix;

[0110] Performing feature extraction on the structured multimodal data set to obtain power grid abnormality features;

[0111] Using the absolute value of the elements of the original low-rank matrix as a parameter importance measurement indicator, generating a power grid anomaly feature mask matrix of the same size as the original low-rank matrix according to the spatiotemporal sparsity and the power grid anomaly feature;

[0112] The original low-rank matrix is ​​pruned to obtain a pruned low-rank matrix, and the power grid anomaly feature mask matrix is ​​element-wise multiplied by the pruned low-rank matrix to obtain a sparse low-rank matrix.

[0113] Specifically, Figure 2 This is a schematic diagram of the fine-tuning process based on a multimodal model for power failures provided by an embodiment of the present invention. This embodiment analyzes the spatial distribution of power grid equipment (such as the geographic distribution of substations and transmission lines) in a structured multimodal dataset and statistically analyzes the density differences of sensor data at different spatial locations in the structured multimodal dataset. For example, the proportion of sensor data in each region to the total data volume is calculated. If the distribution of the proportions is extremely uneven, it indicates significant spatial sparsity. In other embodiments, this embodiment may also use a clustering algorithm to determine data-dense and data-sparse areas. The clustering results are used to quantify the degree of spatial distribution unevenness, thereby identifying areas with concentrated and sparse power grid equipment density and obtaining spatial distribution unevenness characteristics. Simultaneously, this embodiment analyzes the frequency and amplitude of changes in the structured multimodal dataset in the temporal dimension and calculates the average of the absolute values ​​of the differences between data at adjacent time points. A large average value indicates significant temporal data fluctuations and high sparsity. This embodiment quantifies temporal fluctuation characteristics by calculating features such as the average of the absolute values ​​of the differences between data at adjacent time points. This embodiment weightedly fuses the spatial distribution unevenness and temporal fluctuation characteristics to obtain the spatiotemporal sparsity of the structured multimodal dataset, providing a basis for subsequent sparsity modeling.

[0114] Then, this embodiment performs low-rank decomposition on the pre-trained weight matrix W of the multimodal large language model, decomposing it into two initialized low-rank matrices, so that the original parameter matrix is ​​approximately the product of the two low-rank matrices, reducing the model parameter scale and computational complexity. In the knowledge preheating stage, this embodiment uses a structured multimodal data set to preheat training the two initialized low-rank matrices, so that the initialized low-rank matrices preliminarily learn the common features and rules in the structured multimodal data set, and obtain the original low-rank matrix A and the original low-rank matrix B. According to the business needs and data types of the power grid, for image data, this embodiment uses a convolutional neural network to extract high-level semantic features; for text data, it uses word embedding technology to convert it into a low-dimensional vector representation, thereby forming multimodal feature data, and according to the power grid operation rules and fault modes, screens out features that can characterize the abnormal state of the power grid from the multimodal feature data to obtain power grid abnormal features (such as zero-sequence current mutations and node adjacency anomalies). For example, this embodiment The embodiment extracts the mutation characteristics of the current signal through wavelet transform or extracts the node adjacency anomaly through graph neural network. Then, the embodiment uses the absolute value of the elements of the original low-rank matrix A and the original low-rank matrix B as the importance index. The larger the absolute value of the element, the higher the importance of the corresponding parameter in the model. The embodiment combines the spatiotemporal sparsity and the power grid anomaly characteristics to generate a mask matrix of the same size as the original low-rank matrix, and maps the power grid anomaly characteristics to the fault-sensitive area. For example, for each row of the original low-rank matrix A, the parameters that are highly correlated with the anomaly characteristics (such as parameters related to the zero-sequence current mutation) are retained, and the parameters related to the steady-state data are pruned. The embodiment can determine the overall sparsity of the mask matrix according to the spatiotemporal sparsity, and determine the element positions in the mask matrix that should retain larger values ​​(corresponding to key parameters) and the element positions that should be set to smaller values ​​or zero (corresponding to non-key parameters) through the power grid anomaly characteristics. In this embodiment, the power grid anomaly feature mask matrix generation formula is:

[0115]

[0116]

[0117] Where, is the power grid anomaly feature mask matrix of the original low-rank matrix A, the power grid anomaly feature mask matrix The power grid anomaly feature mask matrix is ​​the same size as the original low-rank matrix A. The elements of are 0 or 1, which are used to indicate which elements in the original low-rank matrix A are retained as 1 or set to zero during the pruning process; is the mask symbol; A and B are both original low-rank matrices; is the spatiotemporal sparsity of the original low-rank matrix A, which is used to control the degree of pruning of the original low-rank matrix A; is the power grid anomaly feature mask matrix of the original low-rank matrix B, the power grid anomaly feature mask matrix The size of the original low-rank matrix B is the same as that of the power grid anomaly feature mask matrix The elements of are 0 or 1, which are used to indicate which elements in the original low-rank matrix B are retained as 1 or set to zero during the pruning process; is the spatiotemporal sparsity of the original low-rank matrix B, which is used to control the degree of pruning of the original low-rank matrix B.

[0118] This embodiment prunes the parameters in the original low-rank matrix that are associated with steady-state data and change slowly based on the mask. Combined with the fault prior, the parameters strongly correlated with the abnormal power grid signal (such as the zero-sequence current mutation characteristics and the adjacency of important nodes) are pruned and retained. Elements with smaller absolute values ​​in the matrix are set to zero, and important elements with larger absolute values ​​are retained to obtain a pruned low-rank matrix. The power grid abnormality feature mask matrix and the pruned low-rank matrix are multiplied element by element. The pruned low-rank matrix elements corresponding to positions with large element values ​​in the mask matrix are retained and updated, and the pruned low-rank matrix elements corresponding to positions with small element values ​​are further suppressed or set to zero. Finally, a sparse low-rank matrix is ​​obtained to achieve power feature-oriented sparsity modeling and improve the model's sensitivity to weak fault signals. The calculation formula of the sparse low-rank matrix is:

[0119]

[0120] Where, The sparse low-rank matrix A is obtained by pruning the original low-rank matrix and retaining the key parameters, but most of the non-critical parameters associated with the steady-state data and changing slowly are set to zero; The sparse low-rank matrix after pruning the original low-rank matrix B and retaining the key parameters retains the parameters that are highly correlated with abnormal changes in the power grid.

[0121] For ease of understanding, this embodiment briefly explains the sparsity expectation theorem and the upper bound theorem. For the sparsity expectation theorem, this embodiment sets the original low-rank matrix A and the original low-rank matrix B as two low-rank matrices in LoRA, and the product of the original low-rank matrix A and the original low-rank matrix B is the product matrix BA. Then the expected sparsity of the product matrix BA is:

[0122]

[0123] Where, is the expected sparsity of the product matrix BA; is the actual sparsity of the product matrix BA (the proportion of zero elements); r is the rank of the original low-rank matrix. In matrix decomposition, the rank of the low-rank matrix reflects the intrinsic dimension or number of features of the matrix. In the power grid system, a smaller rank can effectively reduce the number of parameters and simplify the model structure; E(*) is the expected operator. When the grid feature dimension r increases, the model retains more fault features (such as and increase), but through Suppress the growth of redundant parameters and ensure that fine-tuning only modifies no more than The parameters of the proportion maintain the integrity of common knowledge such as Kirchhoff's laws.

[0124] This embodiment describes the specific proof process of the sparsity expectation theorem. The element in the i-th row and j-th column of the product matrix BA can be expressed as , assuming that the parameters in the original low-rank matrix A and the original low-rank matrix B are non-negative, then if and only if there is at least one , making and is non-zero, for each k, The probability of non-zero is , The probability of non-zero is , since the positions of the non-zero elements in the original low-rank matrix A and the original low-rank matrix B are independent and randomly distributed, the probability and The probability that both are non-zero is:

[0125]

[0126] Where, is the element in the i-th row and j-th column of the product matrix, which represents the association strength or weight between different nodes or features in the power grid system; is the element in the i-th row and k-th column of the original low-rank matrix B; is the element in the kth row and jth column of the original low-rank matrix B; k is the latent feature dimension index; for and The probability of being simultaneously non-zero. In the power grid system, this probability reflects the possibility of two related features or states existing at the same time, and is used to measure the correlation and sparsity between matrix elements; is a probability operator.

[0127] therefore, The probability of being 0 is:

[0128]

[0129] Assuming different If there is independence between The probability of both being 0 is:

[0130]

[0131] therefore, The probability of being non-zero is:

[0132]

[0133] Since the product matrix middle elements of , so the expected number of non-zero elements is:

[0134]

[0135] Therefore, the product matrix The expected value of the sparsity is:

[0136]

[0137] in, is the product matrix The total number of non-zero elements in ; p is the number of rows of the original low-rank matrix B, such as the number of sensors at different locations or the operating parameters of different types of equipment; q is the number of columns of the original low-rank matrix A, such as the electrical connection relationship between different nodes or the operating status changes of different devices; is the expected number of non-zero elements; is the actual sparsity of the product matrix BA; for The mathematical expectation of .

[0138] For the upper bound theorem, this embodiment assumes and are two low-rank matrices in LoRA, The sparsity is , The sparsity is ,make , The sparsity is , then for any :

[0139]

[0140] Where, The actual sparsity deviates from the expected value by more than probability; is the actual sparsity of matrix C; for The mathematical expectation of is the sparsity fluctuation tolerance threshold; is an exponential function.

[0141] in Expected value This has been given in the sparsity expectation theorem. This example briefly explains the proof process of the upper bound theorem. According to the McDiarmids inequality, if is a collection The independent random variables in Satisfy the bounded difference condition: for all and all have

[0142]

[0143] Where, is the set of values ​​of a random variable, for example, all possible monitoring value ranges of power grid sensors; is the nth independent random variable in the set X; is the n-dimensional random variable space; is the sparsity calculation function; is the nth element in the input variable of function f; is a random variable The disturbance value of ; is the i-th random variable; For the function f at a given value The output below; For the function f at a given value The output below; is the bounded difference coefficient of the function f with respect to the i-th random variable.

[0144] Then for all Both:

[0145]

[0146] And similarly:

[0147]

[0148] Where, For the function f at a given value The output below; is the expected value of function f; The difference between the output value of function f and the expected value is not less than The probability of n is the total number of random variables.

[0149] In this embodiment, Considered as a representation Function of the total number of nonzero elements in :

[0150]

[0151] in is an indicator variable, satisfying:

[0152]

[0153] Where, is the total number of non-zero elements in matrix C; is the indicator random variable; is the element in the i-th row and j-th column of matrix C.

[0154] Each All depend on random variables ,variable and is independent and through Influence , on this basis we can get the bounded difference:

[0155] Change Effect: Changes will affect all ,in , The changes can The biggest change caused by ;

[0156] Change Effect: Changes will affect all ,in , The changes can The biggest change caused by .

[0157] Therefore, the sum of squares of bounded differences is:

[0158]

[0159] Where, To change the elements of matrix B The maximum impact on the total number of non-zero elements N, that is, when adjusting the weight of the k-th fault feature on the i-th grid node When , it affects the parameters of all q-type physical quantities (such as voltage or current, etc.) associated with the node; To change the elements of matrix A The maximum influence on the total number of non-zero elements N when .

[0160] This embodiment is based on McDiarmids inequality and obtains the probability upper bound of sparsity through bounded difference analysis. For any have:

[0161]

[0162]

[0163] because ,available:

[0164]

[0165] make ,but , substituting into the original inequality we can get:

[0166]

[0167] Where, The actual number of non-zero elements N exceeds its expected value At least probability; is the expected value of the number of non-zero elements in the matrix C; The actual sparsity Deviation from expected sparsity At least probability.

[0168] S4. Generate a conflict mitigation regularization term based on the knowledge retention mask and the sparsified low-rank matrix, and construct a power task joint loss function based on the conflict mitigation regularization term.

[0169] In some embodiments, the power task joint loss function includes a power fault classification task loss and a conflict mitigation regularization term loss, and the step of constructing the power task joint loss function based on the conflict mitigation regularization term includes:

[0170] Performing multimodal encoding on the structured multimodal dataset to obtain a multimodal feature vector;

[0171] Inputting the multimodal feature vector into the multimodal large language model to perform fault classification prediction to obtain a fault classification prediction result;

[0172] Calculating the cross entropy loss between the fault classification prediction result and the true label to obtain the power fault classification task loss;

[0173] Performing soft sparsity regularization on the knowledge-preserving mask and the sparsified low-rank matrix to generate a conflict mitigation regularization term;

[0174] According to the power fault classification task loss and the conflict mitigation regularization term, a power task joint loss function is constructed using weighted summation.

[0175] In order to optimize the update trajectory of LoRA parameters, this embodiment proposes a calibration method based on the general knowledge of the pre-training model. Through the amplitude-guided retention mask mechanism, the new task-related knowledge is guided to migrate to the relatively minor parameter area in the pre-training model, thereby alleviating the parameter conflict between the new task and the pre-training task. Specifically, this embodiment uses the amplitude of the pre-training model parameters as the importance metric. , this embodiment calculates its normalized amplitude (such as through layer normalization or global statistics), and then the importance index is obtained through the following function mapping:

[0176]

[0177] Where, Pre-trained weights The knowledge importance score is used to quantify the criticality of the parameters in the pre-trained model to the general knowledge of the power grid (such as Kirchhoff's law); Pre-trained weights The normalized amplitude value of , which eliminates the dimensional differences of physical quantities in different power grids; is a very small positive number.

[0178] Based on knowledge importance score , a knowledge-preserving mask is generated by the hyperbolic tangent function, mapping the importance to The specific mathematical expression is:

[0179]

[0180] Where, The mask is for knowledge retention. The closer its value is to 1, the more important the parameter is and the more it needs to be retained. is the preset steepness control parameter; e is the natural constant base of the exponential function.

[0181] This embodiment obtains the conflict mitigation regularization term by taking the Frobenius norm of the Hadamard product (element-wise product) of the knowledge-preserving mask and the LoRA low-rank matrix product BA:

[0182]

[0183] Where, To mitigate the conflict regularization loss, the update of BA is suppressed in the high value area of ​​mask M to avoid forgetting the physical laws. In the low value area of ​​mask M, the product matrix BA is allowed to update freely to learn fault diagnosis features. Preserve mask matrix for knowledge; is the product matrix of two low-rank matrices B and A; is the Frobenius norm, that is, the square root of the sum of the squares of all elements in the matrix. In this embodiment, the conflict mitigation regularization term is constructed by applying the Frobenius norm to the Hadamard product, so that the model can effectively balance the relationship between pre-training knowledge retention and specific task knowledge learning during fine-tuning.

[0184] The core of the regularization strategy proposed in this embodiment is to use the parameter amplitude information of the pre-trained model to construct a dynamic knowledge retention mask, thereby selectively protecting key knowledge during the parameter update process and guiding the learning direction of new task-related knowledge, thereby achieving efficient integration of pre-trained knowledge and new task knowledge. In order to improve the flexibility and stability of the multimodal large model during parameter fine-tuning, this embodiment migrates the sparsity modeling and conflict mitigation mechanism described above to the multimodal large model connector part, and replaces the hard pruning scheme with soft sparsity regularization to achieve more refined parameter optimization, thereby obtaining a conflict mitigation regularization term. In some embodiments, the step of generating a conflict mitigation regularization term based on the knowledge retention mask and the sparse low-rank matrix includes:

[0185] By calculating the product of different sparsified low-rank matrices, a low-rank incremental matrix is ​​obtained;

[0186] Multiplying the knowledge-preserving mask by the low-rank increment matrix element-by-element to obtain a mask increment matrix;

[0187] Compute the Frobenius norm of the masked delta matrix to obtain the conflict mitigation regularization term.

[0188] Specifically, this embodiment adjusts the hard pruning scheme in the connector to a regularization method based on soft sparsity. By constraining the sparsity of the parameters using the L1 norm, the conflict mitigation regularization term of the connector is defined as:

[0189]

[0190] Where, A conflict mitigation regularization loss for the connector module, which is used to protect the pre-trained multimodal alignment knowledge in the connector and prevent the destruction of cross-modal association capabilities during fine-tuning; The retention mask matrix of the connector module identifies the common cross-modal knowledge to be protected in the connector; It is the product matrix of the LoRA low-rank matrix, indicating the update direction of the new task parameters; It is the L1 norm, that is, the sum of the absolute values ​​of the matrix elements. It is used to promote the transformation of the parameters of the connector module to a sparse distribution, so that the weights of unimportant parameters tend to zero, thereby reducing parameter redundancy and alleviating the phenomenon of knowledge forgetting.

[0191] Compared to hard pruning, soft sparsity regularization allows parameters to exist at a smaller amplitude, avoiding the performance loss caused by complete pruning. Therefore, this embodiment adds the conflict mitigation regularization term of the connector and the conflict mitigation regularization term of the multimodal large model body to the total loss of parameter fine-tuning, resulting in the joint loss function of the power task:

[0192]

[0193] Where, is the joint loss function for power tasks; Classify mission losses for power failures; is the conflict regularization term for the large language model; Regularization loss for conflict mitigation of connector modules; is the knowledge retention strength coefficient of the large language model, which represents the hyperparameter that controls the degree of general knowledge retention in the large language model; is the connector knowledge retention strength coefficient, which represents a hyperparameter that controls the degree of general knowledge retention in the connector module.

[0194] This example briefly proves the mathematical theorem that the expected sparsity and its upper bound still hold when the conflict mitigation regularization mechanism LoRA is proposed. and are two low-rank matrices in LoRA, where the matrix Each row of the matrix Each column of has uniform sparsity within it, but the sparsity between rows and columns varies. and matrix The average sparsity of and , then the product matrix The expected value of the proportion of non-zero elements in The following inequality is satisfied:

[0195]

[0196] matrix Any element in satisfy:

[0197]

[0198] Through analysis and Can get The probability of non-zero, for a fixed and ,make Representation matrix No. The sparsity of the row, Representation matrix No. The sparsity of the column, for and have:

[0199]

[0200] because No. Line and No. The positions of non-zero elements in the column are independent and uniformly distributed, for each , and Whether is non-zero or not is an independent event, so, and The probability of both being non-zero is:

[0201]

[0202] To prove another stronger conclusion, refer to the sparsity expectation theorem and assume that and All elements in are non-negative, if , then for all , there must be or Established, therefore, The probability is:

[0203]

[0204] therefore, The probability of being non-zero is:

[0205]

[0206] Therefore, the matrix The expectation of the proportion of non-zero elements in is:

[0207]

[0208] for ,function is a convex function. According to Jensen's inequality, for a convex function and a random variable have:

[0209]

[0210] In this embodiment, let the random variable obey ,have:

[0211]

[0212] Based on Jensen's inequality, we have:

[0213]

[0214] That is:

[0215]

[0216] According to the above inequality, we can get:

[0217]

[0218] Through the above-mentioned adjusted steps, this embodiment can migrate the previous sparsity modeling and conflict mitigation mechanism to the model connector part, and achieve more refined parameter optimization in the connector part of the multimodal large model, so that it can better adapt to the multimodal power information processing tasks without forgetting the pre-training knowledge, thereby improving the application performance and generalization ability of the model in the power grid system.

[0219] S5. According to the structured multimodal dataset, the multimodal large language model is fine-tuned and trained using the power task joint loss function to obtain a fine-tuned multimodal large language model.

[0220] In some embodiments, the step of fine-tuning the multimodal large language model using the power task joint loss function based on the structured multimodal dataset to obtain the fine-tuned multimodal large language model includes:

[0221] Comparing the current number of training steps with a preset warm-up step threshold, and entering the knowledge warm-up phase when the current number of training steps is less than the preset warm-up step threshold;

[0222] In the knowledge preheating stage, the total loss value of the power task is calculated using the power task joint loss function according to the structured multimodal data set;

[0223] Back-propagating the total power task loss value to update the sparse low-rank matrix, generating a preliminary optimized low-rank matrix at the end of the knowledge warm-up phase;

[0224] Generate a sparse mask matrix according to the preliminary optimized low-rank matrix and the preset fault feature sparsity parameter, and enter the knowledge integration stage when the current training step number is not less than the preset warm-up step number threshold;

[0225] In the knowledge integration stage, the preliminary optimized low-rank matrix is ​​multiplied element-by-element by the sparsified mask matrix to obtain a key sparsified low-rank matrix;

[0226] Forward propagating the structured multimodal dataset, and backpropagating the power task joint loss function to update the key sparse low-rank matrix to obtain a fine-tuned sparse low-rank matrix;

[0227] Fine-tune the multimodal large language model according to the fine-tuned sparse low-rank matrix to obtain a fine-tuned multimodal large language model.

[0228] To efficiently fine-tune the multimodal large language model in the power scenario, this embodiment sets a cyclic training process based on knowledge preheating and knowledge integration. Specifically, this embodiment initializes data such as the number of large language model layers, the number of connector layers, and the total number of training steps in the multimodal large language model. Circular training is performed for the total number of training steps for the parameters that need to be fine-tuned in the multimodal large language model. Each training step performs the following steps:

[0229] A batch of data is sampled from a structured multimodal dataset to form a training dataset. The current number of training steps is compared with a preset preheating step threshold. When the current number of training steps is less than the preset preheating step threshold, the knowledge preheating phase is entered. In the knowledge preheating phase, this embodiment processes visual data through a visual encoder and a connector to obtain visual features. At the same time, word segmentation is performed on text input to obtain text features. The visual features and text features are concatenated and input into a multimodal large language model. The cross-entropy loss function is used to calculate the power fault classification task loss:

[0230]

[0231]

[0232]

[0233] Where, is a visual feature vector that represents the deep feature expression of the power grid equipment image or video (such as the hot spot area feature in the infrared thermal image); is the feature conversion function of the connector module, which represents the cross-modal fusion component of aligned multimodal data (mapping visual features into semantic space); is a function composition operator, which represents the cascade operation of the visual feature extractor and the connector; is the output of the visual encoder, i.e., the features of the power grid equipment extracted from the original image or video; is the original visual input; is the text feature vector, i.e., the semantic encoding of the power text data; For text segmentation, convert power-related text (e.g., "A-phase grounding current suddenly increases") into word sequences; Enter the original text; is the power fault classification task loss, which is used to quantify the prediction error of the model in the fault classification task; is the cross entropy loss, which is used to calculate the classification error of the multimodal fusion result; is the cross entropy loss function; y is the true label; is a multimodal fusion function, which is used to fuse the feature vectors of different modalities; It is the joint feature vector obtained by fusing the visual feature vector and the text feature vector.

[0234] This embodiment traverses each layer of the large language model of the multimodal large language model and calculates the conflict mitigation regularization term:

[0235]

[0236] Where, For the The knowledge preserving mask matrix of the layer; For the The low-rank matrix B of the layer, whose update direction and amplitude are constrained by the conflict mitigation regularization term during fine-tuning; For the The low-rank matrix A of the layer, whose update direction and amplitude are constrained by the conflict mitigation regularization term during fine-tuning; Updated conflict regularization term for large language models.

[0237] If fine-tuning of the connector is required, iterate over each layer of the connector and calculate the conflict mitigation regularization term:

[0238]

[0239] Where, For the The retention mask matrix of the layer connector module; is the Hadamard product, that is, the element-by-element product; is the pruned sparse matrix B, which only retains the feature weights that are strongly related to power grid anomalies (such as local mutation response parameters); is the pruned sparse matrix A, which only retains the projection coefficients of physical quantities that are critical to fault diagnosis; Updated conflict mitigation regularization loss for connector modules.

[0240] Therefore, this embodiment uses the power task joint loss function to calculate the total loss value of the power task, back-propagates the total loss value of the power task and updates the low-rank matrix B and the low-rank matrix A to obtain the corresponding preliminary optimized low-rank matrix. When the current training step number is equal to the preset preheating step number threshold, a sparse mask matrix is ​​generated according to the preliminary optimized low-rank matrix and the preset fault feature sparsity parameter, and when the current training step number is not less than the preset preheating step number threshold, the knowledge integration stage is entered. In the knowledge integration stage, the preliminary optimized low-rank matrix is ​​combined with the sparse mask matrix. Perform element-by-element multiplication to obtain a key sparse low-rank matrix, forward propagate the structured multimodal dataset, and use the power task joint loss function to backpropagate and update the key sparse low-rank matrix to obtain a fine-tuned sparse low-rank matrix, multiply the fine-tuned sparse low-rank matrices to obtain a fine-tuning incremental matrix, and linearly superimpose the fine-tuning incremental matrix with the pre-trained weight matrix of the corresponding part to obtain the pre-training update weight value, and use the pre-training update weight value to fine-tune the multimodal large language model to obtain a fine-tuned multimodal large language model.

[0241] In response to the parameter redundancy problem existing in the LoRA fine-tuning process (including insufficient parameter sparsity and the conflict between pre-trained knowledge and task knowledge fusion), this embodiment effectively eliminates redundant parameters and identifies key sparse subsets by introducing a one-time pruning mechanism and a knowledge-guided optimization strategy in the parameter fine-tuning stage, thereby alleviating the potential conflict between power task requirements and pre-trained general knowledge. Specifically, this embodiment divides the overall training process into a knowledge warm-up stage and a knowledge integration stage, and achieves the dual goals of parameter sparsity and knowledge fusion through two-stage collaborative optimization. In the early stage of training, this embodiment dynamically identifies and retains sparse parameter subsets through a one-time pruning mechanism. This mechanism screens out parameters that are critical to power fault tasks and have little impact on general knowledge based on parameter amplitude or gradient importance evaluation, forming a sparse parameter skeleton. In subsequent training, the subset parameters remain fixed, and only the remaining parameters are fine-tuned, thereby reducing computational complexity while ensuring the stability of core parameters. In the knowledge warm-up stage, the conflict mitigation regularization mechanism guides the parameter update direction of LoRA away from the key knowledge areas in the pre-trained model (such as the general physical law encoding parameters) through the regularization term, avoiding excessive modification of the pre-trained knowledge. Combining task loss and regularization constraints, the update amplitude of non-sparse parameters is dynamically adjusted to assist in further optimization and screening of sparse subsets. Thus, in the knowledge warm-up stage, by regulating the parameter update direction, a preliminary mapping relationship between sparse parameters and power task characteristics is established, while avoiding direct conflicts between pre-trained knowledge and power grid fault task knowledge. At the same time, in the knowledge integration stage, the conflict mitigation regularization mechanism continues to guide the optimization process based on pre-trained knowledge, allowing non-sparse parameters to be adaptively adjusted on the basis of pre-trained knowledge, gradually integrating power fault tasks into the model, and fine-tuning the weights of the sparse parameter subsets through back propagation of task loss, so that it retains the advantages of the pre-trained model while enhancing its adaptability to power fault detection.

[0242] In summary, this embodiment gradually guides the model to adapt to tasks in the power scenario by dividing the knowledge warm-up stage and the knowledge integration stage, thereby improving the fine-tuning effect. At the same time, through conflict mitigation regularization terms and sparse matrix mask processing, the interference of new task parameter updates on important parameters of the pre-trained model is reduced, while maintaining the sparsity of parameters and improving model efficiency. It should be noted that the fine-tuning of the connector part and the large language model part of the multimodal large language model can be independently controlled, which is convenient for adjusting the fine-tuning strategy according to task requirements.

[0243] S6. Input the real-time operation data of the power grid into the fine-tuned multimodal large language model for inference to obtain a power grid fault diagnosis result, and isolate and control the power grid fault section based on the power grid fault diagnosis result.

[0244] In this embodiment, the real-time operation data of the power grid is input into the fine-tuned multimodal large language model, and the sparse low-rank matrix and knowledge-preserving mask structures within the fine-tuned multimodal large language model are used to identify and judge the fault characteristics of the power grid, and finally locate the fault section of the distribution network (such as a single-phase grounding fault). Based on the output results of the distribution network fault section, the relay protection device is linked to perform fault isolation control, restore the safe operation of the power grid, and achieve rapid diagnosis and accurate isolation of power grid faults, significantly improving the safety and reliability of power grid operation. It should be noted that in this embodiment, the fine-tuned multimodal large language model can also be used for power grid tasks such as load forecasting and anomaly detection. When the power grid task is load forecasting, the fine-tuned multimodal large language model can be used to generate regional power consumption time series forecast data, and load scheduling can be performed based on the generated regional power consumption time series forecast data; when the power grid task is anomaly detection, the fine-tuned multimodal large language model can be used to identify current distortion caused by equipment aging.

[0245] An embodiment of the present invention provides a fine-tuning method based on a multimodal model of power faults. The method obtains a structured multimodal dataset by collecting multi-source heterogeneous data generated during the operation of a power grid; selects a multimodal large language model, and generates an amplitude-guided knowledge-retention mask based on the pre-trained weight matrix of the multimodal large language model; performs power feature-guided sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal dataset to generate a sparse low-rank matrix; generates a conflict mitigation regularization term based on the knowledge-retention mask and the sparse low-rank matrix, and constructs a power task joint loss function based on the conflict mitigation regularization term; fine-tunes the multimodal large language model based on the structured multimodal dataset using the power task joint loss function to obtain a fine-tuned multimodal large language model; inputs real-time power grid operation data into the fine-tuned multimodal large language model for inference to obtain a power grid fault diagnosis result, and isolates and controls the power grid fault section based on the power grid fault diagnosis result. Compared with the existing technology, this method uses multi-source heterogeneous data of the power grid, and utilizes the spatiotemporal sparsity modeling and conflict mitigation regularization strategy to efficiently fine-tune the multimodal large language model, thereby achieving rapid and accurate diagnosis of power grid faults and automatic isolation and control of fault sections, improving the intelligent, safe operation and stability of the power grid system.

[0246] It should be noted that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0247] In one embodiment, Figure 3 As shown, an embodiment of the present invention provides a fine-tuning system based on a multi-modal model of power faults, the system comprising:

[0248] The data acquisition module 101 is used to collect multi-source heterogeneous data generated during the operation of the power grid to obtain a structured multimodal data set;

[0249] a mask generation module 102 for selecting a large multimodal language model and generating an amplitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the large multimodal language model;

[0250] A low-rank sparse module 103 is configured to perform power feature-oriented sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal dataset and generate a sparse low-rank matrix;

[0251] a loss construction module 104 for generating a conflict mitigation regularization term based on the knowledge retention mask and the sparsified low-rank matrix, and constructing a power task joint loss function based on the conflict mitigation regularization term;

[0252] A model building module 105 is configured to fine-tune the multimodal large language model using the power task joint loss function according to the structured multimodal dataset to obtain a fine-tuned multimodal large language model;

[0253] The fault isolation module 106 is used to input the real-time operation data of the power grid into the fine-tuned multimodal large language model for inference, obtain the power grid fault diagnosis result, and isolate and control the power grid fault section according to the power grid fault diagnosis result.

[0254] In this embodiment, the step of generating an amplitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the multimodal large language model includes:

[0255] Calculating a ratio of an absolute value of the pre-trained weight matrix to a Euclidean norm of the pre-trained weight matrix to obtain a normalized weight;

[0256] Calculating the reciprocal absolute value of the logarithm of the normalized weight with the natural logarithm as the base to obtain an importance score;

[0257] A preset steepness control parameter is introduced, and the importance score is smoothed by a hyperbolic tangent function to generate a knowledge-preserving mask.

[0258] In this embodiment, the power feature-oriented sparsity modeling is performed based on the spatiotemporal sparsity characteristics of the structured multimodal dataset to generate a sparse low-rank matrix, including:

[0259] quantifying the spatiotemporal sparsity of the power grid according to the spatial uneven distribution characteristics and temporal fluctuation characteristics of the structured multimodal dataset to obtain the spatiotemporal sparsity of the structured multimodal dataset;

[0260] Performing low-rank decomposition on the model parameters of the multimodal large language model to obtain two initialized low-rank matrices, and performing knowledge warm-up training on the initialized low-rank matrices to obtain the original low-rank matrix;

[0261] Performing feature extraction on the structured multimodal data set to obtain power grid abnormality features;

[0262] Using the absolute value of the elements of the original low-rank matrix as a parameter importance measurement indicator, generating a power grid anomaly feature mask matrix of the same size as the original low-rank matrix according to the spatiotemporal sparsity and the power grid anomaly feature;

[0263] The original low-rank matrix is ​​pruned to obtain a pruned low-rank matrix, and the power grid anomaly feature mask matrix is ​​element-wise multiplied by the pruned low-rank matrix to obtain a sparse low-rank matrix.

[0264] In this embodiment, generating a conflict mitigation regularization term according to the knowledge-preserving mask and the sparsified low-rank matrix includes:

[0265] By calculating the product of different sparsified low-rank matrices, a low-rank incremental matrix is ​​obtained;

[0266] Multiplying the knowledge-preserving mask by the low-rank increment matrix element-by-element to obtain a mask increment matrix;

[0267] Compute the Frobenius norm of the masked delta matrix to obtain the conflict mitigation regularization term.

[0268] In this embodiment, the power task joint loss function includes a power fault classification task loss and a conflict mitigation regularization term loss.

[0269] In this embodiment, constructing a power task joint loss function based on the conflict mitigation regularization term includes:

[0270] Performing multimodal encoding on the structured multimodal dataset to obtain a multimodal feature vector;

[0271] Inputting the multimodal feature vector into the multimodal large language model to perform fault classification prediction to obtain a fault classification prediction result;

[0272] Calculating the cross entropy loss between the fault classification prediction result and the true label to obtain the power fault classification task loss;

[0273] Performing soft sparsity regularization on the knowledge-preserving mask and the sparsified low-rank matrix to generate a conflict mitigation regularization term;

[0274] According to the power fault classification task loss and the conflict mitigation regularization term, a power task joint loss function is constructed using weighted summation.

[0275] In this embodiment, fine-tuning the multimodal large language model based on the structured multimodal dataset and using the power task joint loss function to obtain a fine-tuned multimodal large language model includes:

[0276] Comparing the current number of training steps with a preset warm-up step threshold, and entering the knowledge warm-up phase when the current number of training steps is less than the preset warm-up step threshold;

[0277] In the knowledge preheating stage, the total loss value of the power task is calculated using the power task joint loss function according to the structured multimodal data set;

[0278] Back-propagating the total power task loss value to update the sparse low-rank matrix, generating a preliminary optimized low-rank matrix at the end of the knowledge warm-up phase;

[0279] Generate a sparse mask matrix according to the preliminary optimized low-rank matrix and the preset fault feature sparsity parameter, and enter the knowledge integration stage when the current training step number is not less than the preset warm-up step number threshold;

[0280] In the knowledge integration stage, the preliminary optimized low-rank matrix is ​​multiplied element-by-element by the sparsified mask matrix to obtain a key sparsified low-rank matrix;

[0281] Forward propagating the structured multimodal dataset, and backpropagating the power task joint loss function to update the key sparse low-rank matrix to obtain a fine-tuned sparse low-rank matrix;

[0282] Fine-tune the multimodal large language model according to the fine-tuned sparse low-rank matrix to obtain a fine-tuned multimodal large language model.

[0283] In this embodiment, fine-tuning the multimodal large language model according to the fine-tuned sparse low-rank matrix to obtain a fine-tuned multimodal large language model includes:

[0284] Multiplying the fine-tuned sparsified low-rank matrix to obtain a fine-tuning increment matrix, and linearly superimposing the fine-tuning increment matrix with the pre-training weight matrix to obtain a pre-training update weight value;

[0285] The multimodal large language model is fine-tuned using the pre-trained updated weight values ​​to obtain a fine-tuned multimodal large language model.

[0286] For the specific definition of a fine-tuning system based on a multi-modal model of power failure, please refer to the above-mentioned definition of a fine-tuning method based on a multi-modal model of power failure, which will not be repeated here. A person of ordinary skill in the art will appreciate that the various modules and steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware, software, or a combination of both. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0287] An embodiment of the present invention provides a fine-tuning system based on a multimodal model of power faults. The system collects multi-source heterogeneous data generated during the operation of a power grid through a data acquisition module to obtain a structured multimodal data set; a mask generation module selects a multimodal large language model and generates an amplitude-guided knowledge-preserving mask based on the pre-trained weight matrix of the multimodal large language model; a low-rank sparse module performs power feature-guided sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal data set to generate a sparse low-rank matrix; a loss construction module generates a conflict mitigation regularization term based on the knowledge-preserving mask and the sparse low-rank matrix, and constructs a power task joint loss function based on the conflict mitigation regularization term; a model construction module fine-tunes the multimodal large language model based on the structured multimodal data set using the power task joint loss function to obtain a fine-tuned multimodal large language model; a fault isolation module inputs real-time power grid operation data into the fine-tuned multimodal large language model for inference to obtain a power grid fault diagnosis result, and isolates and controls the power grid fault section based on the power grid fault diagnosis result. Compared with existing technologies, this system uses multi-source heterogeneous data from the power grid, spatiotemporal sparsity modeling of power grid data, and conflict mitigation regularization strategies to efficiently fine-tune the multimodal large language model, thereby achieving rapid and accurate diagnosis of power grid faults and automatic isolation and control of fault sections, improving the intelligent, safe operation and stability of the power grid system.

[0288] In one embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above method are implemented.

[0289] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD).

[0290] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0291] The above-described embodiments merely represent several preferred implementations of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art could make several improvements and substitutions without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be based on the scope of protection of the claims.

Claims

1. A fine-tuning method based on a multimodal model of power faults, characterized in that: include: Collect multi-source heterogeneous data generated during power grid operation to obtain structured multimodal data sets; Selecting a large multimodal language model, and generating an amplitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the large multimodal language model; Performing power feature-oriented sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal dataset to generate a sparse low-rank matrix; generating a conflict mitigation regularization term based on the knowledge-preserving mask and the sparsified low-rank matrix, and constructing a power task joint loss function based on the conflict mitigation regularization term; According to the structured multimodal dataset, fine-tuning the multimodal large language model using the power task joint loss function to obtain a fine-tuned multimodal large language model; The real-time operation data of the power grid is input into the fine-tuned multimodal large language model for reasoning to obtain a power grid fault diagnosis result, and the power grid fault section is isolated and controlled according to the power grid fault diagnosis result.

2. A fine-tuning method based on a multimodal model of power failure according to claim 1, characterized in that: Generating an amplitude-guided knowledge-preserving mask according to a pre-trained weight matrix of the multimodal large language model includes: Calculating a ratio of an absolute value of the pre-trained weight matrix to a Euclidean norm of the pre-trained weight matrix to obtain a normalized weight; Calculating the reciprocal absolute value of the logarithm of the normalized weight with the natural logarithm as the base to obtain an importance score; A preset steepness control parameter is introduced, and the importance score is smoothed by a hyperbolic tangent function to generate a knowledge-preserving mask.

3. The fine-tuning method based on the power fault multimodal model according to claim 1, characterized in that: The power feature-oriented sparsity modeling is performed according to the spatiotemporal sparse characteristics of the structured multimodal data set to generate a sparse low-rank matrix, including: quantifying the spatiotemporal sparsity of the power grid according to the spatial uneven distribution characteristics and temporal fluctuation characteristics of the structured multimodal dataset to obtain the spatiotemporal sparsity of the structured multimodal dataset; Performing low-rank decomposition on the model parameters of the multimodal large language model to obtain two initialized low-rank matrices, and performing knowledge warm-up training on the initialized low-rank matrices to obtain the original low-rank matrix; Performing feature extraction on the structured multimodal data set to obtain power grid abnormality features; Using the absolute value of the elements of the original low-rank matrix as a parameter importance measurement indicator, generating a power grid anomaly feature mask matrix of the same size as the original low-rank matrix according to the spatiotemporal sparsity and the power grid anomaly feature; The original low-rank matrix is ​​pruned to obtain a pruned low-rank matrix, and the power grid anomaly feature mask matrix is ​​element-wise multiplied by the pruned low-rank matrix to obtain a sparse low-rank matrix.

4. The fine-tuning method based on the power fault multimodal model according to claim 1, characterized in that: Generating a conflict mitigation regularization term according to the knowledge preservation mask and the sparsified low-rank matrix comprises: By calculating the product of different sparsified low-rank matrices, a low-rank incremental matrix is ​​obtained; Multiplying the knowledge-preserving mask by the low-rank increment matrix element-by-element to obtain a mask increment matrix; Compute the Frobenius norm of the masked delta matrix to obtain the conflict mitigation regularization term.

5. The fine-tuning method based on a multimodal model of power failure according to claim 1, characterized in that: The power task joint loss function includes the power fault classification task loss and the conflict mitigation regularization term loss.

6. The fine-tuning method based on the power fault multimodal model according to claim 1, characterized in that: The power task joint loss function is constructed based on the conflict mitigation regularization term, including: Performing multimodal encoding on the structured multimodal dataset to obtain a multimodal feature vector; Inputting the multimodal feature vector into the multimodal large language model to perform fault classification prediction to obtain a fault classification prediction result; Calculating the cross entropy loss between the fault classification prediction result and the true label to obtain the power fault classification task loss; Performing soft sparsity regularization on the knowledge-preserving mask and the sparsified low-rank matrix to generate a conflict mitigation regularization term; According to the power fault classification task loss and the conflict mitigation regularization term, a power task joint loss function is constructed using weighted summation.

7. The fine-tuning method based on the power fault multimodal model according to claim 1, characterized in that: The method of fine-tuning the multimodal large language model based on the structured multimodal dataset and using the power task joint loss function to obtain a fine-tuned multimodal large language model includes: Comparing the current number of training steps with a preset warm-up step threshold, and entering the knowledge warm-up phase when the current number of training steps is less than the preset warm-up step threshold; In the knowledge preheating stage, the total loss value of the power task is calculated using the power task joint loss function according to the structured multimodal data set; Back-propagating the total power task loss value to update the sparse low-rank matrix, generating a preliminary optimized low-rank matrix at the end of the knowledge warm-up phase; Generate a sparse mask matrix according to the preliminary optimized low-rank matrix and the preset fault feature sparsity parameter, and enter the knowledge integration stage when the current training step number is not less than the preset warm-up step number threshold; In the knowledge integration stage, the preliminary optimized low-rank matrix is ​​multiplied element-by-element by the sparsified mask matrix to obtain a key sparsified low-rank matrix; Forward propagating the structured multimodal dataset, and backpropagating the power task joint loss function to update the key sparse low-rank matrix to obtain a fine-tuned sparse low-rank matrix; Fine-tune the multimodal large language model according to the fine-tuned sparse low-rank matrix to obtain a fine-tuned multimodal large language model.

8. The fine-tuning method based on the power fault multimodal model according to claim 7, characterized in that: Fine-tuning the multimodal large language model according to the fine-tuned sparse low-rank matrix to obtain a fine-tuned multimodal large language model includes: Multiplying the fine-tuned sparsified low-rank matrix to obtain a fine-tuning increment matrix, and linearly superimposing the fine-tuning increment matrix with the pre-training weight matrix to obtain a pre-training update weight value; The multimodal large language model is fine-tuned using the pre-trained updated weight values ​​to obtain a fine-tuned multimodal large language model.

9. A fine-tuning system based on a multimodal model of power failure, characterized in that: The system comprises: The data acquisition module is used to collect multi-source heterogeneous data generated during the operation of the power grid and obtain a structured multimodal data set; a mask generation module, configured to select a large multimodal language model and generate an amplitude-guided knowledge-preserving mask based on a pre-trained weight matrix of the large multimodal language model; A low-rank sparse module, configured to perform power feature-oriented sparsity modeling based on the spatiotemporal sparse characteristics of the structured multimodal dataset and generate a sparse low-rank matrix; a loss construction module, configured to generate a conflict mitigation regularization term based on the knowledge preservation mask and the sparsified low-rank matrix, and construct a power task joint loss function based on the conflict mitigation regularization term; A model building module, configured to fine-tune the multimodal large language model using the power task joint loss function according to the structured multimodal dataset to obtain a fine-tuned multimodal large language model; The fault isolation module is used to input the real-time operation data of the power grid into the fine-tuned multimodal large language model for reasoning, obtain the power grid fault diagnosis result, and isolate and control the power grid fault section according to the power grid fault diagnosis result.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 8 is implemented.

Images