Training method, device and equipment of multi-modal model of power switch equipment, medium and product

Abstract

The invention relates to a training method and device for a multi-modal model of power switch equipment, equipment, a medium and a product. The method comprises the steps of collecting multi-source heterogeneous data from an actual operation and maintenance scene of a power system, and constructing a multi-modal pre-training data set based on the multi-source heterogeneous data; taking the deep learning model as a language model base, and integrating a plurality of modal encoders to obtain an initial multi-modal large model; training the initial multi-modal large model through a multi-modal pre-training data set by adopting a low-rank adaptive method to obtain a target multi-modal large model; and inputting operation data of the power switch equipment into the target multi-mode large model, and outputting an analysis report or an operation and maintenance suggestion meeting requirements. Therefore, effective cross-modal information association and fusion are performed on various source data related to the power switch equipment, more comprehensive and more accurate cognition of the allowable state of the power switch equipment is facilitated, and the depth and breadth of intelligent operation and maintenance are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart grid, in particular to a power switch device multi-modal model training method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] The power system is the infrastructure for the operation of modern society, among which the power switch device as a key component, bears the core function of controlling and protecting the safe and stable operation of the power grid. With the continuous expansion of the power grid scale, the number of power switch devices has increased significantly, which also puts forward higher requirements on the efficiency and quality of operation and maintenance.

[0003] In the traditional technology, the operation and maintenance mode mainly relies on artificial regular patrol and test, which not only consumes a lot of manpower and material resources, but also has low efficiency, and the inspection results are easily affected by subjective factors, which is difficult to guarantee the consistency and accuracy of evaluation. In view of the problems existing in the traditional technology, the intelligent technology based on multi-modal large language model (MLLM) is introduced into the field of power grid.

[0004] However, in the field of intelligent operation and maintenance of power switch, there is still a lack of large models specially designed and trained. SUMMARY

[0005] Therefore, it is necessary to provide a power switch device multi-modal model training method, device, computer equipment, computer readable storage medium and computer program product capable of deeply understanding the professional knowledge in the field of power switch and realizing intelligent decision of operation and maintenance, in view of the above technical problems.

[0006] In a first aspect, the present application provides a power switch device multi-modal model training method, which comprises:

[0007] Collecting multi-source heterogeneous data from the actual operation and maintenance scene of the power system, and constructing a multi-modal pre-training data set based on the multi-source heterogeneous data;

[0008] Taking a deep learning model as a language model base, and integrating a plurality of modal encoders to obtain an initial multi-modal large model, wherein the multi-modal encoder comprises an image encoder, an audio encoder and a time series encoder;

[0009] Training the initial multi-modal large model through the multi-modal pre-training data set by using a low-rank adaptive method to obtain a target multi-modal large model;

[0010] Inputting the operation data of the power switch device into the target multi-modal large model, and outputting a required analysis report or operation and maintenance suggestion.

[0011] In one of the embodiments, the multi-source heterogeneous data includes at least one of visual information, auditory information, time-series sensor data, and text information;

[0012] The visual information includes at least one of monitoring images, unmanned aerial vehicle inspection photos, and infrared thermal images;

[0013] The auditory information includes device operation sound and / or abnormal discharge recording;

[0014] The time-series sensor data includes at least one of current, voltage, partial discharge signal, vibration, temperature, and humidity;

[0015] The text information includes at least one of device account, operation and maintenance record, defect report, and technical procedure.

[0016] In one of the embodiments, the constructing a multi-modal pre-training data set based on the multi-source heterogeneous data includes:

[0017] Cleaning the multi-source heterogeneous data, removing damaged or irrelevant items, and anonymizing or desensitizing sensitive information related to privacy to obtain cleaned data of various types;

[0018] After size normalization, cropping, and brightness and / or contrast adjustment of the cleaned image data, standardization processing is performed to obtain standard image data;

[0019] After pre-emphasis, framing, and windowing operations on the cleaned audio data, amplitude-related features are extracted, and normalization processing is performed to obtain normalized audio amplitude data;

[0020] The cleaned time-series data is denoised, missing value filled, resampled, and normalized or standardized to obtain processed time-series data;

[0021] After tokenization and stop word removal processing of the cleaned text data, the text data is converted to a preset digital representation data;

[0022] The standard image data, the audio amplitude data, the processed time-series data, and the digital representation data are labeled and integrated into structured data, which is used to construct a multi-modal pre-training data set.

[0023] In one of the embodiments, the image encoder is configured to convert input image data into an image feature sequence;

[0024] The audio encoder is configured to extract audio features from input audio data;

[0025] the time sequence encoder, configured to map input time sequence data into time sequence features;

[0026] the embedding layer in the initial multi-modal large model, configured to extract text features from input text data;

[0027] The initial multi-modal large model is also used to perform multi-modal feature fusion processing on at least two of the image feature sequence, the audio feature, the time sequence feature, and the text feature, model context dependence of the fused features, and semantic understanding to obtain a hidden state representation, and output a required analysis report or operation and maintenance suggestion after the hidden state representation is processed by an output layer and decoding.

[0028] In one of the embodiments, the initial multi-modal large model is trained by using a low-rank adaptive method and the multi-modal pre-training data set to obtain a target multi-modal large model, including:

[0029] The A matrix and the B matrix are injected into selected layers of the initial multi-modal large model; wherein the A matrix and the B matrix are obtained by decomposing a low-rank update amount added by a specified weight matrix;

[0030] The pre-training task facing the operation and maintenance scene is set based on the multi-modal pre-training data set, and the pre-training task includes at least one of multi-modal question answering, state prediction, and report generation;

[0031] The initial multi-modal large model is trained by the pre-training task, and at least one part of parameters of the A matrix and the B matrix, and the multi-modal encoder or the output layer are adjusted according to the training result to obtain a target multi-modal large model.

[0032] In one of the embodiments, during the training process of the initial multi-modal large model, only the A matrix and the B matrix, and at least one part of parameters of the multi-modal encoder or the output layer are optimized, and the parameters of the original weight matrix of the initial multi-modal large model are frozen;

[0033] After the training of the initial multi-modal large model is completed, the updated A matrix and the updated B matrix are merged with the original weight matrix, or the updated A matrix and the updated B matrix are kept separate.

[0034] In a second aspect, the application further provides a training device of a multi-modal model of a power switch device, and the device includes:

[0035] The data set construction module is configured to collect multi-source heterogeneous data from actual operation and maintenance scenes of a power system, and construct a multi-modal pre-training data set based on the multi-source heterogeneous data.

[0036] a model construction module, configured to take a deep learning model as a language model base, and integrate multiple modal encoders to obtain an initial multi-modal large model, wherein the multiple modal encoders include an image encoder, an audio encoder, and a time series encoder;

[0037] a training module, configured to train the initial multi-modal large model by using a low-rank adaptive method and the multi-modal pre-training data set to obtain a target multi-modal large model;

[0038] a processing module, configured to input operation data of a power switching device into the target multi-modal large model, and output a required analysis report or operation and maintenance suggestion.

[0039] In a third aspect, the present application further provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0040] collecting multi-source heterogeneous data from actual operation and maintenance scenarios of a power system, and constructing a multi-modal pre-training data set based on the multi-source heterogeneous data;

[0041] taking a deep learning model as a language model base, and integrating multiple modal encoders to obtain an initial multi-modal large model, wherein the multiple modal encoders include an image encoder, an audio encoder, and a time series encoder;

[0042] training the initial multi-modal large model by using a low-rank adaptive method and the multi-modal pre-training data set to obtain a target multi-modal large model;

[0043] inputting operation data of a power switching device into the target multi-modal large model, and outputting a required analysis report or operation and maintenance suggestion.

[0044] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0045] collecting multi-source heterogeneous data from actual operation and maintenance scenarios of a power system, and constructing a multi-modal pre-training data set based on the multi-source heterogeneous data;

[0046] taking a deep learning model as a language model base, and integrating multiple modal encoders to obtain an initial multi-modal large model, wherein the multiple modal encoders include an image encoder, an audio encoder, and a time series encoder;

[0047] The initial multi-modal large model is trained by using a low-rank adaptive method and the multi-modal pre-training data set, to obtain a target multi-modal large model.

[0048] The operation data of the power switching device is input into the target multi-modal large model, and a required analysis report or operation and maintenance suggestion is output.

[0049] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the following steps:

[0050] Multi-source heterogeneous data is collected from actual operation and maintenance scenarios of a power system, and a multi-modal pre-training data set is constructed based on the multi-source heterogeneous data;

[0051] A deep learning model is used as a language model base, and multiple modal encoders are integrated to obtain an initial multi-modal large model, wherein the multiple modal encoders include an image encoder, an audio encoder, and a time series encoder;

[0052] The initial multi-modal large model is trained by using a low-rank adaptive method and the multi-modal pre-training data set, to obtain a target multi-modal large model;

[0053] The operation data of the power switching device is input into the target multi-modal large model, and a required analysis report or operation and maintenance suggestion is output.

[0054] The training method, device, computer device, computer readable storage medium, and computer program product of the multi-modal model of the power switching device can collect multi-source heterogeneous data from actual operation and maintenance scenarios of a power system, and construct a multi-modal pre-training data set based on the multi-source heterogeneous data, so that the trained model can process multi-modal data, fully understand different types of information, and improve the accuracy of subsequent analysis. A deep learning model is used as a language model base, and multiple modal encoders are integrated to obtain an initial multi-modal large model, wherein the multiple modal encoders include an image encoder, an audio encoder, and a time series encoder, so that the multi-modal data can be more comprehensively understood and recognized. The initial multi-modal large model is trained by using a low-rank adaptive method and the multi-modal pre-training data set, to obtain a target multi-modal large model, so that the amount of parameters that need to be trained can be reduced, and the training efficiency of the model can be improved. The operation data of the power switching device is input into the target multi-modal large model, and a required analysis report or operation and maintenance suggestion is output. Thus, the problems of high training cost and insufficient professionalism of existing large models can be overcome, multi-source data related to the power switching device can be effectively associated and fused across modalities, a more comprehensive and accurate understanding of the allowable state of the power switching device can be formed, and the depth and breadth of intelligent operation and maintenance are improved. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0056] Figure 1 A schematic diagram of the principle of a training method of a multi-modal model of a power switch device provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of the flow of the training method of the multi-modal model of the power switch device in an embodiment;

[0058] Figure 3 A structural block diagram of the initial multi-modal large model in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of the principle of the efficient training of the multi-modal large model in the present embodiment;

[0060] Figure 5 A structural block diagram of the training device of the multi-modal model of the power switch device in an embodiment;

[0061] Figure 6 An internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0062] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0063] It should be noted that the terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the solutions or any combination of a plurality of solutions.

[0064] With the continuous expansion of the power grid, the number of power switching devices has increased significantly, putting higher demands on the efficiency and quality of operation and maintenance. Traditional operation and maintenance modes mainly rely on regular manual inspection and testing, which not only consumes a lot of manpower and resources, but also has low efficiency. Moreover, the inspection results are easily influenced by subjective factors, making it difficult to ensure consistency and accuracy of the evaluation. In addition, the data generated by manual inspection is mostly unstructured information, which is difficult to analyze systematically to support predictive maintenance. Furthermore, for sudden failures or rapidly deteriorating conditions, the response is often not timely enough, which may pose safety hazards and power supply risks.

[0065] In recent years, artificial intelligence technology, especially methods based on deep learning, has begun to be applied to power equipment state monitoring. For example, computer vision technology is used to analyze images and automatically identify the open / close state of disconnectors or abnormalities in the appearance of equipment. Machine learning algorithms are applied to process current and partial discharge signals collected by sensors to evaluate the health status of equipment or provide fault warnings. These technologies have achieved certain results in specific tasks and improved the automation level of operation and maintenance. However, existing intelligent algorithms are mostly designed for single tasks and have relatively independent functions. For example, a model for image analysis cannot handle time series signal data, and a model for signal processing cannot understand text information in operation and maintenance procedure documents. This limitation in capability makes it difficult for them to effectively integrate information from multiple sources such as vision, sensor signals, and text records, and cannot meet the needs of comprehensive judgment and deep analysis in complex operation and maintenance scenarios, limiting their widespread application and deep empowerment in intelligent operation and maintenance of power switching devices.

[0066] At the same time, new generation artificial intelligence large model technologies represented by ChatGPT have developed rapidly and shown strong natural language understanding, instruction following, knowledge integration, and logical reasoning capabilities, showing application potential in multiple professional fields. These large model technologies theoretically provide new possibilities for solving the multi-source information fusion and complex decision-making challenges faced in power switching operation and maintenance.

[0067] However, there is currently a lack of specialized domain large models designed and trained for the highly specialized field of power switching intelligent operation and maintenance. General-purpose large models lack professional knowledge in power system operation mechanisms, equipment characteristics, and operation and maintenance procedures, and direct application in this scenario has limited effectiveness and may have reliability issues. Therefore, there is an urgent need to develop a training method for a power switching device multi-modal model to effectively train large models using real power system operation and maintenance data containing multiple types of information such as images, signals, and text. This will enable them to deeply understand the professional knowledge in the power switching field and have the ability to comprehensively process multi-source heterogeneous information and perform complex analysis and reasoning tasks. Ultimately, it will achieve intelligent decision support for power switching device operation and maintenance, significantly improving operation and maintenance efficiency, accuracy, and safety.

[0068] Exemplary, Figure 1 A schematic diagram of the principle of a training method of a power switching device multi-modal model provided by an embodiment of the present application is shown in Figure 1 The method in the embodiment can be roughly divided into three stages. The first stage is to construct a multi-modal pre-training data set. The second stage is to design a multi-modal large model structure. The third stage is to study an efficient pre-training algorithm. In the first stage, the original data collection is mainly completed, and multi-scene and multi-modal data are obtained. In the second stage, the construction of a multi-modal encoder and a basic large model base is mainly completed, and a multi-modal power large model architecture is obtained on this basis, so that multi-scene and multi-modal power data processing can be realized. In the third stage, the parameters of the large model are trained and optimized through weight decomposition, low-rank self-adaptation and the like, so as to significantly reduce the computing resource requirement of training, and realize efficient and professional multi-modal unified pre-training.

[0069] In an exemplary embodiment, as shown in Figure 2 A training method of a power switching device multi-modal model is provided. The method in the embodiment can include the following steps S201 to S204. Wherein:

[0070] Step S201, collecting multi-source heterogeneous data from actual operation and maintenance scenes of a power system, and constructing a multi-modal pre-training data set based on the multi-source heterogeneous data.

[0071] In the embodiment, a high-quality and structured multi-modal pre-training data set needs to be constructed first to lay a solid data foundation for subsequent model training. For example, multi-source heterogeneous data can be widely collected from actual operation and maintenance scenes of a power system. The multi-source heterogeneous data includes at least one of visual information, auditory information, time series sensor data, and text information.

[0072] Optionally, the visual information includes at least one of monitoring images, unmanned aerial vehicle inspection photos, and infrared thermal images.

[0073] Optionally, the auditory information includes at least one of device operation sound and / or abnormal discharge recording.

[0074] Optionally, the time series sensor data includes at least one of current, voltage, partial discharge signal, vibration, temperature and humidity.

[0075] Optionally, the text information includes at least one of device account, operation and maintenance record, defect report and technical regulation.

[0076] In this embodiment, first, the multi-source heterogeneous data is cleaned and processed to remove damaged or irrelevant items, and sensitive information related to privacy is anonymized or desensitized (to ensure compliance with data use), obtaining cleaned data of various types; after size normalization, cropping, and brightness and / or contrast adjustment of the cleaned image data, standardization processing is performed to obtain standard image data; after pre-emphasis, framing, and windowing operations of the cleaned audio data, amplitude-related features are extracted, and normalization processing is performed to obtain normalized audio amplitude data; after denoising, missing value filling, resampling, and normalization or standardization processing of the cleaned time series data, processed time series data is obtained; after word segmentation and removal of stop words processing of the cleaned text data, the data is converted into preset digital representation data; the standard image data, audio amplitude data, processed time series data, and digital representation data are labeled and processed, and the labeled data is integrated into structured data, which is used to construct a multi-modal pre-training data set.

[0077] For example, for image data , first, size normalization, cropping, and brightness / contrast adjustment are performed, and standardization processing is applied to adjust the pixel value to a uniform scale. Z-score standardization is used, and the calculation formula is as follows:

[0078]

[0079] wherein, represents the normalized box value, and are the pixel mean and standard deviation, respectively.

[0080] For example, for audio data , first, pre-emphasis, framing, windowing, and other operations are performed, and MFCC (Mel-frequency cepstral coefficients, a feature widely used in automatic speech and speaker recognition) or direct processing of waveforms / spectrograms can be extracted, and amplitude normalization processing is performed, and the calculation formula is as follows:

[0081]

[0082] wherein, represents the normalized audio amplitude, represents the amplitude of the audio. It should be understood that specific filtering or time-frequency analysis methods can also be applied to process the audio data.

[0083] For example, for time series data , de-noising, missing value filling, resampling, and normalization / standardization are needed. Common methods include min-max scaling:

[0084]

[0085] Or, Z-score standardization:

[0086]

[0087] wherein, denotes the normalized time series data, denotes the standardized time series data.

[0088] For example, for text data , text preprocessing steps such as cleaning, tokenization, and removing stop words are needed, and it is converted into a digital representation that the model can recognize (such as Token IDs, which are used to distinguish different sentences or paragraphs).

[0089] It should be understood that after all data processing is completed, professional annotation is also needed. For example, according to professional knowledge, an expert in the power field annotates the data to be structured and semantically annotated, and the annotation content involves device component identification, state judgment, abnormal classification, key information extraction, event detection, and important cross-modal correlation annotation, to realize the correlation between the hot spots in the image and the specific temperature readings and related historical records. Finally, all the processed and annotated data are integrated into a structured multi-modal pre-training data set Each sample in the data set contains multiple modal data and its annotation or correlation information, and its structure can be represented as:

[0090]

[0091] wherein, denotes the annotated image data, denotes the annotated audio data, denotes the annotated time series data, denotes the annotated text data, denotes the comprehensive annotation information or pre-training target of the sample.

[0092] Step S202 uses a deep learning model as a language model base and integrates multiple modal encoders to obtain an initial multi-modal large model.

[0093] wherein, the multi-modal encoder includes: an image encoder, an audio encoder, and a time series encoder.

[0094] Exemplarily, the image encoder is configured to convert input image data into an image feature sequence; the audio encoder is configured to extract audio features from input audio data; the time series encoder is configured to map input time series data into time series features; the embedding layer in the initial multi-modal large model is configured to extract text features from input text data; and the initial multi-modal large model is further configured to perform multi-modal feature fusion on at least two of the image feature sequence, the audio features, the time series features, and the text features, to model context dependency and semantic understanding on the fused features to obtain a hidden state representation, and to output a required analysis report or operation and maintenance suggestion through an output layer and decoding processing.

[0095] Exemplarily, Figure 3 A structural block diagram of the initial multi-modal large model in the embodiments of the present application is shown in Figure 3 As shown, the initial multi-modal large model can be divided into a task layer, a model layer, and a data layer. The task layer is configured to determine various tasks (such as partial discharge detection, opening and closing recognition, and monitoring coil current, etc.). The model layer includes a visual encoder, a text encoder, a time series encoder, and a language model base. The data layer includes image data, text data, and time series data, etc.

[0096] Optionally, an advanced architecture based on Transformer is used as a backbone LLM, and a special modality encoder is provided for the backbone LLM to process various input information. The overall architecture is shown in Figure 1 To realize perception of different types of data, the model integrates multiple modality encoders.

[0097] Exemplarily, the image encoder ( ) adopts a Vision Transformer (ViT) structure to convert preprocessed images into an image feature sequence . The audio encoder ( ) selects a Wav2Vec2.0 or HuBERT model to extract audio features from processed audio . The time series encoder ( ) uses a TCN or time series Transformer structure to map processed time series data into time series features . Text information is processed by the embedding layer of the backbone LLM to obtain text features .

[0098] Optionally, after obtaining the feature representations of each modality, multi-modal feature fusion is needed to enable the base large model to comprehensively understand all the input information. A commonly used method is to concatenate the feature sequences of all modalities in the sequence dimension:

[0099]

[0100] wherein, represents the fused feature sequence.

[0101] Optionally, special tokens can also be inserted during concatenation to distinguish different modal sources, or more complex fusion strategies such as cross-attention mechanisms can be used to promote deep interaction between modalities.

[0102] Optionally, the fused feature sequence is fed into a base Transformer large model (which can be based on mature architectures such as GPT, LLaMA, or DeepSeek) for deep processing. The model uses its core multi-layer self-attention mechanism and feedforward network to model the context dependence and high-level semantic understanding of the input fused features, obtaining hidden state representations :

[0103] .

[0104] Finally, these hidden states are processed through an output layer (typically a linear layer) and undergo a decoding process (commonly beam search) to generate the final output :

[0105]

[0106] Step S203, using a low-rank adaptive method, training the initial multi-modal large model through a multi-modal pre-training data set to obtain a target multi-modal large model.

[0107] In this embodiment, considering the huge parameter quantity of the multi-modal large model, the calculation resources required for full training or fine-tuning are very large and costly. Therefore, the parameter efficient fine-tuning (PEFT) technology can be used, specifically the low-rank adaptive (LoRA) method is applied to realize the low-cost and high-efficiency adaptation of the model in the power switch operation and maintenance field. The core idea of LoRA is to freeze most of the original parameters of the pre-trained model during the fine-tuning stage, and only add a bypass, trainable low-rank update quantity for specific weight matrices (weight matrices or feedforward network weights in the Transformer layer) in the model This update quantity is decomposed into the product of two small matrices:

[0108]

[0109] Exemplarily, the trainable A matrix and B matrix can be injected into selected layers of an initial multi-modal large model; the A matrix and B matrix are obtained by decomposing a low-rank update amount added by a specified weight matrix; a pre-training task facing an operation and maintenance scenario is set based on a multi-modal pre-training data set, the pre-training task including at least one of multi-modal question answering, state prediction, and report generation; the initial multi-modal large model is trained through the pre-training task, and at least one of the A matrix and the B matrix and a part of parameters of a multi-modal encoder or an output layer is adjusted according to a training result, to obtain a target multi-modal large model.

[0110] As shown in Figure 4 , rank of the low-rank matrix is much smaller than the dimension of the original matrix. In actual application, for the layer to which LoRA is applied, the forward propagation calculation is adjusted from to

[0111]

[0112] Sometimes a scaling factor is introduced to adjust:

[0113]

[0114] In the training process, only the parameters of the low-rank matrix and are optimized, while the huge remains unchanged. This method significantly reduces the amount of parameters that need to be trained.

[0115] In this embodiment, the efficient pre-training (or fine-tuning) process is implemented as follows: first, a suitable general pre-trained multi-modal large model is loaded as a basis; then, according to a preset strategy, a LoRA module (i.e., adding trainable and matrices) is injected into selected layers of the model; then, using the power switch multi-modal pre-training data set constructed in stage one, a pre-training task facing an operation and maintenance scenario (involving multi-modal question answering, state prediction, report generation, etc.) is designed, and only the LoRA parameters and possibly a small amount of other adjustable parameters (modal encoder or output layer parameters) are optimized and trained.

[0116] After training, the learned low-rank update can be combined with the original weight to reduce the additional computational overhead during inference, or the Separation, facilitate the loading of different LoRA adapters for different sub-tasks. In this way, the application can effectively inject domain knowledge and specific capabilities into large models while significantly reducing computing resource and time costs, making them adapt to the professional needs of power switch device intelligent operation and maintenance, and generating accurate and reliable decision support information.

[0117] Step S204, input the operation data of the power switch device into the target multi-modal large model, and output the required analysis report or operation and maintenance suggestion.

[0118] Optionally, taking the power switch operation and maintenance scene as an example, the content output by the target multi-modal large model can be in the form of text, including state analysis report, fault diagnosis conclusion, risk warning information or response to user operation instructions, etc.

[0119] In this embodiment, LoRA is selected based on its excellent performance and efficiency balance in large model adaptation. More importantly, the parameter efficient fine-tuning strategy itself is one of the core technical means to achieve the low-cost and high-efficiency solution to the field adaptation problem, which can overcome the problems of high training cost and lack of professional models.

[0120] This embodiment realizes the comprehensive processing and deep fusion of multi-source heterogeneous information, and overcomes the defects that the existing single-task intelligent algorithm cannot process multi-modal data and the information utilization is insufficient. The multi-modal large model in this embodiment can simultaneously receive and understand data from multiple sources such as images, audio, time series signals, and text, effectively associate and fuse cross-modal information, and thus form a more comprehensive and accurate understanding of the operation state of the power switch device.

[0121] This embodiment significantly improves the intelligent analysis and decision-making ability of complex operation and maintenance tasks, and solves the problem that the existing algorithm is difficult to meet the demand of complex reasoning and deep understanding. This embodiment can deeply understand the operation and maintenance procedures, execute complex fault diagnosis logic, generate analysis reports or operation and maintenance suggestions that meet the requirements, and improve the depth and breadth of intelligent operation and maintenance through the powerful instruction understanding, knowledge integration and logical reasoning ability of the multi-modal large model, as well as the domain knowledge obtained by pre-training with professional data sets.

[0122] This embodiment realizes efficient and low-cost adaptation and deployment of field-specific large models. In view of the problem that the application of general large models in professional fields is costly and lacks domain knowledge, this embodiment uses parameter efficient fine-tuning technologies such as low-rank self-adaptation (LoRA) for pre-training. This enables the model to quickly and cost-effectively absorb professional knowledge and data characteristics in the field of power switch operation and maintenance while retaining strong general capabilities, significantly reducing the threshold and resource consumption of specialized large models in the application of the power industry.

[0123] The method in the embodiment effectively solves the problems of single mode, insufficient intelligence, migration difficulty and high adaptation cost of large models in existing intelligent operation and maintenance technologies, provides strong technical support for efficient, accurate and intelligent operation and maintenance of large-scale power switch devices, and has significant progress and practical value.

[0124] In the training method of the power switch device multi-modal model, multi-source heterogeneous data is collected from the actual operation and maintenance scene of the power system, and a multi-modal pre-training data set is constructed based on the multi-source heterogeneous data; so that the trained model can process multi-modal data, fully understand different types of information, and improve the accuracy of subsequent analysis. A deep learning model is used as a language model base, and multiple modal encoders are integrated to obtain an initial multi-modal large model, wherein the multi-modal encoders include an image encoder, an audio encoder, and a time series encoder; so that the multi-modal data can be more comprehensively understood and recognized. The initial multi-modal large model is trained by the multi-modal pre-training data set by using a low-rank adaptive method to obtain a target multi-modal large model; so that the amount of parameters that need to be trained can be reduced, and the training efficiency of the model can be improved. The operation data of the power switch device is input into the target multi-modal large model, and a required analysis report or operation and maintenance suggestion is output. Thus, the problems of high training cost and insufficient professionalism of existing large models can be overcome, the multi-source data related to the power switch device can be effectively associated and fused across modalities, a more comprehensive and accurate understanding of the allowable state of the power switch device can be formed, and the depth and breadth of intelligent operation and maintenance are improved.

[0125] It should be understood that, although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by combination are within the scope of protection of the present application.

[0126] Based on the same inventive concept, the embodiments of the present application also provide a power switch device multi-modal model training device for implementing the power switch device multi-modal model training method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more power switch device multi-modal model training device embodiments provided below can refer to the limitations of the power switch device multi-modal model training method described above, which will not be repeated here.

[0127] In one exemplary embodiment, as shown in Figure 5 A power switch device multi-modal model training device is provided, comprising: a data set construction module 501, a model construction module 502, a training module 503, and a processing module 504, wherein:

[0128] The data set construction module 501 is configured to collect multi-source heterogeneous data from power system actual operation and maintenance scenarios, and construct a multi-modal pre-training data set based on the multi-source heterogeneous data.

[0129] The model construction module 502 is configured to use a deep learning model as a language model base and integrate multiple modal encoders to obtain an initial multi-modal large model, wherein the multi-modal encoders include an image encoder, an audio encoder, and a time series encoder.

[0130] The training module 503 is configured to train the initial multi-modal large model using a low-rank adaptive method through the multi-modal pre-training data set to obtain a target multi-modal large model.

[0131] The processing module 504 is configured to input the operation data of the power switch device into the target multi-modal large model and output a required analysis report or operation and maintenance suggestion.

[0132] For example, the multi-source heterogeneous data includes at least one of visual information, auditory information, time series sensor data, and text information; the visual information includes at least one of monitoring images, unmanned aerial vehicle inspection photos, and infrared thermal images; the auditory information includes device operation sound and / or abnormal discharge recording; the time series sensor data includes at least one of current, voltage, partial discharge signal, vibration, temperature, and humidity; and the text information includes at least one of device account, operation and maintenance record, defect report, and technical regulation.

[0133] Exemplarily, the data set construction module 501 is specifically configured to: clean the multi-source heterogeneous data, remove damaged or irrelevant items, and anonymize or desensitize sensitive information related to privacy to obtain cleaned data of various types; perform size normalization, cropping, and brightness and / or contrast adjustment on the cleaned image data, and then perform standardization processing to obtain standard image data; perform pre-emphasis, framing, and windowing operations on the cleaned audio data, extract amplitude-related features, and then perform normalization processing to obtain normalized audio amplitude data; perform denoising, missing value filling, resampling, and normalization or standardization processing on the cleaned time series data to obtain processed time series data; convert the cleaned text data into preset digital representation data after tokenization and stop word removal processing; and perform labeling processing on the standard image data, audio amplitude data, processed time series data, and digital representation data, and integrate the labeled data into structured data, which is used to construct a multi-modal pre-training data set.

[0134] Exemplarily, the image encoder is configured to convert input image data into an image feature sequence; the audio encoder is configured to extract audio features from input audio data; the time series encoder is configured to map input time series data into time series features; the embedding layer in the initial multi-modal large model is configured to extract text features from input text data; and the initial multi-modal large model is further configured to perform multi-modal feature fusion processing on at least two of the image feature sequence, the audio features, the time series features, and the text features, perform context-dependent modeling and semantic understanding on the fused features to obtain a hidden state representation, and output a required analysis report or operation and maintenance suggestion through an output layer and decoding processing.

[0135] Exemplarily, the training module 503 is specifically configured to: inject trainable A matrices and B matrices into selected layers of the initial multi-modal large model; wherein the A matrices and the B matrices are obtained by decomposing a low-rank update added by a specified weight matrix; set a pre-training task facing an operation and maintenance scenario based on the multi-modal pre-training data set, the pre-training task including at least one of multi-modal question answering, state prediction, and report generation; train the initial multi-modal large model through the pre-training task, and adjust the A matrices and the B matrices and at least part of parameters of the multi-modal encoder or the output layer according to a training result to obtain a target multi-modal large model.

[0136] Exemplarily, during the training process of the initial multi-modal large model, only the A matrices and the B matrices and at least part of the parameters of the multi-modal encoder or the output layer are optimized, and the parameters of the original weight matrix of the initial multi-modal large model are frozen.

[0137] Exemplarily, after the initial multi-modal large model training is completed, the updated A matrix and the updated B matrix are merged with the original weight matrix, or the updated A matrix and the updated B matrix are kept separate.

[0138] The modules in the training apparatus of the power switching device multi-modal model can be implemented wholly or partially by software, hardware, and combinations thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so that the processor invokes and executes the operations corresponding to the modules.

[0139] In an exemplary embodiment, a computer device, which can be a terminal, has an internal structure diagram as shown in Figure 6 The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, mobile cellular network, near field communication (NFC), or other technologies. The computer program is executed by the processor to implement a training method of a power switching device multi-modal model. The display unit of the computer device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0140] Those skilled in the art can understand that Figure 6 The structure shown in the above description is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. Specifically, the computer device can include more or fewer components than those shown in the diagram, or combine certain components, or have a different arrangement of components.

[0141] In an exemplary embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method steps in the above embodiments when executing the computer program.

[0142] In an exemplary embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method steps in the above embodiments when executing the computer program.

[0143] In an exemplary embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method steps in the above embodiments when executing the computer program.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., and is not limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., and is not limited thereto.

[0145] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, it should be considered as the scope of the present application.

[0146] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A training method for a multimodal model of power switchgear, characterized in that, The method includes: Collect multi-source heterogeneous data from actual operation and maintenance scenarios of power systems, and construct a multimodal pre-training dataset based on the multi-source heterogeneous data; Using a deep learning model as the language model base and integrating multiple modal encoders, an initial multimodal large model is obtained, wherein the multimodal encoders include: an image encoder, an audio encoder, and a temporal encoder; A low-rank adaptive method is used to train the initial multimodal large model using the multimodal pre-training dataset to obtain the target multimodal large model; The operating data of the power switching equipment is input into the target multimodal large model, and the analysis report or operation and maintenance suggestions that meet the requirements are output.

2. The method according to claim 1, characterized in that, The multi-source heterogeneous data includes at least one of the following: visual information, auditory information, time-series sensor data, and text information; The visual information includes at least one of the following: surveillance images, drone inspection photos, and infrared thermal images. The auditory information includes: device operating sounds, and / or recordings of abnormal discharges; The time-series sensor data includes at least one of the following: current, voltage, partial discharge signal, vibration, temperature, and humidity. The text information includes at least one of the following: equipment ledger, operation and maintenance records, defect reports, and technical procedures.

3. The method according to claim 1, characterized in that, The construction of the multimodal pre-training dataset based on the multi-source heterogeneous data includes: The multi-source heterogeneous data is cleaned to remove damaged or irrelevant entries, and sensitive information involving privacy is anonymized or desensitized to obtain various types of cleaned data. After cleaning, the image data is normalized in size, cropped, and its brightness and / or contrast is adjusted. Then, it is standardized to obtain standard image data. The cleaned audio data is pre-emphasized, framed, and windowed. After extracting amplitude-related features, it is normalized to obtain normalized audio amplitude data. The cleaned time series data is subjected to denoising, missing value imputation, resampling, and normalization or standardization to obtain the processed time series data; After the cleaned text data is segmented and stop words are removed, it is converted into preset numerical representation data; The standard image data, the audio amplitude data, the processed time-series data, and the digital representation data are labeled, and the labeled data is integrated into structured data. The structured data is used to construct a multimodal pre-training dataset.

4. The method according to claim 1, characterized in that, The image encoder is used to convert the input image data into an image feature sequence; The audio encoder is used to extract audio features from the input audio data; The timing encoder is used to map the input timing data into timing features; The embedding layer in the initial multimodal large model is used to extract text features from the input text data; The initial multimodal large model is also used to perform multimodal feature fusion processing on at least two of the image feature sequence, the audio features, the temporal features, and the text features, and then perform context-dependent modeling and semantic understanding on the fused features to obtain a hidden state representation. After the hidden state representation is processed by the output layer and decoding, an analysis report or operation and maintenance suggestions that meet the requirements are output.

5. The method according to any one of claims 1 to 4, characterized in that, The method employs a low-rank adaptive approach, training the initial multimodal large model using the multimodal pre-training dataset to obtain the target multimodal large model, including: Trainable A and B matrices are injected into selected layers of the initial multimodal large model; wherein the A and B matrices are obtained by decomposing a low-rank update amount added to a specified weight matrix; Based on the multimodal pre-trained dataset, pre-training tasks for operation and maintenance scenarios are set up. The pre-training tasks include at least one of multimodal question answering, state prediction, and report generation. The initial multimodal large model is trained through the pre-training task, and the A matrix and the B matrix, as well as at least some parameters of the multimodal encoder or output layer, are adjusted according to the training results to obtain the target multimodal large model.

6. The method according to claim 5, characterized in that, During the training process of the initial multimodal large model, only the A matrix and the B matrix, as well as at least a portion of the parameters of the multimodal encoder or output layer, are optimized, and the parameters of the original weight matrix of the initial multimodal large model are frozen. After the initial training of the multimodal large model is completed, the updated A matrix and the updated B matrix are selected to be merged with the original weight matrix, or the updated A matrix and the updated B matrix are kept separate.

7. A training device for a multimodal model of power switchgear, characterized in that, The device includes: The dataset construction module is used to collect multi-source heterogeneous data from actual operation and maintenance scenarios of power systems, and to construct a multi-modal pre-training dataset based on the multi-source heterogeneous data; The model building module is used to use a deep learning model as the base of a language model and integrate multiple modal encoders to obtain an initial multimodal large model, wherein the multimodal encoders include: an image encoder, an audio encoder, and a temporal encoder; The training module is used to train the initial multimodal large model using the multimodal pre-training dataset with a low-rank adaptive method to obtain the target multimodal large model; The processing module is used to input the operating data of the power switching equipment into the target multimodal large model and output an analysis report or operation and maintenance suggestions that meet the requirements.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images