GIS insulation state diagnosis method based on cross-modal fusion and large language model

Abstract

The application relates to the technical field of insulation diagnosis, and specifically provides a GIS insulation state diagnosis method based on cross-modal fusion and a large language model, wherein multi-source heterogeneous sensing data of GIS equipment insulation states are first input into a multi-scale embedding model to complete space-time alignment, and embedded vector sequences of various modes are obtained; then a cross-modal enhancement model is used to fuse mode features and cross-modal interaction features, and an enhanced feature vector is generated; after the enhanced feature vector is mapped into a soft prompt vector, text prompts obtained through GIS fault knowledge graph matching and retrieval are input into a pre-trained large language model together, and finally, equipment insulation state diagnosis results are output. The GIS insulation state diagnosis method based on cross-modal fusion and the large language model can solve the problems that GIS multi-source heterogeneous data is difficult to be deeply fused, a model has weak data missing resistance, and diagnosis results lack explainability and knowledge reasoning capability.

Description

Technical Field

[0001] This invention relates to the field of insulation diagnostic technology, and in particular to a GIS insulation condition diagnostic method based on cross-modal fusion and a large language model. Background Technology

[0002] Gas-insulated switchgear (GIS) is a core high-voltage integrated equipment for ensuring the safe operation of the power grid, and its insulation condition has a significant impact on the safe and stable operation of the power system. Due to the complex electromagnetic environment on site, the weak anti-interference capability of single monitoring methods, and the diverse and complex characteristics of fault types, it is necessary to diagnose the insulation condition of GIS equipment.

[0003] The common GIS insulation status diagnosis methods in the existing technology mainly include the logic diagnosis method based on multi-dimensional electrical and mechanical parameter monitoring and the data-driven diagnosis method based on lightweight convolutional neural networks and transfer learning. The former realizes fault diagnosis through multi-parameter index discrimination and process-oriented processing, while the latter completes fault classification by collecting operating data and training neural network models. However, they have the following defects: (1) Data fusion only stays on the surface and lacks cross-modal deep complementarity and missing repair capabilities. It has extremely poor robustness when facing data loss or sensor failure and cannot achieve information completion; (2) The diagnosis model is a "black box" mode with insufficient interpretability. It does not integrate expert knowledge and fault mechanism and cannot provide reasoning basis and disposal suggestions; (3) It does not design an adaptive feature extraction mechanism for multi-source heterogeneous spatiotemporal signals. The signal information at different scales is severely lost, making it difficult to fully capture fault features.

[0004] There are currently no effective solutions to the problems in related technologies, such as the difficulty in deeply integrating multi-source heterogeneous data in GIS, the weak resistance of models to missing data, and the lack of interpretability and knowledge reasoning ability in diagnostic results. Summary of the Invention

[0005] The present invention provides a GIS insulation status diagnosis method based on cross-modal fusion and large language model, which at least solves the problems in related technologies such as the difficulty in deep fusion of multi-source heterogeneous data in GIS, the weak resistance of models to data loss, and the lack of interpretability and knowledge reasoning ability of diagnostic results.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The first aspect of this invention provides a GIS insulation status diagnosis method based on cross-modal fusion and a large language model. The method includes: inputting multi-source heterogeneous sensing data of the insulation status of GIS equipment into a multi-scale embedding model; performing spatiotemporal alignment of the multi-source heterogeneous sensing data based on modal features of the multi-source heterogeneous sensing data by the multi-scale embedding model, and outputting embedding vector sequences corresponding to each modality; inputting each embedding vector sequence and a gas vector into a cross-modal enhancement model; fusing cross-modal interaction features obtained through cross-modal interaction learning with the corresponding modal features by the cross-modal enhancement model, and outputting an enhanced feature vector; wherein the gas vector is a gas vector... The component data is obtained by nonlinear transformation through a fully connected layer network and then by linear projection. The enhanced feature vector is mapped to generate a soft cue vector, which is then input into a pre-constructed GIS fault knowledge graph for matching and retrieval. The triple knowledge of fault phenomenon-physical mechanism-treatment measures is extracted and converted into text cue. The soft cue vector and the text cue are input into a pre-trained large language model, which outputs the diagnostic results of the GIS equipment. The large language model is trained using GIS equipment insulation fault diagnosis samples, the soft cue vector, and the text cue from the GIS fault knowledge graph.

[0008] Preferably, before inputting the multi-source heterogeneous sensing data of the insulation state of the GIS equipment into the multi-scale embedding model, the method includes: collecting the multi-source heterogeneous sensing data of the insulation state of the GIS equipment; wherein the multi-source heterogeneous sensing data includes: ultra-high frequency signals, ultrasonic signals, and vibration signals; preprocessing the multi-source heterogeneous sensing data to obtain preprocessed standard multi-source heterogeneous sensing data; wherein the preprocessing includes: temporal alignment based on linear interpolation methods and physical scale alignment based on standardization methods.

[0009] Preferably, the method involves inputting multi-source heterogeneous sensing data on the insulation status of GIS equipment into a multi-scale embedding model. The multi-scale embedding model then performs spatiotemporal alignment based on the modal features of the multi-source heterogeneous sensing data, outputting embedding vector sequences corresponding to each modality. This includes the following steps: inputting multi-source heterogeneous sensing data on the insulation status of GIS equipment into a multi-scale embedding model; capturing the modal features of the multi-source heterogeneous sensing data through the asymmetric dual-channel architecture of the multi-scale embedding model; and mapping each modal feature to a unified dimensional space through sinusoidal position encoding and linear projection, performing spatiotemporal alignment, and outputting embedding vector sequences corresponding to each modality.

[0010] Preferably, the embedding vector sequences and gas vectors are jointly input into a cross-modal enhancement model. The cross-modal enhancement model fuses the cross-modal interaction features learned through cross-modal interaction with the corresponding modal features to output an enhanced feature vector. This includes the following steps: inputting the embedding vector sequences and gas vectors into the cross-modal enhancement model; performing self-attention calculation through the cross-modal enhancement model to extract the global context features corresponding to each embedding vector sequence and the gas vector; using the global context features of the target modality as the query vector and the global context features of the other modalities as the key vector and value vector, performing cross-attention association interaction to generate cross-modal interaction features; and using adaptive learning fusion weights to dynamically weight and sum the cross-modal interaction features with the corresponding modal features to output an enhanced feature vector.

[0011] Preferably, an adaptive learning fusion weight is used to dynamically weight and sum the cross-modal interaction features with the corresponding modal features to output an enhanced feature vector. This includes the following steps: concatenating the modal features of the target modality with the corresponding cross-modal interaction features to obtain fused input features; inputting the fused input features into a gated fully connected layer to perform a linear transformation, outputting gated intermediate features; performing normalization on the gated intermediate features to generate adaptive learning fusion weights with values ​​ranging from 0 to 1; and weighting and summing the cross-modal interaction features with the corresponding modal features based on the adaptive learning fusion weights to output an enhanced feature vector.

[0012] Preferably, before inputting the soft hint vector into the pre-constructed GIS fault knowledge graph, the method includes: collecting GIS fault-related knowledge and constructing knowledge entries; wherein, the GIS fault-related knowledge includes: GIS insulation fault types, fault physical mechanisms, sensitive monitoring modes, equipment fault locations, and power industry maintenance procedures and handling standards; encapsulating the knowledge entries into structured knowledge triples according to their association relationships, and establishing an association mapping between fault phenomena, physical mechanisms, and handling measures; and performing standardized storage and retrieval index construction on the structured knowledge triples to obtain the pre-constructed GIS fault knowledge graph.

[0013] Preferably, the enhanced feature vector, after mapping, generates a soft cue vector, which is then input into a pre-built GIS fault knowledge graph for matching and retrieval. This extracts the triple knowledge of fault phenomenon-physical mechanism-treatment measures and converts it into text cue, comprising the following steps: inputting the enhanced feature vector into a modality alignment adapter to generate a soft cue vector consistent with the embedding space of the large language model; inputting the soft cue vector into the pre-built GIS fault knowledge graph to perform feature matching and association retrieval, extracting structured triple knowledge of fault phenomenon-physical mechanism-treatment measures; and converting the extracted triple knowledge into text cue that can be input into the large model.

[0014] Preferably, before inputting the soft cue vector and the text cue into the large language model, the method includes: fixing all pre-trained weights of the base model of the initial large language model; bypassing the attention module of the Transformer layer of the large language model and adding a preset number of trainable low-rank matrices; wherein the preset number is between 2 and 4; iteratively training the parameters of the low-rank matrices based on GIS equipment insulation fault diagnosis samples, the soft cue vector, and the text cue of the GIS fault knowledge graph, combined with a joint loss function; wherein the joint loss function is composed of a weighted average of the cross-entropy loss generated by diagnosis and the feature reconstruction loss; when the iterative training reaches the convergence condition, the final large language model adapted to GIS insulation fault diagnosis is obtained.

[0015] Preferably, the soft cue vector and the text cue are jointly input into a large language model, and the large language model outputs the diagnostic results of the GIS device, including the following steps: concatenating the soft cue vector and the text cue, and inputting them together into a pre-trained large language model; performing self-attention calculation through the attention module of the large language model to associate the soft cue vector and the text cue; and outputting the diagnostic results of the GIS device through forward propagation calculation; wherein, the diagnostic results include fault type identification, fault location, and handling suggestions.

[0016] A second aspect of the present invention provides an electronic device, comprising: a processor, and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the aforementioned GIS insulation status diagnosis method based on cross-modal fusion and large language model.

[0017] A third aspect of the present invention provides a non-transitory machine-readable medium storing computer instructions, characterized in that the computer instructions are used to cause the computer to execute the aforementioned GIS insulation status diagnosis method based on cross-modal fusion and large language model.

[0018] A fourth aspect of the present invention provides a computer program product, including a computer program / instruction, characterized in that, when the computer program / instruction is executed by a processor, it implements the aforementioned GIS insulation status diagnosis method based on cross-modal fusion and large language model.

[0019] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0020] This invention provides a GIS insulation status diagnosis method based on cross-modal fusion and a large language model. It uses a multi-scale embedding model to perform spatiotemporal alignment of modal features from multi-source heterogeneous sensing data. This spatiotemporal alignment eliminates fusion barriers caused by differences in data dimensions and sampling frequencies, enabling effective fusion of different data types and solving the problem of deep fusion of multi-source heterogeneous data, thus ensuring the accuracy of data fusion. Furthermore, it utilizes a cross-modal enhancement model to interactively fuse features from various modalities. Through cross-modal interactive learning, it indirectly completes the features of damaged modalities, strengthens feature expression, and improves the model's robustness in the face of data gaps. By inputting the soft cue vector generated by feature mapping and the triplet knowledge text cue retrieved from the GIS fault knowledge graph into a pre-trained large language model trained on professional samples, the diagnostic results are combined with fault mechanisms and handling logic, solving the problems of lack of interpretability and knowledge reasoning ability in the diagnostic results and improving the guidance of the diagnostic results. Attached Figure Description

[0021] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other embodiments based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a GIS insulation status diagnosis method based on cross-modal fusion and a large language model, which is an embodiment of the present invention.

[0023] Figure 2 This is an embodiment of the present invention: an architecture diagram of a GIS equipment insulation status diagnosis system based on multi-scale embedding and knowledge-guided large model.

[0024] Figure 3 This is a flowchart of the time-frequency domain feature extraction and embedding process performed by the multi-scale embedding model in an embodiment of the present invention.

[0025] Figure 4 This is a flowchart illustrating the cross-modal enhancement model performing self-attention interaction and adaptive gating fusion according to an embodiment of the present invention.

[0026] Figure 5 This is a flowchart illustrating the training process of the low-rank matrix of a large language model according to an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the structure of an electronic device created by the present invention. Detailed Implementation

[0028] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0029] like Figure 1 As shown, in order to address the problems of difficulty in deep fusion of multi-source heterogeneous data in GIS, weak model resistance to missing data, and lack of interpretability and knowledge reasoning ability in diagnostic results, the embodiments of this invention provide a GIS insulation status diagnosis method based on cross-modal fusion and large language model.

[0030] The embodiment of the present invention provides a GIS insulation status diagnosis method based on cross-modal fusion and large language model, which includes the following steps S1 to S4.

[0031] Step S1: Input the multi-source heterogeneous sensing data of the insulation status of the GIS equipment into the multi-scale embedding model. The multi-scale embedding model performs spatiotemporal alignment based on the modal features of the multi-source heterogeneous sensing data and outputs the embedding vector sequence corresponding to each modality.

[0032] Step S2: Input the various embedded vector sequences and the gas vector into the cross-modal enhancement model. The cross-modal enhancement model will fuse the cross-modal interaction features learned through cross-modal interaction with the corresponding modal features and output the enhanced feature vector. The gas vector is obtained by nonlinearly transforming the gas component data through a fully connected layer network and then processing it through linear projection.

[0033] Step S3: The soft hint vector generated by the enhanced feature vector after mapping is input into the pre-constructed GIS fault knowledge graph for matching and retrieval, extracting the triple knowledge of fault phenomenon-physical mechanism-treatment measures and converting it into text hints.

[0034] Step S4: Input the soft cue vector and text cue together into the pre-trained large language model, and output the diagnostic results of the GIS equipment from the large language model; wherein, the large language model is trained using GIS equipment insulation fault diagnosis samples, soft cue vectors and text cue from the GIS fault knowledge graph.

[0035] The insulation condition of GIS equipment refers to the electrical insulation performance and healthy operating status of the internal insulation structure of the GIS equipment, which may include insulation defects, partial discharge, insulation aging, etc. Deterioration of gas insulation performance, etc.

[0036] Multi-source heterogeneous sensing data refers to data that can directly characterize insulation defects, partial discharge, insulation aging, and other properties of GIS equipment. Changes in insulation state, such as deterioration of gas insulation performance, are detected by sensing data from various sources with different data formats and representational dimensions, collected by multiple types of sensing devices. These data may include ultra-high frequency signals, ultrasonic signals, and vibration signals. Each type of sensing data corresponds to a specific mode.

[0037] A sensor array consisting of various types of sensing devices is deployed on the GIS equipment body, including ultra-high frequency sensors, ultrasonic sensors, and accelerometers.

[0038] Modal characteristics of multi-source heterogeneous sensing data refer to the inherent data representation attributes and physical characteristics of different types of sensing data, which may include time domain characteristics, frequency domain characteristics, energy characteristics, etc.

[0039] The Multi-Scale Spatiotemporal Embedding Model (MSTEN) is a deep learning feature embedding and spatiotemporal alignment model that adopts a dual-channel parallel extraction plus linear projection architecture. One channel uses one-dimensional dilated convolution to construct the temporal feature extraction structure, while the other channel uses short-time Fourier transform combined with a residual network to construct the frequency domain feature extraction structure.

[0040] After multi-source heterogeneous sensing data is input into MSTEN, based on the principles of multi-scale feature extraction, spatiotemporal coding alignment and unified dimension mapping, the embedded vector sequence corresponding to each modality is output.

[0041] Step S1 in the method described above provided by the embodiments of the present invention can achieve standardized encoding and spatiotemporal dimension unification of multi-source heterogeneous sensing data, providing a regular and aligned feature input basis for subsequent cross-modal interactive learning.

[0042] Cross-modal enhancement models are feature enhancement models used for the interaction and fusion of multimodal embedded vector sequences. They can be constructed using an architecture that combines self-attention and cross-attention, integrating modules for global feature extraction, cross-modal interaction association, and basic feature fusion to achieve complementary enhancement of multimodal features.

[0043] After inputting the embedding vector sequences of each modality and the gas vector into the model, based on the principle of self-attention global feature modeling and cross-attention cross-modal interaction association, the global context features of each modality are first extracted, and then the interaction features are obtained through cross-modal interaction learning. After weighted fusion with the original modal features, the enhanced feature vector is output.

[0044] Step S2 in the method described above provided by the embodiments of the present invention can realize the interaction and complementarity of multimodal features, enhance the overall expression effect of features, make up for the defects and deficiencies of single-modal data, improve the stability of the model in scenarios with abnormal or missing data, and provide reliable feature guarantee for subsequent accurate and interpretable insulation state diagnosis.

[0045] Soft suggestion vectors are semantic feature suggestion vectors adapted for knowledge graph retrieval. They are generated by mapping enhanced feature vectors through dimensional mapping and semantic space alignment.

[0046] A GIS fault knowledge graph is a structured professional knowledge base that stores information related to insulation faults in GIS equipment. It is constructed by collecting knowledge such as fault types, mechanisms, monitoring modes, fault locations, and maintenance procedures to form knowledge entries, and building a structured triple of fault phenomenon-physical mechanism-treatment measures. The graph is then built through standardized storage and indexing.

[0047] The soft suggestion vector is matched with the features of triples in the knowledge graph for similarity retrieval. The fault triple knowledge with the highest relevance is selected and then converted into readable text suggestions according to a fixed format.

[0048] Step S3 in the method described above provided by the embodiments of the present invention can achieve a precise correlation between equipment characteristics and fault expertise. By transforming data characteristics into interpretable knowledge prompts, it provides knowledge support for subsequent insulation condition diagnosis that is both inferential and instructive.

[0049] The GIS equipment insulation fault diagnosis sample is a standardized fault diagnosis dataset that includes multi-source monitoring data, fault types, fault mechanisms, maintenance basis, and standard diagnostic conclusions for various insulation fault scenarios of GIS equipment.

[0050] The large language model is a domain-adapted pre-trained language model. Based on a general pre-trained language model, it uses GIS equipment insulation fault diagnosis samples as training data. Soft cue vectors and textual cue from the GIS fault knowledge graph are used as model input features and supervision signals. Through supervised fine-tuning training, the model learns the mapping relationship between data features, fault knowledge, and diagnostic conclusions, resulting in a dedicated large language model adapted for GIS insulation fault diagnosis.

[0051] After inputting both soft cue vectors and text cues, the large language model performs semantic fusion and knowledge association processing on the two. Based on the principle of joint reasoning of data feature representation and professional knowledge, it combines fault knowledge and diagnostic logic to complete the reasoning calculation and output the GIS equipment insulation status diagnosis result.

[0052] The output diagnostic results include fault type, fault physical mechanism, fault location, risk level, and maintenance and handling suggestions. Based on the fault type, mechanism, and risk level in the diagnostic results, the degree of insulation abnormality and fault status of GIS equipment can be directly determined, achieving accurate diagnosis of insulation status.

[0053] Step S4 in the method described above provided by the present invention enables joint reasoning of data features and professional fault knowledge, significantly improving the interpretability and professionalism of diagnostic results, making the diagnostic conclusions consistent with fault mechanisms and industry handling standards, and effectively improving the accuracy and practical guidance value of GIS equipment insulation status diagnosis.

[0054] The GIS insulation status diagnosis method based on cross-modal fusion and large language model provided by the embodiments of the present invention performs spatiotemporal alignment of modal features of multi-source heterogeneous sensing data through a multi-scale embedding model. The spatiotemporal alignment eliminates the fusion obstacles caused by differences in data dimensions and sampling frequencies, realizes the effective fusion of different types of data, solves the problem of difficult deep fusion of multi-source heterogeneous data, and ensures the accuracy of data fusion.

[0055] By leveraging a cross-modal enhancement model to interactively fuse features from various modalities, and indirectly completing the features of damaged modalities through cross-modal interactive learning, the model's feature representation is strengthened, its ability to resist interference when faced with missing data is improved, and its robustness is enhanced.

[0056] By inputting the soft cue vector generated by feature mapping and the triplet knowledge text cue retrieved from the GIS fault knowledge graph into a pre-trained large language model trained on professional samples, the diagnostic results are combined with the fault mechanism and handling logic, which solves the problem of the lack of interpretability and knowledge reasoning ability of the diagnostic results and improves the guidance of the diagnostic results.

[0057] Furthermore, combined Figure 2 As shown, the method provided in the embodiment of the present invention preferably further includes, before step S1: collecting multi-source heterogeneous sensing data on the insulation status of GIS equipment; wherein, the multi-source heterogeneous sensing data includes: ultra-high frequency signals, ultrasonic signals and vibration signals; preprocessing the multi-source heterogeneous sensing data to obtain preprocessed standard multi-source heterogeneous sensing data; wherein, the preprocessing includes: time alignment based on linear interpolation method and physical scale alignment based on standardization method.

[0058] Ultra-high frequency (UHF) signals can be acquired by UHF sensors to capture nanosecond-level electromagnetic pulses generated by air gap discharge or metal particle discharge inside insulators. The sampling rate is usually at the GHz level.

[0059] Acoustic emission (AE) signals can be acquired by ultrasonic sensors to capture the sound waves of shell vibration caused by discharge or mechanical impact, and are highly sensitive to free particles and mechanical loosening.

[0060] Vibration signals (VIB) can be collected by an accelerometer and mainly reflect the operating characteristics of the circuit breaker and the vibration of the external environment.

[0061] The linear interpolation method specifically involves resampling the collected multi-source heterogeneous sensing data to a unified time grid to achieve temporal alignment of multimodal data.

[0062] Standardization methods include Z-Score standardization, Min-Max normalization, and mean normalization. The preferred embodiment of this invention uses Z-Score standardization, which calculates the mean and standard deviation of multi-source heterogeneous sensing data, normalizes the raw data according to a formula, and thus cleans the data, eliminating differences in dimensions, amplitudes, and distributions. This method can unify the numerical scale and distribution of multi-source heterogeneous data, reduce the interference of data heterogeneity on subsequent fusion, and improve data quality and model processing accuracy.

[0063] Assuming that within a certain sampling window, the first... Types of modes The original signal sequence is ,in Let be the time step. The Z-Score normalization method is used for processing, and its calculation formula is as follows:

[0064] (1).

[0065] in, Indicates the first The mode at time The normalized value, Represents the original sampled values. This represents the mean value of the signal within the sampling window. This represents the standard deviation of the signal within the sampling window. This represents a small constant used to prevent the denominator from being zero.

[0066] The method provided by the embodiments of the present invention, by combining comprehensive acquisition of multi-source heterogeneous sensing data with standardized cleaning and preprocessing, can achieve comprehensive capture of multi-dimensional information on the insulation status of GIS equipment, avoid the information limitations of single-modal data, provide standardized and unified basic data for feature extraction and fusion of subsequent models, and ensure the smooth progress of subsequent diagnostic processes.

[0067] Furthermore, such as Figure 2 , Figure 3 As shown, step S1 of the method provided in the embodiment of the present invention preferably includes the following steps: inputting multi-source heterogeneous sensing data of the insulation state of GIS equipment into a multi-scale embedding model; capturing the modal features of the multi-source heterogeneous sensing data through the asymmetric dual-channel architecture of the multi-scale embedding model; mapping each modal feature to a unified dimensional space through sinusoidal position encoding and linear projection, performing spatiotemporal alignment, and outputting the embedding vector sequence corresponding to each modality.

[0068] The asymmetric dual-channel architecture, namely the differential feature extraction and spatiotemporal alignment architecture, includes a temporal capture channel (channel one), a frequency domain capture channel (channel two), and an embedding fusion and sinusoidal position coding module.

[0069] When multi-source heterogeneous sensing data is input into a multi-scale embedding model, some data enters channel one and passes through a one-dimensional dilated convolutional layer, exponentially expanding the receptive field without reducing resolution, capturing long-range temporal dependencies, and extracting temporal features. Other data enters channel two, undergoes a short-time Fourier transform (STFT) to generate a time-frequency spectrogram, and then uses a ResNet (residual network) to extract its two-dimensional texture features. After flattening the two-dimensional features, the frequency domain feature vector is obtained.

[0070] The operation process of channel one can be preferably represented as follows:

[0071] .

[0072] in, This indicates a modified linear unit activation function. The convolution kernel is represented by the first... One weight parameter, The dilation rate, used to control the receptive field size, represents the dilation rate. This indicates the bias term.

[0073] The extracted time-domain and frequency-domain features are concatenated and then fed into a linear projection layer. This layer maps the concatenated features to a unified embedding dimension pre-defined by the Transformer-like model. Feature dimension alignment is achieved. To preserve the positional information of the sequence (crucial for determining the discharge phase), sinusoidal position coding is introduced. Position anchoring is performed, and the final generated embedding vector for:

[0074] .

[0075] in, Indicates the first The final embedding vector sequence of each modality. Representing time-domain characteristics, Represents frequency domain characteristics, This indicates a feature vector concatenation operation performed on time-domain and frequency-domain features. This represents a learnable linear projection matrix used for dimension alignment. This represents the sinusoidal positional encoding vector. Presentation layer normalization operation.

[0076] Furthermore, The input dimension is , is the dimension of the original feature vector obtained by concatenating the time-domain and frequency-domain features of the three modes: ultra-high frequency signal, ultrasonic signal, and vibration signal. The output dimension is That is, the preset unified embedding dimension.

[0077] Learnable linear projection matrix A linear transformation from the original feature space to the unified embedding space is achieved through matrix multiplication, i.e. ,in Let be the original feature vector of the m-th mode. A learnable linear projection matrix specifically for the m-th mode. The uniform-dimensional embedding vector obtained after mapping; for each modality They are independent of each other and are learned and updated separately through gradient backpropagation during the model training process.

[0078] Furthermore, feature concatenation involves concatenating time-domain and frequency-domain features along the feature dimension. That is, if the modal feature dimension of the target modality is... Cross-modal interaction feature dimension is Then the dimension of the merged input features after concatenation is... This is different from sequence splicing in the time dimension.

[0079] The method provided by the embodiments of the present invention can effectively solve the problem of deep fusion of multi-source heterogeneous sensing data of GIS equipment due to huge differences in physical dimensions and sampling frequency by multi-scale embedding, asymmetric dual-channel feature capture and unified dimensional spatiotemporal alignment. It maximizes the preservation of fault features of each modality and completes high-quality spatiotemporal alignment, providing accurate and complete feature support for subsequent cross-modal complementary enhancement, fault diagnosis and interpretable reasoning, and significantly improving data adaptability and fault information integrity.

[0080] Furthermore, such as Figure 2 , Figure 4 As shown, step S2 of the method provided in the embodiment of the present invention preferably includes the following steps: inputting each embedded vector sequence and the gas vector into the cross-modal enhancement model; performing self-attention calculation through the cross-modal enhancement model to extract the global context features corresponding to each embedded vector sequence and the gas vector; using the global context features of the target modality as the query vector and the global context features of the other modalities as the key vector and value vector, performing cross-attention association interaction to generate cross-modal interaction features; and using adaptive learning fusion weights to dynamically weight and sum the cross-modal interaction features with the corresponding modal features to output the enhanced feature vector.

[0081] Gas component data (GAS) can be acquired through gas component sensors, including... Gas pressure, trace water content, and decomposition products, such as , Concentration is a low-frequency scalar data.

[0082] The collected gas component data were resampled by linear interpolation to align with the time sequence of the UHF, ultrasonic, and vibration signals, and then Z-Score normalization was applied to align the gas component data with the physical scale of the UHF, ultrasonic, and vibration signals.

[0083] The standardized gas component data is nonlinearly transformed by a fully connected layer network to obtain semantic feature vectors. These semantic feature vectors are then linearly projected using a gas mode-specific linear projection matrix to map them to the same embedding dimension as the waveform signal modes (UHF signals, ultrasonic signals, and vibration signals), resulting in gas vectors. Since the gas component data is a static scalar and lacks temporal structure, its embedding vector is not superimposed with sinusoidal positional encoding and is directly used as the embedding vector for a single time step. This obtained gas vector can be directly input into the cross-modal enhancement model to participate in subsequent cross-modal dimensions.

[0084] Gas mode dedicated linear projection matrix It is mapped to a unified embedding dimension using a linear transformation rule, i.e. ,in, This is the embedding vector for the gas modes, i.e., the gas vector. The semantic feature vector is obtained after nonlinear transformation of gas component data through a fully connected layer, and its dimension is... , For dimension ( The trainable weight matrix is ​​trained independently of the projection matrix of the waveform signal mode. For the linear projection layer of the gas modes, A unified embedding dimension is pre-defined for the model. Ultimately, the embedding vectors for the four modalities... , , , All dimensions are This enables unified dimensional space mapping of multi-source heterogeneous data.

[0085] When the embedding vector sequences corresponding to the four modalities are input into the cross-modal augmentation model, the model first performs internal autocorrelation calculations to extract its own global contextual features. For example, for modalities... Input features Generate query vectors respectively Key vector Sum value vector The preferred calculation formula is:

[0086] .

[0087] in, These represent the trainable weight matrices for query, key, and value, respectively. These represent the generated query, key, and value matrices, respectively.

[0088] Subsequently, attention scores are calculated and weighted summation is performed to obtain the self-attention features. :

[0089] .

[0090] in, This represents the intra-modal features extracted through the self-attention mechanism. This represents a scaling factor for the feature dimension, used to prevent the gradient from vanishing due to excessively large dot product values. This represents the normalized exponential function used to generate probabilistic attention maps.

[0091] In order to utilize modality (e.g., vibration) to enhance modes (e.g., UHF), using the global context features of the target modality as the query vector and the global context features of other modalities as the key and value vectors, cross-modal attention computation is performed to conduct cross-attention association interactions and generate cross-modal interaction features; at this time, From modality ,and From modality This means "query". exist The relevant information in the data. The calculated cross-modal features are denoted as follows: .

[0092] The adaptive learning fusion weights are dynamic weighting coefficients obtained through adaptive learning of the cross-modal enhancement model. They are used to characterize the contribution weights of the original global context features of the corresponding modality and the cross-modal interaction features in the feature enhancement process.

[0093] The adaptive learning fusion weights are autonomously optimized and generated by the cross-modal augmentation model during training and inference based on the backpropagation of the model loss function. There is no need to manually preset a fixed ratio. The model adaptively updates the weight parameters according to the characteristics of different modalities and the feature distribution of the input samples.

[0094] By using the adaptive learning fusion weights, the weighted proportions of original modal features and cross-modal interaction features are dynamically allocated. The fusion weights are adaptively adjusted to address the differences in feature importance among different samples and modalities, thereby achieving a dynamic balance between preserving original features and enhancing cross-modal associations.

[0095] The method provided by the embodiments of the present invention can adaptively adapt to the feature distribution of different modalities and samples, accurately balance the fusion ratio of original features and cross-modal features, improve the rationality and robustness of cross-modal feature fusion, strengthen feature expression ability, and thus improve the technical effect of improving the model's cross-modal processing accuracy and task execution performance.

[0096] Furthermore, the method described above in the embodiments of the present invention employs adaptive learning fusion weights to dynamically weight and sum the cross-modal interaction features with the corresponding modal features, outputting an enhanced feature vector. Preferably, this method includes the following steps: concatenating the modal features of the target modality with the corresponding cross-modal interaction features to obtain fused input features; inputting the fused input features into a gated fully connected layer to perform a linear transformation, outputting gated intermediate features; performing normalization on the gated intermediate features to generate adaptive learning fusion weights with values ​​ranging from 0 to 1; and, based on the adaptive learning fusion weights, weighting and summing the cross-modal interaction features with the corresponding modal features to output an enhanced feature vector.

[0097] Modal features of the target mode With corresponding cross-modal interaction features Perform feature concatenation to obtain fused input features. Input the fused input features into a gated fully connected layer, passing them through the weights of this fully connected layer. With bias term Perform a linear transformation and output gated intermediate features.

[0098] The gated intermediate features are normalized using the Sigmoid activation function to generate adaptive learning fusion weights with values ​​ranging from 0 to 1. Based on adaptive learning, the weights are fused. Assign cross-modal interaction features and weights ( Assign modal features to the target modality, perform weighted summation, and output the enhanced feature vector. .

[0099] Specifically, adaptive learning fusion weights The preferred calculation formula is:

[0100] .

[0101] in This represents the gating weight coefficient, with a value range of [value range missing]. , This represents the Sigmoid activation function. This represents the weights of the fully connected layers in a gated network. Indicates the bias term. This indicates a splicing operation.

[0102] Final Enhancement Features We obtain the result by weighted summation:

[0103] (7).

[0104] This step implements a logical "soft switch": when cross-modal information When valid and relevant, Increase; conversely, decrease, thus ensuring the robustness of the system.

[0105] The method provided by the embodiments of the present invention can adaptively balance the fusion ratio of modal features and cross-modal interaction features, suppress cross-modal noise interference, improve the accuracy and robustness of feature fusion, enhance feature representation ability, and thus improve the performance of cross-modal tasks.

[0106] Furthermore, the method provided in the embodiments of the present invention preferably includes the following steps before step S3: collecting GIS fault-related knowledge and constructing knowledge entries; wherein, the GIS fault-related knowledge includes: GIS insulation fault types, fault physical mechanisms, sensitive monitoring modes, equipment fault locations, and power industry maintenance procedures and handling standards; encapsulating the knowledge entries into structured knowledge triples according to their association relationships, and establishing an association mapping between fault phenomena, physical mechanisms, and handling measures; performing standardized storage and retrieval index construction on the structured knowledge triples to obtain a pre-constructed GIS fault knowledge graph.

[0107] We collect relevant professional knowledge on GIS insulation fault types, fault physical mechanisms, sensitive monitoring modes, equipment fault locations, and power industry maintenance procedures and handling standards from sources such as power industry standards, equipment manufacturer technical data, on-site operation and maintenance fault data, academic research results, and industry expert experience. We then organize and integrate this knowledge to construct standardized GIS fault knowledge entries.

[0108] Based on the inherent logic and relationships between knowledge items, discrete knowledge items are encapsulated into machine-recognizable structured knowledge triples, establishing a one-to-one correspondence between fault phenomena, fault physical mechanisms, and fault handling measures.

[0109] The structured knowledge triples mentioned above are standardized and stored in a unified manner, and an efficient retrieval index is built for them, ultimately forming a pre-built GIS fault knowledge graph that can be directly accessed.

[0110] The method described above, provided by the embodiments of the present invention, can construct a standardized, structured, and rapidly searchable GIS fault knowledge system, providing accurate knowledge support for subsequent fault diagnosis and analysis, and improving the standardization and efficiency of fault judgment and handling.

[0111] Furthermore, such as Figure 2 As shown, step S3 of the method provided in the embodiment of the present invention preferably includes the following steps: inputting the enhanced feature vector into the modality alignment adapter to generate a soft cue vector consistent with the embedding space of the large language model; inputting the soft cue vector into a pre-constructed GIS fault knowledge graph, performing feature matching and association retrieval, and extracting structured triple knowledge of fault phenomenon-physical mechanism-treatment measures; and converting the extracted triple knowledge into text cue that can be input into the large model.

[0112] Due to the enhanced features In a continuous vector space, it cannot be directly understood by an LLM (Large Language Model). Therefore, embodiments of this invention utilize a modal alignment adapter (MLP Adapter) to map it as a "soft prompt".

[0113] The soft cue vector is used to retrieve information from the "GIS fault knowledge graph" based on features. For example, if the features indicate a 50Hz correlation, the knowledge triple <free particles, sensitive to, 50Hz vibration> is retrieved and converted into a text cue (Context).

[0114] The method provided by the embodiments of the present invention can achieve alignment and adaptation between multimodal features and the embedding space of a large language model, and complete accurate and efficient fault knowledge retrieval and structured extraction with the help of knowledge graphs, providing standardized and reliable input prompts for fault diagnosis of large language models and improving the accuracy of fault reasoning.

[0115] Furthermore, such as Figure 2 , Figure 5 As shown, before step S4, the method provided in the embodiment of the present invention preferably includes the following steps: fixing all pre-trained weights of the base model of the initial large language model; bypassing the attention module of the Transformer layer of the large language model and adding a preset number of trainable low-rank matrices; wherein the preset number is between 2 and 4; iteratively training the parameters of the low-rank matrices based on the GIS equipment insulation fault diagnosis samples, soft cue vectors, and text cue from the GIS fault knowledge graph, combined with the joint loss function; wherein the joint loss function is composed of the cross-entropy loss generated by diagnosis and the feature reconstruction loss weighted; when the iterative training reaches the convergence condition, the final large language model adapted to GIS insulation fault diagnosis is obtained.

[0116] The large language model provided in this invention is based on the LoRA (Low-Rank Adaptation) architecture, which maintains the pre-trained weights of the initial large language model. The process remains unchanged, except for adding two low-rank matrices as bypasses to the attention module of the Transformer layer. and .

[0117] The initial large language model refers to the original large language model (such as LLaMA and ChatGLM) that has not been specifically adapted for the GIS insulation fault field but has the ability to understand and generate general language.

[0118] The base model, which is the same entity as the initial large language model, specifically refers to the general large language model ontology with all pre-trained weights fixed as the underlying infrastructure, and its original parameters are not updated during training.

[0119] A trainable low-rank matrix is ​​a learnable parameter matrix with a very small number of parameters. Only this matrix is ​​trained without changing the weights of the base model. Increasing the number of pre-set trainable low-rank matrices allows for the injection of GIS insulation fault expertise with a minimal number of parameters, enabling the general-purpose large model to adapt to power equipment diagnostic scenarios while retaining the base model's general language capabilities.

[0120] The preferred preset quantity in the embodiments of the present invention is 2, which includes the simplest combination of 1 dimension reduction matrix and 1 dimension increase matrix. This minimizes the number of training parameters, reduces computational costs, effectively avoids catastrophic forgetting of the model, and balances adaptation efficiency and diagnostic accuracy.

[0121] Based on the base model with frozen weights, soft cue vectors and text cue from the GIS fault knowledge graph are concatenated and input into the model. The real diagnostic conclusions in the GIS insulation fault diagnosis samples are used as supervision labels, and the mask signal reconstruction is used as the self-supervised objective. The loss is calculated by substituting the joint loss function, which is composed of cross-entropy loss and feature reconstruction loss weighted together. The joint loss function is used as the optimization objective, and the parameters of the bypass low-rank matrix are updated iteratively only by forward propagation. The iteration continues until the loss converges, thus completing the professional knowledge adaptation.

[0122] Furthermore, the forward propagation calculation process is preferably expressed as the following formula:

[0123] .

[0124] in, This represents the output vector of the current layer. This represents the input vector of the current layer. This represents the original weight matrix of the frozen large model, with dimension 1. , Denotes a trainable dimensionality-reduced matrix with dimension . Initialize as a Gaussian distribution. Denotes a trainable, up-dimensional matrix with dimension . Initialize to zero. Denotes rank, and satisfies , This represents the scaling constant.

[0125] In this way, only a very small number of parameters need to be fine-tuned. and This forces large models to learn specific failure modes in the GIS field while retaining their general language understanding capabilities.

[0126] Joint loss function The preferred calculation formula is:

[0127] .

[0128] in Indicating the first in the diagnostic sample A real token, This represents the conditional probability generated by the large model. This represents a segment of the original signal that was randomly masked during the input phase. This represents a signal segment that represents the reconstructed enhanced feature vector output by the cross-modal enhancement model. This represents the hyperparameter weights that balance the two loss terms.

[0129] The final large language model obtained after the above training can directly complete fault reasoning based on multimodal monitoring features and knowledge graphs, and output interpretable natural language diagnostic reports (including fault type, mechanism, and handling suggestions); the training and deployment costs are extremely low, it retains general language capabilities, and there is no catastrophic forgetting; combined with knowledge graphs, it achieves accurate reasoning, improving the accuracy and practicality of fault diagnosis.

[0130] Furthermore, step S4 in the method provided in the embodiments of the present invention preferably includes: concatenating the soft cue vector and the text cue, and inputting them together into a pre-trained large language model; performing self-attention calculation through the attention module of the large language model to associate the soft cue vector and the text cue; and outputting the diagnostic results of the GIS device through forward propagation calculation; wherein the diagnostic results include fault type identification, fault location positioning, and handling suggestions.

[0131] The soft cue vector output by the modality alignment adapter is concatenated with the text cue obtained from the GIS fault knowledge graph retrieval to form unified input data, which is then input into the pre-trained large language model.

[0132] The Transformer attention module of the large language model performs self-attention operations to establish semantic associations between the multimodal features of the soft cue vectors and the fault knowledge of the text prompts, thereby achieving deep fusion of features and knowledge.

[0133] Through multi-layer forward propagation reasoning using a large language model, diagnostic results are output based on fused semantic information, specifically including three parts: fault type identification, fault location, and handling suggestions.

[0134] The fault type identification and fault location in the diagnostic results are based on the fault phenomena and physical mechanisms in the atlas, forming a precise fault judgment and spatial location result. Among them, the fault type identification clearly indicates whether it is "insulator surface discharge", "metal particles" or "floating potential"; the fault location, combined with the time difference and amplitude attenuation of multiple sensors, gives the specific location of the gas chamber where the fault source is located.

[0135] The generated disposal recommendations are specific operation and maintenance guidelines formed based on the disposal measures in the map, providing specific operational guidance such as "recommend power outage for maintenance", "shorten the inspection cycle" or "conduct X-ray flaw detection".

[0136] The method provided by the embodiments of the present invention can realize semantic-level fusion reasoning of multimodal monitoring features and GIS fault expertise, improve the accuracy and interpretability of diagnosis, and output integrated diagnostic results of fault determination, location and operation and maintenance guidance.

[0137] In summary, as Figure 2 As shown in the embodiments of this invention, high-dimensional alignment of heterogeneous data is achieved through the MSTEN network, the data missing problem is solved by utilizing a cross-modal enhancement model, and low-cost knowledge injection is achieved through a LoRA fine-tuning architecture. The entire process logic is closed-loop, forming an end-to-end GIS intelligent diagnostic system with self-healing capabilities and strong interpretability.

[0138] The technical effects achievable by the embodiments of the present invention include:

[0139] (1) This invention overcomes the bottleneck of inefficient alignment and fusion of multi-source heterogeneous sensing data due to significant differences in spatiotemporal scales. It designs a "Multi-Scale Spatiotemporal Embedding Network (MSTEN)," abandoning the simplistic and crude data splicing method of traditional approaches and creatively employing an asymmetric dual-channel embedding architecture. By utilizing dilated convolution to capture transient pulses of nanosecond-level UHF / AE signals, and by using STFT time-frequency spectra and ResNet to capture the texture features of long-term vibration signals, static gas scalars are mapped into the same high-dimensional space through linear projection and sinusoidal position encoding. This refined alignment strategy maximizes the preservation of the unique characteristics of different physical modes, solving the problem of deep semantic interaction of multi-source heterogeneous data within a unified spatiotemporal dimension from the data source.

[0140] (2) This invention addresses the weakness of single-modal monitoring systems in terms of poor robustness under sensor failure or strong interference environments. By constructing a "Cross-Modal Semantic Complementary Enhancement Network (CM-SCEN)," this invention changes the vulnerability of existing models that fail once data is missing. Utilizing a Transformer-based cross-attention mechanism and an adaptive gating fusion strategy, the network can dynamically perceive the signal-to-noise ratio and effectiveness of each modality. When a key modality (such as UHF) is damaged or lost, the model can automatically use the physical correlation of other intact modalities (such as vibration or sound waves) to perform "association" and feature reconstruction, giving the diagnostic system a self-healing ability similar to complementary biological senses, significantly improving its availability in harsh on-site communication environments.

[0141] (3) This invention achieves a leap from "black box" label classification to "white box" knowledge reasoning, giving the diagnostic results strong interpretability. It abandons the traditional "dumb" diagnostic mode of deep learning that only outputs probability labels, proposing a "knowledge graph-guided large model LoRA fine-tuning" architecture. Through a modal alignment adapter, physical signals are transformed into "soft hints" that the large model can understand, combined with retrieval enhancement (RAG) of the GIS fault knowledge graph. This enables the model not only to identify fault types but also to generate natural language reports containing fault cause analysis, risk assessment, and specific handling suggestions based on retrieved physical mechanisms (such as triplet knowledge), greatly assisting the decision-making process of maintenance personnel.

[0142] (4) This invention overcomes the difficulties of deploying general-purpose large models in industrial vertical fields, such as high training costs and easy forgetting. The invention introduces the "Low-Rank Adaptation (LoRA)" fine-tuning technology, which changes the expensive computing power required for full fine-tuning of large models. By freezing the base model parameters and training a low-rank matrix with very few parameters only in the attention layer, the training and deployment costs are significantly reduced (memory usage is reduced by more than 90%). It also effectively avoids the catastrophic forgetting of the original general language capabilities when the model learns GIS professional features, thus opening up the "last mile" for the deployment of general artificial intelligence technology in the field of power equipment condition maintenance.

[0143] (5) It enhances the comprehensive perception and identification capabilities for weak early faults and complex faults. Relying on multi-scale embedding and cross-modal interaction mechanisms, this invention can achieve complementarity between "macroscopic trends" and "microscopic details" in the feature extraction stage. The low-frequency trend of the vibration signal can help locate mechanical loosening, while the high-frequency pulse of the UHF signal can accurately capture micro air gap discharges. The combination of the two complements each other, effectively solving the problem of insufficient sensitivity of a single signal to complex faults (such as "particle jumping + partial discharge"), and significantly reducing the false alarm rate and the missed detection rate.

[0144] (6) An end-to-end self-supervised optimization closed loop was established, enhancing the model's generalization ability to unseen fault samples. This invention introduces "feature reconstruction loss" in the loss function design, forcing the network to actively learn the nonlinear mapping relationship between different modes during training (i.e., how to derive electromagnetic wave features from sound waves). This self-supervised learning mechanism enables the model to optimize feature representation by mining the inherent physical consistency of the data, even in the absence of sufficient labeled samples. Therefore, it can maintain good diagnostic stability when facing novel interference or rare fault modes never seen in substations. The embodiments of this invention also provide a non-transient machine-readable medium storing a computer program, wherein the computer program, when executed by the computer's processor, is used to enable the computer to execute the GIS insulation status diagnosis method based on cross-modal fusion and a large language model, as described in the embodiments of this invention.

[0145] This invention also provides an intelligent diagnostic system for the insulation status of GIS equipment suitable for high-voltage substation environments. The system first deploys multi-dimensional sensing units at key locations within the GIS equipment, including ultra-high frequency sensors for capturing partial discharge electromagnetic waves, ultrasonic and accelerometer sensors for monitoring mechanical vibration and particle agitation, and interfaces for connecting to existing SF6 gas online monitoring systems. After data acquisition, the system performs wavelet threshold denoising and Z-Score normalization preprocessing via an edge computing gateway to eliminate differences in physical dimensions.

[0146] Subsequently, the data is fed into the Multi-Scale Spatiotemporal Embedding Network (MSTEN), which utilizes an asymmetric dual-channel architecture. On the one hand, it extracts transient pulse features of high-frequency signals through dilated convolution, and on the other hand, it extracts texture features of long-term signals through time-frequency transformation combined with residual networks. Finally, it maps all modes to a unified high-dimensional vector space for alignment.

[0147] Based on this, the system runs a cross-modal complementary enhancement module, which uses the cross-attention mechanism of Transformer to calculate the correlation weights between modes in real time. When a sensor (such as UHF) is subjected to strong interference, resulting in a decrease in signal quality, the system can automatically increase the weights of other related modes (such as sound waves) to repair the damaged features.

[0148] Finally, the processed feature vectors are injected through an adapter into a large language model (such as ChatGLM) that has undergone low-rank LoRA fine-tuning. Combined with pre-built power industry standards and fault knowledge graphs, a complete diagnostic report is output, including fault type identification, physical mechanism analysis, and operation and maintenance recommendations. This embodiment demonstrates the complete technical path of this method to achieve deep fusion of multi-source data and interpretable diagnosis without the need for full retraining of a large model.

[0149] Embodiments of the present invention also provide a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the GIS insulation status diagnosis method based on cross-modal fusion and large language model, an embodiment of the present invention.

[0150] Embodiments of this invention also provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the GIS insulation status diagnosis method based on cross-modal fusion and a large language model, as described in this invention.

[0151] refer to Figure 6The present invention will now describe a structural block diagram of an electronic device that can serve as an embodiment of the present invention, serving as an example of a hardware device applicable to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0152] like Figure 6 As shown, the electronic device includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 502 or loaded from storage unit 508 into RAM (Random Access Memory) 503. RAM 503 can also store various programs and data required for the operation of the electronic device. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An I / O interface (Input / Output Interface) 505 is also connected to bus 504.

[0153] Multiple components in the electronic device are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information into the electronic device. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disks and optical discs. Communication unit 509 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0154] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as a computer program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0155] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of embodiments of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] It should be noted that the term "comprising" and its variations used in the embodiments of this invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "a plurality" mentioned in the embodiments of this invention are illustrative and not restrictive, and those skilled in the art should understand that unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0158] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0159] The steps described in the method embodiments provided by the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0160] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0161] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A GIS insulation status diagnosis method based on cross-modal fusion and large language model, characterized in that, The method includes: Multi-source heterogeneous sensing data of the insulation status of GIS equipment is input into a multi-scale embedding model. The multi-scale embedding model performs spatiotemporal alignment based on the modal features of the multi-source heterogeneous sensing data and outputs an embedding vector sequence corresponding to each modality. The embedded vector sequences and the gas vector are input into the cross-modal enhancement model. The cross-modal enhancement model fuses the cross-modal interaction features learned through cross-modal interaction with the corresponding modal features and outputs an enhanced feature vector. The gas vector is obtained by nonlinearly transforming the gas component data through a fully connected layer network and then processing it through linear projection. The enhanced feature vector is mapped to generate a soft hint vector, which is then input into a pre-constructed GIS fault knowledge graph for matching and retrieval. The triple knowledge of fault phenomenon-physical mechanism-treatment measures is extracted and converted into text hints. The soft cue vector and the text cue are input together into a pre-trained large language model, which outputs the diagnostic results of the GIS equipment. The large language model is trained using GIS equipment insulation fault diagnosis samples, the soft cue vector, and the text cue from the GIS fault knowledge graph.

2. The GIS insulation status diagnosis method based on cross-modal fusion and large language model according to claim 1, characterized in that, Before inputting the multi-source heterogeneous sensing data on the insulation status of GIS equipment into the multi-scale embedding model, the method includes: Collect multi-source heterogeneous sensing data on the insulation status of the GIS equipment; wherein, the multi-source heterogeneous sensing data includes: ultra-high frequency signals, ultrasonic signals, and vibration signals; The multi-source heterogeneous sensing data is preprocessed to obtain preprocessed standard multi-source heterogeneous sensing data; wherein, the preprocessing includes: temporal alignment based on linear interpolation method and physical scale alignment based on standardization method.

3. The GIS insulation status diagnosis method based on cross-modal fusion and large language model according to claim 1, characterized in that, Multi-source heterogeneous sensing data on the insulation status of GIS equipment is input into a multi-scale embedding model. The multi-scale embedding model performs spatiotemporal alignment based on the modal features of the multi-source heterogeneous sensing data and outputs embedding vector sequences corresponding to each modality. The process includes the following steps: Input multi-source heterogeneous sensing data on the insulation status of GIS equipment into a multi-scale embedding model; The modal features of the multi-source heterogeneous sensing data are captured through the asymmetric dual-channel architecture of the multi-scale embedding model. By using sinusoidal position encoding and linear projection, the features of each modality are mapped to a unified dimensional space, spatiotemporally aligned, and the embedded vector sequence corresponding to each modality is output.

4. The GIS insulation status diagnosis method based on cross-modal fusion and large language model according to claim 1, characterized in that, The embedded vector sequences and the gas vector are input into a cross-modal enhancement model. The cross-modal enhancement model then fuses the cross-modal interaction features learned through cross-modal interaction with the corresponding modal features to output an enhanced feature vector. This process includes the following steps: The embedded vector sequences and the gas vector are input together into the cross-modal enhancement model; Self-attention computation is performed through the cross-modal enhancement model to extract the global context features corresponding to each of the embedded vector sequences and the gas vector; Using the global context features of the target modality as the query vector and the global context features of the other modalities as the key vector and value vector, cross-attention association interaction is performed to generate cross-modal interaction features; An adaptive learning fusion weight is used to dynamically weight and sum the cross-modal interaction features with the corresponding modal features, and output an enhanced feature vector.

5. The GIS insulation status diagnosis method based on cross-modal fusion and large language model according to claim 4, characterized in that, The method employs adaptive learning to fuse weights, dynamically weighting and summing the cross-modal interaction features with the corresponding modal features to output an enhanced feature vector, including the following steps: The modal features of the target modality are concatenated with the corresponding cross-modal interaction features to obtain fused input features; The fused input features are input into a gated fully connected layer to perform a linear transformation, and the gated intermediate features are output. Normalization is performed on the gated intermediate features to generate adaptive learning fusion weights with values ​​ranging from 0 to 1; Based on the adaptive learning fusion weights, the cross-modal interaction features and the corresponding modal features are weighted and summed to output an enhanced feature vector.

6. The GIS insulation status diagnosis method based on cross-modal fusion and large language model according to claim 1, characterized in that, Before inputting the soft cue vectors into the pre-built GIS fault knowledge graph, the method includes: Collect GIS fault-related knowledge and construct knowledge entries; wherein, the GIS fault-related knowledge includes: GIS insulation fault types, fault physical mechanisms, sensitive monitoring modes, equipment fault locations, and power industry maintenance procedures and handling standards; The knowledge items are encapsulated into structured knowledge triples according to their relationships, and an association mapping of fault phenomena, physical mechanisms, and treatment measures is established. Standardized storage and retrieval indexes are built on the structured knowledge triples to obtain a pre-constructed GIS fault knowledge graph.

7. The GIS insulation status diagnosis method based on cross-modal fusion and large language model according to claim 1, characterized in that, The enhanced feature vector is mapped to generate a soft hint vector, which is then input into a pre-constructed GIS fault knowledge graph for matching and retrieval. The triple knowledge of fault phenomenon-physical mechanism-treatment measures is extracted and converted into text hints, including the following steps: The enhanced feature vector is input into the modality alignment adapter to generate a soft cue vector that is consistent with the embedding space of the large language model; The soft hint vector is input into a pre-constructed GIS fault knowledge graph, and feature matching and association retrieval are performed to extract structured triplet knowledge of fault phenomenon-physical mechanism-treatment measures. The extracted triple knowledge is transformed into text prompts that can be input into large models.

8. The GIS insulation status diagnosis method based on cross-modal fusion and large language model according to claim 1, characterized in that, Before inputting the soft cue vector and the text cue into the large language model, the method includes: Fix all pre-trained weights of the base model of the initial large language model; Bypass the attention module of the Transformer layer of the large language model and add a preset number of trainable low-rank matrices; wherein the preset number is between 2 and 4. Based on the GIS equipment insulation fault diagnosis samples, the soft hint vector, and the text hints from the GIS fault knowledge graph, the parameters of the low-rank matrix are iteratively trained using a joint loss function; wherein, the joint loss function is a weighted sum of the cross-entropy loss generated by the diagnosis and the feature reconstruction loss. When the iterative training reaches the convergence condition, the final large language model adapted for GIS insulation fault diagnosis is obtained.

9. The GIS insulation status diagnosis method based on cross-modal fusion and large language model according to claim 1, characterized in that, The soft cue vector and the text cue are input together into a large language model, and the large language model outputs the diagnostic results of the GIS device, including the following steps: The soft cue vector and the text cue are concatenated and input together into the pre-trained large language model; Self-attention is calculated using the attention module of the large language model to associate the soft cue vector with the text cue; The diagnostic results of the GIS equipment are output through forward propagation calculation; wherein, the diagnostic results include fault type identification, fault location, and handling suggestions.

10. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the GIS insulation status diagnosis method based on cross-modal fusion and large language model according to any one of claims 1 to 9.

11. A non-transitory machine-readable medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the GIS insulation status diagnosis method based on cross-modal fusion and large language model according to any one of claims 1 to 9.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the GIS insulation status diagnosis method based on cross-modal fusion and large language model as described in any one of claims 1 to 9.

Images