A substation equipment fault diagnosis method and system based on a large model

Abstract

The present application relates to the technical field of electric digital data processing, and particularly relates to a power transformation equipment fault diagnosis method and system based on a large model; the method comprises: performing feature extraction processing on multi-modal inspection data to generate multi-modal feature representation. The multi-modal feature representation is calculated forwardly through the backbone network of a mixed expert multi-modal large model to obtain intermediate feature representation, and a dynamic routing weight distribution is calculated based on the intermediate feature representation through a dynamic routing gate network. According to the dynamic routing weight distribution, a corresponding low-rank adaptive expert network is activated from a plurality of low-rank adaptive expert networks to perform feature fusion on the intermediate feature representation to generate a fused feature representation. A fault classification result and a repair suggestion text are synchronously generated. In this way, the technical problem of insufficient model reasoning robustness of the prior art in the face of knowledge conflicts and modal missing scenes is solved, and the knowledge conflicts among multiple tasks are alleviated.

Description

Technical Field

[0001] This invention relates to the field of electrical digital data processing technology, and in particular to a method and system for fault diagnosis of substation equipment based on a large model. Background Technology

[0002] Vision-Language Multimodal Large Models (VLMs), as a cutting-edge technology in the field of artificial intelligence, can jointly understand the semantics of images and text, providing a new technical approach for the automated inspection and fault diagnosis of critical power facilities such as substation equipment. In the traditional power grid inspection paradigm, fault diagnosis mainly relies on the on-site inspection and experience judgment of maintenance personnel, or on extracting the appearance features of equipment based on traditional image processing techniques such as edge detection and template matching. This method heavily depends on expert prior knowledge, has strong subjectivity in fault identification, and is easily affected by complex on-site environments such as lighting and occlusion. It also lacks robustness and is difficult to deeply fuse and comprehensively analyze multi-source information such as images, inspection record text, and sensor time-series data.

[0003] In the field of multimodal data fusion and fault diagnosis, existing technologies have yielded some insights. One existing technology (application publication number CN121659183A) discloses a method and system for predicting transformer faults in converter stations based on multimodal fusion. This method extracts feature vectors from instrument images, sensor time-series data, and fault text data using a multi-scale convolutional neural network, a spatiotemporal feature extraction network, and a semantic coding model, respectively. It then uses an evidence generator to dynamically weight and fuse these features with overall uncertainty as the weight, ultimately mapping the result to a fault probability distribution. This approach improves the accuracy of fault prediction by fusing multi-source data. However, the core optimization goal of this method is to achieve end-to-end fault probability prediction, and its model architecture and training strategy are designed specifically for single regression or classification tasks. When facing multi-task requirements in actual substation inspection scenarios, such as the need for the model to simultaneously output deterministic fault classification results and natural language-described handling suggestions, this type of solution is insufficiently adaptable. Therefore, existing fault diagnosis technologies based on multimodal fusion suffer from insufficient robustness of model reasoning when facing scenarios with knowledge conflicts and modal gaps. They struggle to balance the stability of deterministic classification output with the flexibility of generative description and the ability to adapt to incomplete input data in a single model. Summary of the Invention

[0004] To address the aforementioned shortcomings or deficiencies, this invention provides a method and system for fault diagnosis of substation equipment based on a large model, which can solve the technical problem of insufficient robustness of model reasoning in existing technologies when facing scenarios with knowledge conflicts and modal gaps.

[0005] This invention provides a method for fault diagnosis of substation equipment based on a large model, comprising: performing feature extraction processing on multimodal inspection data to generate multimodal feature representations; inputting the multimodal feature representations into a preset hybrid expert multimodal large model, which includes a backbone network, a dynamic routing gating network, and multiple low-rank adaptive expert networks; performing forward computation on the multimodal feature representations through the backbone network to obtain intermediate feature representations, and calculating dynamic routing weight distributions based on the intermediate feature representations through the dynamic routing gating network; activating corresponding low-rank adaptive expert networks from among the multiple low-rank adaptive expert networks according to the dynamic routing weight distributions to perform feature fusion on the intermediate feature representations to generate fused feature representations; and simultaneously generating fault classification results and maintenance suggestion texts based on the fused feature representations.

[0006] According to a second aspect, the present invention provides a fault diagnosis system for substation equipment based on a large model, characterized in that it includes a processor and a memory, wherein the memory stores a computer program.

[0007] The technical solution of this invention is achieved through five core steps: feature extraction, hybrid expert model computation, dynamic routing, feature fusion, and synchronous output. Specifically, feature extraction is performed on multimodal inspection data to generate multimodal feature representations, providing a structured input data foundation for subsequent model understanding. These multimodal feature representations are then input into a pre-defined hybrid expert multimodal large model comprising a backbone network, a dynamic routing gating network, and multiple low-rank adaptive expert networks, constructing the core architecture for dynamic adaptive computation. Forward computation is performed through the backbone network to obtain intermediate feature representations, and the dynamic routing gating network is used to calculate the dynamic routing weight distribution based on these intermediate feature representations, completing the evaluation of the input data pattern and pre-planning of the computation path. Based on the dynamic routing weight distribution, corresponding expert networks from multiple low-rank adaptive expert networks are activated to perform feature fusion on the intermediate feature representations, generating a fused feature representation. This achieves dynamic and refined feature enhancement based on the input content. Finally, based on this fused feature representation, fault classification results and maintenance suggestion text are generated synchronously, completing the integrated output of diagnosis and decision-making.

[0008] In this technical solution, the present invention addresses the knowledge conflict problem in multi-task joint optimization as described in the background technology. It introduces a multi-low-rank adaptive expert network architecture controlled by a dynamic routing gating network. This architecture differentiates the discriminative feature learning required for fault classification from the detailed semantic feature learning required for maintenance text generation within the model through a dynamic routing mechanism. Specifically, the dynamic routing weight distribution adaptively allocates the computation of different feature vectors to expert networks specializing in different tasks based on the content of the intermediate feature representation. This achieves relative isolation and targeted enhancement of feature spaces for different task requirements during the feature fusion stage, fundamentally mitigating knowledge interference caused by multi-task optimization in a unified feature space.

[0009] To address the issue of insufficient robustness of model inference in modality-missing scenarios described in the background art, this technical solution utilizes a collaborative mechanism of a dynamic routing gating network and multiple low-rank adapting expert networks. When the input data lacks a certain modality (e.g., only text or only images exist), the generated intermediate feature representation will primarily reflect the characteristics of the existing modality. The routing weight distribution calculated by the dynamic routing gating network based on this intermediate feature representation tends to activate low-rank adapting expert networks specializing in processing the current existing modality, thereby automatically adjusting the computational path within the model. This allows the model to bypass the pre-defined, fixed process that relies on complete modality interactions, and instead perform feature fusion calculations adapted to the current input conditions. This enables the model to maintain stable and reliable inference performance in real-world scenarios with incomplete modalities without relying on mandatory cross-modal alignment operations, effectively suppressing output errors or "illusion" phenomena caused by noisy features or missing features. Therefore, the technical solution of this invention, by constructing a hybrid expert multimodal large model architecture based on dynamic routing, solves the technical problem of insufficient robustness of model inference in the face of knowledge conflicts and modality-missing scenarios in existing technologies. This solution can dynamically allocate computing resources based on input data, alleviate knowledge conflicts between multiple tasks, and adaptively handle incomplete modal inputs. In this way, it can collaboratively improve the accuracy of fault classification, the reliability of maintenance suggestion generation, and the model's adaptability to incomplete data inputs in complex and ever-changing real power grid inspection scenarios. Attached Figure Description

[0010] Figure 1 This is a flowchart of a fault diagnosis method for substation equipment based on a large model according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall architecture of a multimodal power inspection intelligent analysis system based on a hybrid expert model architecture, according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the robustness comparison experiment results of a substation equipment inspection scenario under the missing mode according to an embodiment of the present invention; Figure 4 This is a dynamic routing weight heatmap of different types of data features in the underlying feature extraction stage according to an embodiment of the present invention; Figure 5 This is a schematic diagram of an end-to-end multimodal large model architecture for intelligent inspection tasks of substation equipment according to an embodiment of the present invention. Figure 6 This is a schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present invention. Detailed Implementation

[0011] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] During the development of this invention, researchers discovered a strong dynamic correlation between the modal completeness of input data and the optimal computational path within the model through extensive experiments and data analysis: when the input modality is complete (containing both images and text), the model needs to perform deep cross-modal alignment and interaction; when the input lacks a single modality (such as missing images or missing text), the optimal computational path should avoid performing pre-defined mandatory cross-modal interactions, instead relying on single-modal deep inference or cross-modal prior knowledge completion. Based on this relationship, this invention innovatively proposes this technical solution, utilizing a hybrid expert large model architecture that includes a dynamic routing gating network and multiple low-rank adaptive expert networks. By dynamically calculating routing weights based on the input content, it intelligently selects and activates the most suitable expert network for each feature vector, thereby achieving adaptive adjustment and optimization of the model's internal computational flow to the input data conditions, embodying the core concept of "dynamic perception, on-demand allocation, and collaborative enhancement."

[0013] Specifically, through comparative experiments, the invention team discovered that traditional multimodal fusion methods based on fixed architecture suffer from poor robustness under modality loss and knowledge conflicts during multi-task joint optimization. Their fixed cross-attention computation units generate a large number of noisy features when modality loss is present, and unified parameter updates struggle to accommodate the different needs of classification tasks for discriminative features and generation tasks for detailed features. In contrast, the proposed hybrid expert method based on dynamic routing improves the model's inference stability for incomplete modal inputs. By customizing low-rank adaptive expert networks for different task preferences, decoupling and targeted enhancement of classification and generation tasks at the feature level are achieved. Furthermore, by weighted fusion of expert network outputs based on routing weight distribution, dynamic and flexible integration of multi-source heterogeneous features is realized. This solves the problems of insufficient modality loss adaptability and mutual constraints on multi-task performance faced by existing technologies in complex real-world inspection scenarios, improving the overall accuracy, reliability, and scenario generalization ability of the fault diagnosis system.

[0014] Therefore, this invention provides a large-model-based fault diagnosis method for substation equipment, applicable to a smart grid inspection and fault diagnosis system (hereinafter referred to as the "system"). This system can load and execute a computer program stored in its memory, and run the large-model-based substation equipment fault diagnosis method described in the following embodiments, to automate the analysis and diagnosis of multimodal inspection data of substation equipment (such as transformers), thereby achieving accurate identification of equipment faults and intelligent generation of maintenance decisions. Specifically, this system can be deployed in various computing and execution environments, including but not limited to: cloud server clusters, edge computing gateways in power grid dispatch centers, local industrial control computers in substations, and mobile inspection terminals used by maintenance personnel, either as software algorithm modules or as a combination of hardware and software.

[0015] like Figure 1 As shown, the method may include: Step S110: Perform feature extraction processing on the multimodal inspection data to generate multimodal feature representations.

[0016] Multimodal inspection data refers to the raw data collection containing multiple information modalities collected during actual substation inspections. It typically includes at least images of substation equipment taken by visible light cameras or inspection robots, as well as inspection text records (such as fault descriptions and historical maintenance logs) recorded by maintenance personnel or generated by the equipment. Multimodal feature representation refers to the unified data format formed by combining the aforementioned heterogeneous raw data into real-valued vectors at the feature level through a specific encoding model, allowing for direct processing by deep learning models.

[0017] Specifically, the system can perform preprocessing (such as image scaling and normalization) and depth feature extraction on the images of substation equipment through the visual encoding module, and output image feature vectors; at the same time, it can perform word segmentation, word embedding and context encoding on the inspection text records through the text encoding module, and output text feature vectors; finally, the image feature vectors and text feature vectors are combined (for example, by concatenating them in the sequence dimension) to form a multimodal feature representation.

[0018] For example, the system receives a 224-pixel × 224-pixel image of a transformer oil conservator and the text "Slight oil seepage on the bushing". The visual encoding module uses a Vision Transformer (ViT) to encode the image into a vector sequence containing 196 image patch features, each with a dimension of 256; the text encoding module encodes the text into a vector sequence containing 5 token features, each with a dimension of 256. Finally, the system concatenates the two to generate a multimodal feature representation sequence of length 201 and a feature dimension of 256.

[0019] In some embodiments, the system can calculate the number of patch divisions of the input image using the following formula (1): (1); Formula (1) is the image grid division formula, used to determine the total number of square patches into which a two-dimensional image is divided. H represents the vertical pixel height of the input image in pixels; W represents the horizontal pixel width of the input image in pixels; P represents the side length of a single image patch in pixels; and N represents the total number of image patches obtained in the final division. For example, assuming an input infrared image of a power equipment with a size of 224×224 pixels (i.e., H=224, W=224), if the size of each image patch is set to P=16 pixels, the total number of image patches can be calculated using the formula. indivual.

[0020] Next, in this embodiment, the system can also calculate the feature sequence of the initial visual lexical units using the following formula (2): (2); Formula (2) is the initial visual feature mapping formula, which is used to convert the divided one-dimensional image block sequence into a high-dimensional feature vector sequence. R represents the initial visual word sequence obtained after linear projection; N represents the set of real numbers; and N represents the total number of image patches calculated by the above formula (1). This represents the visual embedding dimension set by the Qwen2.5-VL model (a multimodal visual language model), which is the length of the vector after each image patch is mapped. For example, continuing from the above example, if the set visual embedding dimension... =768, then the output of formula (2) is a matrix containing 196 vectors, each with a dimension of 768, that is ∈ .

[0021] Furthermore, in this embodiment, the system can also perform rotational position encoding injection on the initial visual lexical units using the following formula (3): (3); Formula (3) is the rotation position encoding transformation formula, which is used to integrate the position information in two-dimensional space into the feature vector without introducing additional learnable parameters. RoPE represents the rotation position encoding function; The initial visual word sequence generated in formula (2) is represented. Specifically, this formula enables the model to perceive the relative and absolute spatial positional relationships of each image patch in the original image by performing a rotation transformation based on position coordinates on each pair of continuous dimensions in the feature vector.

[0022] Furthermore, in this embodiment, the system can also calculate the high-level semantic features extracted by the deep backbone network using the following formula (4): (4); Formula (4) is a formula for iterative extraction of deep visual features, which is used to aggregate mid-to-high-level semantic representations with diagnostic significance layer by layer. This represents the feature sequence output by the l-th layer of the visual Transformer; This represents the nonlinear transformation function of the l-th visual backbone network (i.e., the Transformer encoder layer); Indicates the first The input feature sequence of the layer (or the initial layer l=0); L represents the total number of layers in the Qwen2.5-VL Visual Transformer, which is a positive integer constant. For example, if the total number of visual Transformer layers is set to L=24, then this formula describes the process of features undergoing 24 consecutive deep feature extractions and semantic aggregations from the initial state.

[0023] Finally, in this embodiment, the system can also calculate the visual feature vector of the aligned multimodal unified latent space using the following formula (5): (5); Formula (5) is a cross-modal dimension projection formula used to map visual features to a preset hidden dimension consistent with text features. This represents the aligned visual feature sequence of the final output; This represents the final visual feature sequence output after processing by an L-layer deep backbone network. This represents the learnable weight matrix of two linear projection layers; d represents the learnable bias vector of the two linear projection layers; d represents the uniform hidden dimension set by the language model in the multimodal large model. For example, if the uniform hidden dimension is set to d=4096, then the output of formula (5) is... Given a matrix containing 196 vectors, where each vector has a dimension of 4096, that is... This feature can be directly used for subsequent interaction and fusion with text features.

[0024] Therefore, by combining the above formulas (1) to (5), the system can complete the entire processing chain from raw pixel input to high-level semantic feature extraction and cross-modal dimension alignment for substation inspection images. This series of calculations not only effectively preserves the subtle fault textures on the equipment surface (such as color drift and local deformation caused by overheating), but also ensures the accuracy of spatial location information through rotational position encoding, and finally structures the heterogeneous image data into a standardized high-dimensional feature representation, providing a solid and high-quality visual perception foundation for subsequent multimodal fusion and large-scale intelligent reasoning.

[0025] Step S120: Input the multimodal feature representation into the preset hybrid expert multimodal large model.

[0026] The Hybrid Expert Multimodal Large Model comprises a backbone network, a dynamic routing gating network, and multiple low-rank adapting expert networks. It is a parameterized deep learning model architecture specifically designed for processing multi-source heterogeneous data. It consists of three core components: a backbone network, typically a pre-trained multimodal large language model (such as Qwen2.5-VL), serving as a shared computational foundation for basic feature extraction and fusion; a dynamic routing gating network, a small neural network used to dynamically calculate a set of probability distribution weights based on input features to determine which expert networks will participate in subsequent computations; and multiple low-rank adapting expert networks, a set of parallel-deployed, parameterized neural network modules. Each expert network is implemented using Low-Rank Adaptation (LoRA) technology, focusing on learning feature transformations and enhancements under specific modes (such as visual-dominated, text-dominated, or cross-modal fusion).

[0027] Specifically, the system can use the model loading interface to pass the multimodal feature representation generated in step S110 as an input sequence to the pre-trained hybrid expert multimodal large model. After receiving the input, the model's internal backbone network, dynamic routing gating network, and low-rank adaptive expert network will prepare to perform subsequent inference based on the preset connection relationships and computation graph.

[0028] For example, the system loads a hybrid expert multimodal large model obtained by fine-tuning a Qwen2.5-VL-7B (7 billion parameters) model. The backbone network of this model is a Qwen2.5-VL Transformer decoder, the dynamic routing gating network is a two-layer perceptron, and it is configured with 8 independent low-rank fitting expert networks (each expert network has a low-rank dimension r=8). The system inputs a multimodal feature representation sequence of length 201 into this model.

[0029] Step S130: Perform forward computation on the multimodal feature representation through the backbone network to obtain the intermediate feature representation, and calculate the dynamic routing weight distribution based on the intermediate feature representation through the dynamic routing gating network.

[0030] Intermediate feature representation refers to the feature sequence obtained after multimodal feature representation has been processed by the attention mechanism and feedforward network of several layers of the backbone network (usually the last layer or a specified layer), which contains deep cross-modal interaction and contextual semantic information. Dynamic routing weight distribution refers to a discrete probability distribution vector output by the dynamic routing gating network, where the value of each dimension corresponds to the weight or probability of a low-rank adaptive expert network being activated under the current input, and the sum of the weights of all dimensions is 1.

[0031] Specifically, the system can call the forward propagation function of the backbone network to perform self-attention and layer feedforward computation on the input multimodal feature representation, outputting an intermediate feature representation. Subsequently, the system inputs this intermediate feature representation into a dynamic routing gating network. The dynamic routing gating network first performs normalization on the input feature sequence (e.g., layer normalization), then maps each feature vector to a scoring vector through its internal linear projection layer (trainable weight matrix), and finally performs normalization processing on the scoring vector (e.g., using the Softmax function) to obtain the dynamic routing weight distribution.

[0032] For example, after the backbone network performs 32 layers of Transformer decoding on the input sequence, it outputs a dimension of The intermediate feature representation (i.e., 201 feature vectors, each 1024-dimensional) is obtained. The dynamic routing gating network performs layer normalization on this intermediate feature representation and then passes it through a weight matrix. A linear projection is performed to generate an 8-dimensional routing score vector for each position in the sequence. Then, a softmax operation is performed on the score vector for each position, ultimately outputting a vector with the following dimensions. The dynamic routing weight distribution is given by the value in the i-th row and j-th column, where the value represents the probability weight of the i-th feature vector in the sequence activating the j-th low-rank adaptive expert network.

[0033] Step S140: Based on the dynamic routing weight distribution, activate the corresponding low-rank adaptation expert network from multiple low-rank adaptation expert networks to perform feature fusion on the intermediate feature representation and generate a fused feature representation.

[0034] Activating the corresponding low-rank adapting expert network refers to selecting the top k expert networks with the highest weights from the expert network set to form a subset of expert networks for each feature vector in the intermediate feature representation, based on the dynamic routing weight distribution. Feature fusion refers to the process of performing parallel computation and weighted summation on the input intermediate feature representation using the activated expert network subset. Specifically, this includes: each activated expert network performing a low-rank linear transformation on the input features to generate a feature transformation output; performing a weighted fusion calculation on all feature transformation outputs based on the routing weights corresponding to that expert network to generate a feature fusion result; and finally, adding the feature fusion result element-wise to the original intermediate feature representation to generate a fused feature representation.

[0035] Specifically, the system can be implemented through the following sub-steps: First, based on a preset number of activations k (e.g., k=2), for each feature vector in the intermediate feature representation sequence, select the two expert network indices with the largest weight values ​​from the dynamic routing weight distribution to form an activated expert network subset for that feature vector. Then, input the feature vector in parallel into each low-rank fitting expert network in its corresponding activated expert network subset. Each expert network uses its internal low-rank decomposition matrix... and (Where r is the low-rank dimension and d is the feature dimension) Perform a linear transformation on the input features (the formula can be simplified to: ,in A feature increment is generated using the scaling factor. Next, based on the routing weights corresponding to the two activated expert networks, the two generated feature increments are weighted and summed to obtain the fused feature increment of the feature vector. Finally, the fused feature increments of all feature vectors in the sequence are added element-wise with their corresponding original intermediate feature vectors to obtain the final fused feature representation sequence.

[0036] For example, for a feature vector x (1024 dimensions) in the intermediate feature representation sequence, the two expert networks with the highest weights in its dynamic routing weight distribution are expert #3 (weight 0.6) and expert #5 (weight 0.4). The system inputs x into expert #3 and expert #5 respectively. Expert #3, through its low-rank parameter... The increment was calculated. Expert #5 calculated the increment. The weighted fusion result is: The final fused feature vector at this position is: Perform this operation on all 201 positions in the sequence to obtain a dimension of The fusion feature representation.

[0037] Step S150: Based on the fusion feature representation, simultaneously generate fault classification results and maintenance suggestion text.

[0038] Here, fault classification result refers to a definite category label output by the model, corresponding to a predefined set of fault types (such as "normal", "bushing leakage", "winding deformation", "joint overheating", etc.). Maintenance suggestion text refers to descriptive text generated by the model in natural language form, which provides handling measures, operating procedures, or precautions for the identified fault. Synchronous generation means that the model's two independent output modules (classification head and text decoder) share the same fused feature representation as input, producing structured classification results and unstructured text results in parallel or sequentially during the same forward propagation process.

[0039] Specifically, the system can be implemented through two parallel output processing modules: In the classification output submodule, the system first performs a global pooling operation on the fused feature representation sequence (e.g., averaging the feature values ​​of all feature vectors in the sequence along the sequence dimension) to obtain a global classification feature vector. Then, this classification feature vector is input into a linear classifier (fully connected layer), which outputs a class probability distribution. The final fault classification result is determined by taking the index corresponding to the maximum probability. In the text output submodule, the system inputs the fused feature representation sequence into an autoregressive text decoder (typically reusing part of the decoding layer of the backbone network or a separate decoding head). The decoder uses a start symbol (e.g., ...) <bos>Starting with the currently generated word history and fused feature representation, the probability distribution of the next word is predicted iteratively, and the word is determined by sampling (e.g., greedy decoding) until a terminator (e.g., a terminator is generated) is produced. <eos>(or until the maximum generated length is reached, all generated tokens are then concatenated to obtain the complete repair suggestion text.)

[0040] For example, the system receives dimensions as follows: The fusion feature representation is obtained. The classification output submodule performs global average pooling on it to obtain a 1024-dimensional classification feature vector, which is then input into a linear layer with 5 output neurons (corresponding to 5 fault types). After softmax, the probability distribution is obtained as follows: Taking the maximum index of 3 (assuming the index starts from 0), the output fault classification result is "connector overheating". At the same time, the text output submodule uses the fused feature representation as context to autoregressively generate a word sequence: "First, confirm the location of the overheating connector and retest it using an infrared thermometer; second, request a power outage and check the torque of the connector fastening bolts...", and finally outputs this text as a maintenance suggestion.

[0041] In some embodiments, the system can characterize the dimensional structure of the visual lexical sequence output by the visual encoding module using the following formula (6): (6); Formula (6) defines the dimension of the visual feature sequence, specifying the size of the visual data matrix generated after image encoding. V represents the visual token sequence output by the visual encoding module; R represents the set of real numbers; N represents the total number of image blocks (Tokens) calculated through the image patching process described above; and d represents the latent space dimension consistent with the language side. For example, if the input image is divided into 196 image blocks (N=196), and the unified latent space dimension set by the model is 4096 (d=4096), then formula (6) indicates that the visual token sequence V is a real matrix containing 196 vectors, each with a dimension of 4096, i.e.: .

[0042] Next, in this embodiment, the system can also convert the original text input into a discrete token index sequence using the following formula (7): (7); Formula (7) is a text segmentation serialization formula used to represent natural language instructions or describe the digital representation after parsing by the segmenter. T represents the one-dimensional text word sequence after processing by the segmenter; N represents the set of non-negative integers (usually used to represent the word index ID); M represents the total number of words after the input text is segmented. These represent the IDs of the discrete tokens arranged sequentially in the text sequence. For example, if the user inputs the text command "Check main transformer oil temperature", after segmentation and mapping by the word segmenter, it may result in a sequence containing 5 tokens (M=5), i.e. .

[0043] Furthermore, in this embodiment, the system can also implement the vectorized embedding mapping of text tokens using the following formula (8): (8); Formula (8) is a word embedding conversion formula used to convert discrete integer token ID sequences into continuous, semantically rich high-dimensional floating-point feature vector sequences. This represents the text feature sequence after the text input has been mapped through the embedding layer; Let T represent the nonlinear transformation function of the text embedding layer; T represent the original text word sequence obtained by formula (7) above; R represent the set of real numbers; M represent the total number of text words; and d represent the unified latent space dimension aligned with visual features. This means that each text word is represented as a vector of length d, and the entire sentence constitutes an M×d dimensional matrix.

[0044] Furthermore, in this embodiment, the system can also construct a unified input sequence for image-text fusion using the following formula (9): (9); Formula (9) is a multimodal sequence concatenation formula used to fuse preprocessed visual features and text features in the sequence dimension to form a single hybrid input stream. X represents the final unified multimodal input sequence. This represents the sequence concatenation operator; The prefix text token at the beginning of the sequence (typically used to guide the model into multimodal processing); V represents the visual lexical sequence. The appended text token feature (usually containing a specific end-of-sequence character or instruction) represents the text at the end of the sequence; (N+M) represents the total length of the concatenated sequence (i.e., the total number of tokens). Through this concatenation method, text and image information are integrated into the same linear space.

[0045] Finally, in this embodiment, the system can also perform rotation position encoding injection on the unified input sequence using the following formula (10): (10); Formula (10) is a rotation position encoding transformation formula, which aims to endow a unified sequence that integrates image and text features with a clear positional awareness capability. X represents the unified multimodal input sequence after concatenation; This represents the rotation position encoding function. This formula injects two-dimensional spatial positional information into the sequence without parameters by performing a position-based rotation transformation on the feature vector in the complex domain. This allows subsequent attention mechanisms to calculate weights based on the relative and absolute positions of the tokens, thereby enhancing the stability of cross-modal alignment.

[0046] Therefore, through the combination of the above formulas (6) to (10), the system can complete the entire process from inputting the original substation equipment image and text command to constructing a unified multimodal joint feature representation rich in spatial location information. This series of calculation processes not only realizes the standardization and vectorization of heterogeneous data (image pixels and text characters), but also breaks down the barriers between modalities through clever sequence splicing and position encoding strategies, enabling the model to accurately establish a strong correlation between visual abnormal signs (such as heating areas) and text commands (such as "find overheating fault") in the deep semantic space, laying a solid foundation for subsequent multimodal fusion reasoning.

[0047] In some embodiments, the system can calculate the query, key, and value feature matrix in the l-th layer multi-head attention mechanism using the following formula (11): (11); Formula (11) is the feature projection transformation formula, which is used to normalize the input features in each layer of the multimodal backbone network and map them to the attention computation space. Q represents the query feature matrix, which is used to actively match other features; K represents the key feature matrix, which is used to be queried to calculate the attention score; V represents the value feature matrix, which contains the actual information content; This represents the RMSNorm normalization function, used to stabilize data distribution; Indicates the first The input feature sequence of the layer (when l=1, it is the unified sequence output by the formula (10) above); , , These are learnable projection matrices used to linearly map input features to the query, key, and value dimension space.

[0048] Next, in this embodiment, the system can also implement the core process of multi-head scaling dot product attention calculation through the following formula (12): (12); Formula (12) is the multi-head attention score formula, which is used to measure the degree of correlation between different modal features. This indicates that the query matrix is ​​multiplied by the transpose of the key matrix, and the result reflects the original similarity between features; is a scaling factor used to prevent the gradient of the Softmax function from vanishing due to an excessively large inner product result; M is the attention mask matrix used to mask words at future positions in autoregressive scenarios such as text generation, or to control the visibility of visual and text segments during multimodal fusion, ensuring that the model can only focus on legitimate contextual information. This represents the numerical feature matrix within the attention head; Used to convert similarity scores into probability distributions.

[0049] Furthermore, in this embodiment, the system can also obtain the final output of the attention layer after multi-head splicing and linear transformation using the following formula (13): (13); Formula (13) is a multi-head self-attention aggregation formula, which is used to globally integrate local features scattered across different attention heads. This represents the output feature sequence of the l-th layer multi-head self-attention (MSA) module; It represents the complete process of multi-head self-attention operation, which includes the calculation steps of formula (11) and formula (12) and captures the complex global dependencies in multimodal data by splicing the outputs of multiple heads and performing linear transformation.

[0050] Furthermore, in this embodiment, the system can also use the following formula (14) to combine residual connection and normalization to output the final processing result of the current layer and pass it to the next layer: (14); Formula (14) is the formula for residual connection and feature update, which is used to alleviate the gradient vanishing problem in the training process of deep networks and retain the original input information. This represents the original input features of the l-th layer; This represents the feature increment (i.e., attention output) extracted through the multi-head self-attention mechanism described above. This represents the final output sequence after feature fusion and updating of the l-th layer. This residual connection structure ensures that the model can still retain early low-level visual features and early textual instruction information when stacking multiple layers of networks.

[0051] Therefore, by combining the above formulas (11) to (14), the system can construct a powerful cross-modal interactive attention fusion mechanism. Under this mechanism, textual terms (such as "overheating fault") can dynamically retrieve key "local evidence" from visual terms (such as thermal imaging features of the equipment's heating area) through attention weights during the generation process; at the same time, visual terms can also absorb semantic constraints in textual instructions. This two-way deep interaction improves the model's semantic alignment effect and fault understanding ability in complex substation inspection scenarios.

[0052] In some embodiments, the system can utilize a deep language decoder to perform contextual modeling and semantic enhancement of multimodal fusion features using the following formula (15): (15); Formula (15) is a multimodal semantic aggregation formula, which is used to transform the unified multimodal feature sequence that has been fused in the previous text into a hidden state sequence rich in deep semantic information. This represents the final fused hidden state obtained after transformation by the L-layer Qwen2.5-VL language decoder backbone network; H(0) represents the nonlinear transformation function of the multimodal backbone network; H(0) represents the initial hidden state sequence output by the multimodal fusion module. The goal of this stage is to extract a "unified context representation" from the initial fusion features that can be used by downstream tasks.

[0053] Next, in this embodiment, the system can define the basic input form of the language decoding module using the following formula (16): (16); Formula (16) defines the input feature dimension, specifying the data specifications for the language decoding module. H(0) represents the hidden state sequence input to the language decoding module; R represents the set of real numbers; S represents the total sequence length, obtained by adding the number of visual lexical units N and the number of text lexical units M; d represents the unified hidden dimension set by the Qwen2.5-VL language model. This ensures that heterogeneous graph and text features are aligned in the vector space.

[0054] Furthermore, in this embodiment, the system can implement the self-attention context aggregation process of the l-th layer decoder using the following formula (17): (17); Among them, formula (17) is the backbone self-attention update formula, which is used to aggregate text context within the language decoding layer. This represents the intermediate feature sequence of the l-th layer after being updated by the attention mechanism; This represents the multi-head self-attention computation function; This represents the root mean square layer normalization function, used to stabilize the input distribution; Indicates the first The output feature sequence of the layer (the input of Equation 16 when l=1); M is the attention mask matrix, used to implement autoregressive constraints in text generation tasks, while generally allowing full visibility of visual features.

[0055] Then, in this embodiment, the system can perform a feedforward network transformation on the attention output using the following formula (18) to extract more complex semantic features: (18); Among them, formula (18) is the backbone feedforward transformation formula, which is used to provide stable and universal language and cross-modal context modeling capabilities. This represents the output characteristics of the backbone path after processing by the l-th layer feedforward network; Represents the nonlinear transformation function of a feedforward neural network; This indicates that the attention output is normalized. This process preserves the original input information through residual connections.

[0056] Furthermore, in this embodiment, the system can calculate the specific semantic compensation increment of the MoE-LoRA (Mixture of Experts - Low-Rank Adaptation) bypass using the following formula (19): (19); Formula (19) is the expert semantic increment calculation formula, which is used to achieve feature enhancement rather than replacement in the form of "incremental injection". This represents the single-token feature vector of the current input; This represents the additional directional semantics added to the token by the k-th LoRA expert; α is a scaling factor used to control the strength of the increment. and These are the dimensionality reduction and dimensionality increase low-rank projection matrices of the k-th LoRA expert, respectively. This design significantly reduces the number of trainable parameters.

[0057] Then, in this embodiment, the system can dynamically route the Token using the following formula (20), adaptively selecting the most suitable expert according to semantic requirements: (20); (twenty one); (twenty two); Formulas (20) to (22) together constitute the dynamic routing mechanism. This represents the normalized representation of the i-th token in layer l. This is a learnable weight matrix for the router, used to map token features to the expert scoring space; This represents the scoring vector for each expert; This represents the expert weight vector after Softmax normalization; K is the total number of experts in the bypass.

[0058] Finally, in this embodiment, the system can calculate the weighted bypass semantic increment using the following formula (23): (twenty three); Formula (23) is the Top-p route aggregation formula, which is used to control the amount of computation and achieve adaptive enhancement. This represents the set of experts selected using a Top-p strategy (such as Top-1 or Top-2). This represents the final sum of bypass semantic increments for the token. This mechanism makes it easier to route visually relevant tokens to experts skilled in visual evidence aggregation, while textual reasoning tokens are routed to experts skilled in knowledge representation.

[0059] Then, in this embodiment, the system can inject the bypass increment into the backbone layer output in a residual manner using the following formula (24) to obtain the final hidden state of the layer: (twenty four); Formula (24) is the residual injection fusion formula, which is used to combine the specific enhancement effect of the bypass with the general semantics of the main branch. This represents the final hidden state sequence of the l-th layer after MoE-LoRA enhancement. By incrementing... Superimposed on the main output While maintaining the original semantic coherence, the system achieves dynamic adaptation to different fault modes and missing scenarios.

[0060] Therefore, by combining the above formulas (15) to (24), the system can construct an intelligent decoding mechanism in the language decoding stage that has both powerful general semantic modeling capabilities and can achieve dynamic adaptive enhancement through the MoE-LoRA architecture. This not only ensures a deep understanding of complex fault characteristics and text instructions in substation inspection scenarios, but also improves the robustness and generalization ability of the model when facing degraded scenarios such as modality loss and noise interference through expert routing and residual injection strategies.

[0061] In some embodiments, the system can globally converge the feature sequences output by the multimodal fusion module using the following formula (25) to extract a compact representation for classification: (25); Formula (25) is a global average pooling formula used to compress a variable-length feature sequence into a fixed-dimensional global representation vector. This represents the final global representation vector, which serves as the input feature for the classification head. This indicates the number of elements contained in the preset Token set Ω; Let Ω represent the feature vector of the i-th token in the set; ∑ represents the summation operation. In practical engineering configurations, the set Ω can be flexibly configured according to different inspection scenarios. It can contain only the special token representing the final answer, or it can contain all image and text tokens to fully utilize all modal information for fault identification. For example, in a preferred embodiment, the set Ω can be configured to contain the positions corresponding to all image and text feature vectors, that is, to use the global information of the entire sequence for classification; another configuration is that Ω only contains the feature vector positions representing special markers in the sequence, which are optimized during training to aggregate the semantics of the entire sequence.

[0062] Next, in this embodiment, the system can also calculate the probability distribution of the fault type using formula (26) and output the final predicted label: (26); Formula (26) is a linear classification and probability mapping formula used to achieve a nonlinear mapping from deep visual-text features to specific fault categories. p represents the probability distribution vector of each fault category predicted by the model, and the sum of all its elements is 1; This is the normalized exponential function, used to convert the output value into a probabilistic form; This is a learnable classification weight matrix; For bias terms; The global representation vector is calculated using the above formula (25); This indicates the category index corresponding to the highest probability, which is the final fault diagnosis result output by the system. Furthermore, the maximum value of the probability distribution p can also serve as a confidence index for the model, used to trigger automatic high-confidence alarms or prompt manual review.

[0063] Furthermore, in this embodiment, the system can also generate autoregressive text using the following formula (27) and output specific processing suggestions: (27); Formula (27) is a lexical probability prediction formula, which is used to predict the next most likely lexical term at each decoding step. This means predicting the next word based on the previously generated word sequence x≤t. The conditional probability distribution; This represents the hidden state sequence output by the decoder at the current generation step t (corresponding to the output of Formula 16 at a specific position). Used to transform the output into a probability distribution on the vocabulary; and These represent the weights and biases of the vocabulary projection layer, respectively. The model iteratively executes this formula, continuously sampling or selecting the words with the highest probabilities until a complete solution text is generated or the preset maximum length is reached.

[0064] Therefore, by combining the above formulas (25) to (27), the system can construct a dual-branch task output architecture. On the one hand, by using the efficient classification path of formulas (25) and (26), the system can accurately and quickly characterize the faults of the power equipment; on the other hand, by using the generative path of formula (27), the system can output detailed and professional on-site handling guidance and work order text for the identified fault types, thereby forming a complete intelligent diagnostic closed loop that can both "see through" the equipment status and "speak" to give suggestions.

[0065] The technical solution adopted in this embodiment is achieved through five core steps: feature extraction, hybrid expert model computation, dynamic routing, feature fusion, and synchronous output. Specifically, feature extraction is performed on multimodal inspection data to generate multimodal feature representations, providing a structured input data foundation for subsequent model understanding. These multimodal feature representations are then input into a pre-defined hybrid expert multimodal large model, which includes a backbone network, a dynamic routing gating network, and multiple low-rank adaptive expert networks, constructing the core architecture for dynamic adaptive computation. Forward computation is performed through the backbone network to obtain intermediate feature representations, and the dynamic routing gating network is used to calculate the dynamic routing weight distribution based on these intermediate feature representations, completing the evaluation of the input data pattern and pre-planning of the computation path. Based on the dynamic routing weight distribution, corresponding expert networks from multiple low-rank adaptive expert networks are activated to perform feature fusion on the intermediate feature representations, generating a fused feature representation. This achieves dynamic and refined feature enhancement based on the input content. Finally, based on this fused feature representation, fault classification results and maintenance suggestion text are generated synchronously, completing the integrated output of diagnosis and decision-making.

[0066] In this embodiment, addressing the knowledge conflict problem in multi-task joint optimization described in the background technology, the present embodiment introduces a multi-low-rank adaptive expert network architecture controlled by a dynamic routing gating network. This architecture differentiates the discriminative feature learning required for fault classification from the detailed semantic feature learning required for maintenance text generation within the model through a dynamic routing mechanism. Specifically, the dynamic routing weight distribution adaptively allocates the computation of different feature vectors to expert networks specializing in different tasks based on the content of the intermediate feature representation. This achieves relative isolation and targeted enhancement of feature spaces for different task requirements during the feature fusion stage, fundamentally mitigating knowledge interference caused by multi-task optimization in a unified feature space.

[0067] To address the issue of insufficient robustness of model inference in modality-missing scenarios described in the background art, this technical solution utilizes a collaborative mechanism of a dynamic routing gating network and multiple low-rank adapting expert networks. When the input data lacks a certain modality (e.g., only text or only images exist), the generated intermediate feature representation will primarily reflect the characteristics of the existing modality. The routing weight distribution calculated by the dynamic routing gating network based on this intermediate feature representation tends to activate low-rank adapting expert networks specializing in processing the current existing modality, thereby automatically adjusting the computational path within the model. This allows the model to bypass the pre-defined, fixed process that relies on complete modality interactions and instead perform feature fusion calculations adapted to the current input conditions. This enables the model to maintain stable and reliable inference performance in real-world scenarios with incomplete modalities without relying on mandatory cross-modal alignment operations, effectively suppressing output errors or "illusion" phenomena caused by noisy features or missing features. Therefore, the technical solution of this embodiment, by constructing a hybrid expert multimodal large model architecture based on dynamic routing, solves the technical problem of insufficient robustness of model inference in the face of knowledge conflicts and modality-missing scenarios in existing technologies. This solution can dynamically allocate computing resources based on input data, alleviate knowledge conflicts between multiple tasks, and adaptively handle incomplete modal inputs. In this way, it can collaboratively improve the accuracy of fault classification, the reliability of maintenance suggestion generation, and the model's adaptability to incomplete data inputs in complex and ever-changing real power grid inspection scenarios.

[0068] In another embodiment, such as Figure 2 This diagram illustrates the overall architecture of a multimodal intelligent power inspection and analysis system based on a hybrid expert (MoE) architecture. The system aims to accurately classify equipment faults and simultaneously generate actionable maintenance recommendations by deeply fusing image features and textual descriptions of substation equipment. The specific processing flow is as follows: First, at the input layer, the system receives images of substation equipment and corresponding equipment nameplates or inspection record text from inspection terminals. This raw multimodal inspection data then enters the encoding layer for preliminary feature extraction. In the encoding layer, the visual encoder (such as a Transformer-based visual model) is responsible for segmenting the input image, linear projection, and deep feature extraction, generating an initial visual feature sequence of fixed dimensions. Simultaneously, the text encoder (such as BERT or GPT-based text models) performs word segmentation, word embedding, and context encoding on the input natural language text, generating a corresponding text feature sequence. Next, the initial visual feature sequence and text feature sequence are fed into the core multimodal fusion module. This module, through specific attention mechanisms or feature splicing methods, breaks down the information barriers of a single modality, semantically aligning and interacting visual anomalies in the image (such as overheating, leakage, and damage) with structured parameters or historical descriptions in the text, generating a unified feature representation (i.e., a fused feature representation) containing rich cross-modal information. Subsequently, the fused feature representation is distributed in parallel to two key downstream modules: the Language Decoding Module (MoE) and the Task Output Module. The Language Decoding Module (MoE) introduces a hybrid expert mechanism, containing multiple parallel low-rank adapting expert networks and a gated routing network. For the current fused feature, the gated network dynamically calculates the weights of each expert and activates the top-k most suitable expert networks. These selected expert networks perform low-rank linear transformations and residual connections on the fused feature, quickly extracting the key feature responses for the current task. Finally, in the Task Output Module, the system achieves specific business objectives through a dual-head structure. On one hand, the classification head receives the fused feature representation, processes it through global pooling and a linear classifier, and outputs the probability distribution of the fault category to which the device belongs (e.g., judging it as "casing leakage" or "joint overheating"). On the other hand, the text generation head utilizes the high-quality feature representation enhanced by the MoE to drive the language model in an autoregressive manner, generating logically coherent natural language maintenance suggestion text containing specific operational steps word by word. This end-to-end architecture design not only ensures the accuracy of power equipment fault diagnosis, but also significantly improves the level of intelligent assistance in on-site operation and maintenance.

[0069] In another embodiment, such as Figure 3 This paper presents the robustness comparison test results for a substation equipment inspection scenario under missing modes. The experiment aims to verify the anti-interference capability and performance stability of the multimodal model (denoted as MoE-LoRA) based on hybrid expert (MoE) and low-rank adaptation (LoRA) technology proposed in this embodiment when dealing with single-mode data loss or acquisition failure in a real-world complex environment.

[0070] The specific experimental setup and data presentation are as follows: First, the experiment constructed three different input conditions to simulate actual working conditions: 3.1. Dual-modal input: The system simultaneously receives complete infrared images of the substation equipment and corresponding inspection text records. Under these conditions, the Baseline model, representing the traditional baseline method, achieved an accuracy of 0.84; while the MoE-LoRA model proposed in this embodiment improved the accuracy to 0.92. This indicates that, under ideal conditions with complete modalities, the scheme in this embodiment possesses stronger feature fusion and discrimination capabilities.

[0071] 3.2. Image Only: Simulating scenarios where text sensor malfunctions or manual omissions in recording data, the system only receives image input from the power equipment. Under this missing text modality, the accuracy of the Baseline model plummeted to 0.38; in contrast, the accuracy of the MoE-LoRA model remained high at 0.82.

[0072] 3.3. Text Only: In scenarios simulating image acquisition equipment failure or image file corruption, the system only accepts text input. The accuracy of the Baseline model further deteriorated to 0.25; while the accuracy of the MoE-LoRA model remained stable at 0.88.

[0073] Secondly, a more in-depth analysis of the above experimental data is conducted: In unimodal input scenarios with only images or only text, baseline models, due to their over-reliance on cross-validation of multimodal features, suffer severe performance degradation when a modality is missing, as their feature space changes drastically, blurring the model's decision boundaries. In contrast, the MoE-LoRA model, thanks to the dynamic routing mechanism of low-rank fitting parameters in its hybrid expert network, automatically increases the weight proportion of the expert network corresponding to the remaining available modalities when a modality is missing, forcing the model to focus on effective unimodal feature extraction and inference. This dynamic compensation mechanism enables the model to output reliable fault diagnosis results even when faced with incomplete inputs such as data transmission packet loss and occasional sensor failures common in industrial settings, improving robustness by more than double compared to traditional baseline methods.

[0074] Therefore, according to the above implementation method, while ensuring optimal dual-modal collaboration, the system, through a specific network architecture design, demonstrates strong fault tolerance and practical value in real power inspection environments with incomplete data and missing modalities, effectively solving the "vulnerability" pain point faced by traditional multimodal models in practical applications.

[0075] In another embodiment, such as Figure 4 This presentation showcases a heatmap of dynamic routing weights for different types of data features (tokens) in the underlying feature extraction stage of a multimodal model based on a hybrid expert (MoE) architecture. The heatmap visually demonstrates how this embodiment's solution, through a fine-grained expert division of labor mechanism, performs in-depth specialized processing and precise routing of information from different modalities and semantic levels.

[0076] The specific routing strategies and weight distribution characteristics are as follows: First, the system divides the input multimodal fusion features into four representative semantic categories: "abnormal image token", "background token", "rule text token", and "instruction token". For each token category, the underlying gating network dynamically calculates and assigns weights to three parallel expert branches: "visual expert network", "text expert network", and "fusion expert network".

[0077] Secondly, the model exhibits significantly differentiated and reasonable routing preferences for data inputs of different natures: 4.1. For "Abnormal Image Tokens": These tokens carry key visual cues of faults such as overheating and damage on the surface of power equipment and are highly sensitive to image details. The model assigns a very high weight (0.92) to the visual expert network, indicating that the system can accurately route image-related abnormal features to the most proficient processing unit; at the same time, only a very small weight (0.08) is assigned to the text expert, reflecting the specialization of feature processing.

[0078] 4.2. For "Rule Text Tokens": These tokens typically represent standard operating parameters or structured attribute text of a device (such as rated voltage, model number, etc.). The model primarily routes them to the text expert network (weight 0.85) to ensure accurate parsing of the semantics of this type of normalized text.

[0079] 4.3. Regarding "Instruction Tokens": These tokens contain inspection instructions or specific operational requests issued by users. Since they usually require comprehensive understanding in conjunction with the context, the model allocates its maximum weight (0.78) to the fusion expert network specifically responsible for deep interaction of cross-modal information, thereby ensuring that the instructions can be accurately responded to in a multimodal context.

[0080] 4.4. Regarding "Background Tokens": These tokens represent unimportant environmental interference information in the image. The system assigns them relatively balanced low weights (0.42, 0.25, and 0.12 respectively), indicating that the model can effectively identify this type of low-information data and reduce its resource consumption in the overall computation.

[0081] Therefore, according to the above implementation method, the system can clearly verify the dynamic scheduling capability of the hybrid expert architecture through visualized routing weight distribution. This mechanism improves the model's efficiency in extracting core fault features and its accuracy in understanding complex instructions when facing complex multimodal power inspection data, and is the key technical support for the performance breakthrough achieved by this solution at the feature processing level.

[0082] In another embodiment, such as Figure 5 This diagram illustrates an end-to-end multimodal large-scale model architecture for intelligent inspection tasks of substation equipment. This architecture deeply integrates image visual features and text semantic features, and achieves efficient fault diagnosis and intelligent decision support through specific hybrid expert (MoE) and low-rank adaptation (LoRA) technologies. The specific processing flow and module functions are as follows: First, at the input layer, the system receives visible light images or infrared thermal images of the substation equipment in parallel, along with the corresponding equipment nameplate text or historical inspection records. This raw, heterogeneous data then enters the encoding layer for preliminary feature extraction. In the encoding layer, the visual encoding module uses a ViT or Convolutional Neural Network (CNN) combined with a Multilayer Perceptron (MLP) structure to transform the input image into a compact visual token sequence. Simultaneously, the text encoding module uses a tokenizer and embedding layer to transform natural language text into a text token sequence containing contextual semantics. Next, the visual token sequence and the text token sequence are fed into the core multimodal fusion module. This module, built on the Transformer attention mechanism, breaks down the physical barriers between image pixels and text words through a cross-modal alignment algorithm. It establishes a deep semantic association between abnormal signs of equipment appearance (such as abnormal temperature areas or component damage) and the operating parameters and fault phenomena described in the text, generating a unified feature representation rich in cross-modal interaction information. Subsequently, the fused unified feature sequence is fed into the language decoding module. This module innovatively embeds a MoE-LoRA bypass module based on the traditional self-attention mechanism. This module contains two parallel low-rank adapting expert networks (…). and The system includes a gated router. The router dynamically calculates and assigns weights based on the semantic complexity of the current input features, activating the top-k most suitable expert networks to perform low-rank linear transformations and residual injection on the features. This mechanism allows the model to flexibly utilize different expert capabilities when handling complex power fault reasoning tasks, preserving the powerful generative capabilities of large models while significantly reducing computational resource consumption. Finally, the unified semantic feature sequence, enhanced by the MoE-LoRA module, is routed to the downstream task output module, achieving parallel processing of two tasks. On one hand, the classification output head receives the feature sequence, performs global average pooling and a linear classifier, and outputs the probability distribution of the fault category to which the device belongs (e.g., determining it as "transformer bushing leakage" or "bus joint overheating"). On the other hand, the text generation head utilizes the feature sequence rich in multimodal information to drive the language model in an autoregressive manner, generating a logically rigorous processing suggestion report containing specific maintenance steps and natural language descriptions word by word. This dual-head output architecture design enables the system not only to accurately identify fault types but also to provide directly executable operation and maintenance guidance, improving the intelligence and automation level of power inspection.

[0083] In some embodiments, the step of performing forward computation on the multimodal feature representation through a backbone network to obtain intermediate feature representations includes: Obtain multimodal feature representations that include image feature vectors and text feature vectors.

[0084] Image feature vectors refer to the sequence of real-valued vectors representing the semantic information of an image, output after deep feature extraction from the original inspection image by a visual encoder (such as a visual transformer ViT). Text feature vectors refer to the sequence of real-valued vectors representing the semantic information of a text, output after semantic encoding of the original inspection text by a text encoder (such as a Transformer-based word embedding model). Multimodal feature representation specifically refers to the result of combining the feature vectors of the above two modalities in a data structure, and is the direct input to the backbone network.

[0085] Specifically, the system invokes the data preprocessing pipeline to load the image feature vector and text feature vector preprocessed by the visual encoding module and text encoding module, respectively. These two feature vectors must satisfy the dimension alignment condition, meaning their feature dimensions (e.g., the length of each vector) are the same, only their sequence lengths (i.e., the number of vectors) may differ. For example, the system obtains an image feature vector with dimensions of... (representing 196 image patch features, each feature having 768 dimensions), and a text feature vector with dimensions of... (This represents 15 text lexical features, each with 768 dimensions). These two sets of data together constitute the multimodal feature representation of the current instance.

[0086] Perform sequence concatenation on the image feature vector and the text feature vector to generate a unified input sequence.

[0087] Sequence concatenation is a data operation that connects two or more feature sequences end-to-end along the sequence dimension (usually the first dimension) to merge them into a longer sequence. A unified input sequence refers to the single feature sequence formed after concatenation, containing information from all input modalities. It serves as the overall input to the backbone network, facilitating cross-modal global computation through the model's internal attention mechanism.

[0088] Specifically, the system uses a tensor concatenation operation (corresponding to the `torch.cat` function in PyTorch and the `tf.concat` function in TensorFlow) to concatenate the image feature vector sequence and the text feature vector sequence along the sequence dimension (i.e., the first dimension). The concatenation must follow a preset order, such as placing the image feature sequence before the text feature sequence, or vice versa; this order must be consistent with the format agreed upon during model training. For example, the system concatenates the above-mentioned dimension... The image feature vector and dimension are The text feature vectors are concatenated along the sequence dimension (dim=0) to obtain a dimension of The unified input sequence consists of 196 vectors from images and 15 vectors from text.

[0089] A unified input sequence is fed into the backbone network, and cross-modal feature interaction and context encoding are performed through an attention mechanism to output the encoded sequence.

[0090] In this embodiment, the backbone network specifically refers to the Transformer decoder part of a pre-trained multimodal large language model (such as Qwen2.5-VL). The attention mechanism is a core component of the Transformer model, enabling interaction between information at any position in the sequence by calculating the relevance weights between query, key, and value vectors. Cross-modal feature interaction refers to the mutual information retrieval and fusion between image vectors and text vectors through attention weights. Context encoding refers to the new representation of each position's features after aggregating information from all other relevant positions in the sequence, thus containing global contextual semantics. The encoded sequence refers to the feature sequence obtained after deep semantic fusion, obtained by processing the unified input sequence through multiple layers (e.g., the last L layers) of Transformer blocks in the backbone network.

[0091] Specifically, the system feeds a unified input sequence into the backbone network. In each layer of the network, the input sequence is first normalized (LayerNorm), and then multi-head attention is calculated. During attention calculation, a pre-defined attention mask matrix controls the visibility of vectors at different positions. For example, a full visibility mask can be set for image vectors (allowing attention to all positions in the sequence), while a causal mask (allowing attention only to the current position and previous text positions, but allowing attention to all image positions) can be set for text vectors. This achieves autoregressive constraints on text generation and allows text to freely focus on image evidence. The result of the attention calculation is then processed through residual connections, normalized again, and then nonlinearly transformed through a feed-forward network, followed by another residual connection to obtain the output of that layer. This process is stacked multiple times (L times). For example, the system will process a sequence of dimensions... The unified input sequence is input into a backbone network with 32 Transformer blocks. In the l-th layer, the input... Perform the calculation: , , , After 32 layers of computation, the output dimension remains the same. Encoded sequence .

[0092] Extract intermediate feature representations from the encoded sequence.

[0093] In this embodiment, intermediate features specifically refer to a particular subsequence or aggregated features selected from the encoded sequence for subsequent dynamic routing and feature fusion. Depending on the task design, it can be the entire encoded sequence, or a portion of the encoded sequence representing a specific modality or task, such as taking only the features corresponding to all text positions, or the average of all features.

[0094] Specifically, the system performs indexing or aggregation operations on the encoded sequence according to a preset extraction strategy. A common strategy is to directly use the entire encoded sequence as an intermediate feature representation to preserve detailed information at all positions. Another strategy is to extract only text-related feature segments, or to process image and text feature segments separately. The extraction operation can be a simple slicing operation or a pooling operation. For example, the system sets the extraction strategy to "use the complete encoded sequence." Therefore, the intermediate feature representation is the entire dimension output from the above steps. of If the strategy is "take features from all text locations", then the system will... Slice the text and extract the last 15 vectors (corresponding to the initial 15 text words), resulting in a vector with dimension [missing information]. The intermediate features are represented.

[0095] Therefore, according to the above implementation method, the system can normalize heterogeneous image and text features into a unified sequence format, utilize the powerful attention mechanism of the backbone network to achieve deep cross-modal semantic alignment and contextual understanding, and flexibly extract intermediate feature representations suitable for downstream dynamic routing and decision-making, providing a high-quality feature foundation rich in semantic information for the entire diagnostic process.

[0096] In some embodiments, the dynamic routing weight distribution is calculated based on intermediate feature representations through a dynamic routing gating network, including: The intermediate feature representation is normalized to obtain the normalized intermediate feature representation used as input to the dynamic routing gating network.

[0097] Normalization is a data standardization process designed to adjust the distribution of input data to a stable state with zero mean and unit variance, thereby accelerating the training convergence of deep neural networks and improving model stability. In this step, it specifically refers to the layer normalization operation performed on the intermediate feature representation sequence along the feature dimension (i.e., the last dimension).

[0098] Specifically, the system call layer normalization calculation unit first calculates the mean and variance of each feature vector in the intermediate feature representation sequence, and then uses these two statistics to scale and shift the feature vector, as shown in the formula: , where x is the input feature vector, μ and These are the mean and variance, γ and β are learnable scaling and translation parameters, and ε is a very small constant (e.g., 1e-5) to prevent division by zero errors. The output y is the normalized intermediate feature representation. For example, the dimension of the input intermediate feature representation is... This indicates that there are 128 feature vectors in a batch, each with 1024 dimensions. The system independently performs the above-mentioned layer normalization calculation on each 1024-dimensional vector, and the output dimension is also 1. The normalized intermediate feature representation.

[0099] By using the routing weight calculation layer configured through the dynamic routing gating network, a linear projection is performed on the normalized intermediate feature representation to generate a routing score vector.

[0100] The routing weight calculation layer is a trainable linear transformation layer (fully connected layer) in the dynamic routing gating network. Linear projection is a linear transformation operation that maps the input vector from the original feature space to the target space through a weight matrix. The routing score vector refers to the unnormalized original score vector output after linear projection, where the value of each dimension corresponds to the initial "score" of a low-rank adaptive expert network.

[0101] Specifically, the system uses normalized intermediate features as input to the routing weight calculation layer. This layer consists of a trainable weight matrix. and a bias vector Parameterization. For each feature vector in the input sequence. (e.g., 1024-dimensional), calculate its routing score vector. .in, The dimension is E is the total number of low-rank adaptive expert networks, and D is the dimension of the input features (1024). It is an E-dimensional vector. The operation results in an E-dimensional route scoring vector for each input feature vector. For example, suppose the total number of low-rank adaptation expert networks is E=8. The parameters of the route weight calculation layer... , For each of the 128 feature vectors in the batch, the system performs the linear projection described above, generating 128 8-dimensional route scoring vectors. The overall output dimension is... .

[0102] The routing score vector is normalized to output the dynamic routing weight distribution.

[0103] In this context, normalization specifically refers to the application of the Softmax function. The Softmax function transforms the original scores of a set of components (i.e., the various dimensions of the route scoring vector) into a valid probability distribution, such that the sum of all components is 1, and the value of each component is between 0 and 1. The dynamic route weight distribution refers to the route scoring vector processed by the Softmax function, where the value of each dimension represents the probability weight of activation of the corresponding low-rank adaptive expert network.

[0104] Specifically, the system scores the route vector. The Softmax function is applied to each dimension of an E-dimensional vector. The formula for calculating the Softmax value of its j-th dimension is: Calculated That is, the dynamic routing weight distribution corresponding to this feature location, which satisfies For example, for a certain feature location, its 8-dimensional route scoring vector is: After Softmax calculation, the resulting dynamic route weight distribution may be as follows: This indicates that, given the current input, the fourth expert network is most likely to be activated (weight 0.50), followed by the first (weight 0.12).

[0105] The dimension of the routing scoring vector is associated with multiple low-rank adaptation expert networks.

[0106] Here, "correlation" refers to the design ensuring that the number of dimensions (i.e., length) of the routing scoring vector is consistent with the total number of low-rank adaptation expert networks configured in the system, forming a one-to-one mapping relationship. The value of the j-th dimension in the routing scoring vector directly determines the intensity of the tendency for the j-th low-rank adaptation expert network to be activated in subsequent steps.

[0107] Specifically, during the system initialization and construction of the dynamic routing gating network, the output dimension E of the routing weight calculation layer is explicitly set to be equal to the number of instantiated low-rank adaptation expert networks. This ensures that the calculated routing score vector naturally possesses E dimensions, each inherently and sequentially associated with a specific expert network. For example, the system creates 8 low-rank adaptation expert networks during initialization, numbered as follows: to Accordingly, the weight matrix of the routing weight calculation layer in a dynamic routing gating network. It is set to 8 rows. Therefore, each route score vector calculated is 8-dimensional, where the 0th scalar value is... Association, the first scalar value and Related, and so on.

[0108] In summary, based on the above implementation method, the system can stabilize semantically rich intermediate feature representations and convert them into initial scores for each expert network through a lightweight learnable mapping layer. Finally, a clear and interpretable dynamic routing weight distribution is obtained through probabilistic normalization. This distribution probabilistically quantifies the suitability of different expert networks for processing each feature vector in the current input, providing a reliable decision basis for subsequent adaptive and refined feature fusion.

[0109] In some embodiments, based on the dynamic routing weight distribution, a corresponding low-rank adaptation expert network is activated from multiple low-rank adaptation expert networks to perform feature fusion on the intermediate feature representation and generate a fused feature representation, including: Based on the dynamic routing weight distribution, a subset of activated expert networks is selected from multiple low-rank adaptation expert networks.

[0110] Here, filtering refers to the process of comparing and ranking the numerical values ​​of each dimension in the dynamic routing weight distribution to select the top k expert networks with the highest weight values. The activated subset of expert networks refers to the set of low-ranking adaptive expert networks that are determined to participate in the current feature vector calculation after filtering, and its size is fixed at k (k is usually less than the total number of expert networks E).

[0111] Specifically, the system performs a Top-k selection operation on the dynamic routing weight distribution vector corresponding to each feature location. This operation identifies the k largest elements in the vector and their corresponding expert network indices. The k low-ranking adapted expert networks pointed to by these indices constitute the subset of activated expert networks for that feature location. Simultaneously, these k weight values ​​are recorded for subsequent weighted fusion. For example, the number of activations is set to k=2. For a given feature location, its dynamic routing weight distribution is as follows: After performing the Top-2 selection, the two elements with the highest weights are selected: 0.50 (index 3) and 0.12 (index 0). Therefore, the subset of the expert network activated at this position is { The corresponding record weight is} .

[0112] By using each low-rank adapted expert network in the expert network subset, low-rank linear transformations are performed in parallel on the intermediate feature representations to obtain multiple feature transformation outputs.

[0113] The low-rank linear transformation is a parameter-efficient linear transformation method that decomposes a standard full-rank weight matrix into the product of two low-rank matrices, thus significantly reducing the number of trainable parameters. For an input feature vector x, its transformation can be expressed as: Where A and B are low-rank matrices, It is a fixed scaling factor. The feature transformation output refers to the feature increment obtained after the input feature vector has been transformed by a low-rank adapting expert network. .

[0114] Specifically, the system will use each feature vector in the intermediate feature representation. The values ​​are distributed in parallel to each low-rank adapting expert network in its corresponding subset of activated expert networks. Each expert network contains a pair of low-rank matrices. and , where r is the low-rank dimension (much smaller than the feature dimension d), and d is the input feature dimension. For the expert network e, it is calculated as: Each expert network independently computes its own transformation output for the current feature vector. For example, suppose the feature dimension d = 1024, the low-rank dimension r = 8, and the scaling factor... =0.5. For the eigenvector (1024-dimensional), its activated expert subset is { System parallel calls and . Use its private parameters calculate ; use calculate Ultimately, two 1024-dimensional feature transformation outputs are obtained. and .

[0115] Based on the dynamic routing weights corresponding to the subset of the expert network, a weighted fusion calculation is performed on the outputs of multiple feature transformations to generate the feature fusion result.

[0116] Weighted fusion computation is a linear combination operation that sums multiple feature transformation outputs after scaling them according to the dynamic routing weights of their corresponding expert networks. The feature fusion result is a single feature vector obtained after weighted summation, which incorporates the knowledge of multiple activated expert networks.

[0117] Specifically, the system retrieves the k dynamic routing weights previously recorded that correspond to the activated expert network subset. Since these weights come from the original dynamic routing weight distribution, their sum is usually not 1 (because only the Top-k values ​​are taken), so they need to be renormalized first. Then, transform the output of k features. Perform weighted summation: The result This is the feature fusion result (a feature increment) at that feature location. For example, in the example above, the activation expert subset { The corresponding original weights are Reunification: The feature fusion result is calculated as follows: .

[0118] The feature fusion result is superimposed with the intermediate feature representation to output the fused feature representation.

[0119] In this context, "overlay" specifically refers to the residual join operation, which is the feature fusion result (feature increment). ) and the original intermediate feature representation (feature vector) The elements are added together one by one. The fused feature representation is a new sequence of feature vectors obtained after superposition. It retains the original semantic information extracted by the backbone network and injects the incremental expert knowledge selected and fused by the dynamic routing mechanism.

[0120] Specifically, the system analyzes the feature vector at each position in the sequence. Perform vector addition: All positions Combining them in the original order yields the final fused feature representation sequence. For example, for feature vectors... Assuming its value is a 1024-dimensional vector, the calculated feature fusion result... It is also a 1024-dimensional vector. (Fusing feature vectors) Each element is composed of and Adding the elements at corresponding positions yields: After performing this operation on all 128 feature locations in the batch, the output dimension is... The fusion feature representation.

[0121] Therefore, according to the above implementation method, the system can intelligently and dynamically select the most relevant set of expert networks for processing based on the specific content of different feature vectors in the input data; generate diverse feature enhancement schemes through parallel and efficient low-rank linear transformation; perform refined weighted fusion based on the routing weights to achieve selective integration of knowledge; and finally, inject the fused expert knowledge stably into the original features through residual superposition, ultimately generating a fused feature representation rich in dynamic adaptability enhancement information, providing crucial and high-quality input features for subsequent accurate classification and text generation tasks.

[0122] In some embodiments, based on the fused feature representation, fault classification results and maintenance suggestion text are generated simultaneously, including: Global pooling is performed on the fused feature representation to obtain the classification feature vector.

[0123] Global pooling is an operation that aggregates feature sequences along the sequence dimension (usually the first dimension), aiming to compress variable-length feature sequences into a fixed-length feature vector to facilitate subsequent classifier processing. Common global pooling methods include Global Average Pooling (GAP) and Global Max Pooling (GMP). This embodiment prioritizes Global Average Pooling. The classification feature vector refers to the single feature vector obtained after global pooling. It summarizes the global semantic information of the entire fused feature sequence and is specifically used for fault category discrimination.

[0124] Specifically, the system performs a global average pooling operation on the fused feature representation sequence. This operation calculates the average value across all positions in the sequence (i.e., all feature vectors) on each feature dimension. For a feature vector with dimension 1... The fusion feature representation (S is the sequence length, D is the feature dimension) is calculated using global average pooling along the sequence dimension S, and the calculation formula is as follows: Where j ranges from 1 to D. Output the result. It is a D-dimensional vector. For example, the dimension of the fused feature representation is... This indicates a sequence length of 128 and a feature dimension of 1024. The system performs global average pooling on this sequence, calculating the average of the feature values ​​at 128 positions in each of the 1024 feature dimensions. The result is a 1024-dimensional classification feature vector. .

[0125] The classification feature vector is input into the classifier for processing, the class probability distribution is output and the classification label is determined, and the fault classification result is obtained.

[0126] In this embodiment, the classifier specifically refers to a module consisting of a linear transformation layer (fully connected layer) followed by a Softmax function. The category probability distribution is a discrete probability distribution vector, where the value of each dimension represents the probability that the input sample belongs to the corresponding fault category, and the sum of the probabilities of all dimensions is 1. The classification label is the identifier (usually an integer index or a string name) corresponding to the category with the highest probability value determined according to the category probability distribution. The fault classification result is the final determined classification label.

[0127] Specifically, the system will classify feature vectors Input a linear classification layer. This layer consists of a trainable weight matrix. and a bias vector Parameterization is performed, where C is the predefined total number of fault categories. The resulting logits (raw scores) vector is calculated. Then, the Softmax function is applied to z to transform it into a class probability distribution p: Finally, the system uses the argmax operation to find the index of the maximum value in the probability distribution p. This index corresponds to the final fault classification label. For example, if the number of fault categories C=5, they correspond to "normal," "bushing leakage," "winding deformation," "joint overheating," and "core grounding," respectively. Classification feature vector 1024 dimensions. Linear classification layer parameters. The 5-dimensional logits vector z is calculated, and the probability distribution is obtained after softmax. Taking argmax to get index 2 (assuming it starts from 0), the fault classification result is "winding deformation".

[0128] The fused feature representation is input into the text decoder, and a word sequence is generated in an autoregressive manner to obtain the maintenance suggestion text.

[0129] The text decoder is an autoregressive generative model based on the Transformer architecture. It predicts the next word based on the input context (fused feature representation) and the already generated partial sequences. The autoregressive approach means the generation process is iterative; at each step, the model uses all previously generated words as input to predict the next word, repeating this process until a terminator is generated or the maximum length is reached. The word sequence is the basic unit sequence obtained after text segmentation. The repair suggestion text is a readable string converted back from the final generated word sequence.

[0130] Specifically, the system will represent the complete fused feature sequence. (dimension) This is provided as additional contextual input to the text decoder. Decoding starts from a specific starting word (such as...). <bos>The process begins at step t (t starts from 0). The decoder's input is the sequence of the first t generated tokens after being transformed by the embedding layer. and context The decoder calculates the hidden state for the current step. The probability distribution of the next word is obtained by passing it through a linear projection layer to a logits vector of vocabulary size V, and then through Softmax. The system determines the (t+1)th word based on a preset decoding strategy (such as greedy decoding, i.e., selecting the word with the highest probability). The terminology is appended to the generated sequence and used as input for the next step. This process is repeated until a terminating terminology (such as...) is generated. <eos>(or reaching the preset maximum generation length) Finally, the generated word sequence (excluding the starting word) is converted into a coherent repair suggestion text string through the decoding function of the word segmenter. For example, the system uses a vocabulary containing 50,000 words. The starting word ID is 1, and the ending word ID is 2. Fusion Feature Representation Dimensions The generation process is as follows: The initial input is [ <bos>], decoder combination Predict the next word, and greedily decode to select the word with ID 305 (corresponding to "first"). The sequence becomes [ <bos>First of all. Next step, with [ <bos>First, [[] is the input, combined with] Predicting, select the term with ID 1024 (corresponding to "confirm"). Continuing in this manner, a sequence may be generated. Remove the start and end tokens, and then extract the intermediate ID sequence. Converted to text: "First, confirm the location of the heating connector and retest it using an infrared thermometer. Second, request a power outage and check if the torque of the connector fastening bolts meets the standard..." This is the maintenance suggestion text.

[0131] Therefore, according to the above implementation method, the system can drive two decoupled output modules in parallel and efficiently based on the same fused feature representation rich in dynamic enhancement information: on the one hand, it outputs structured fault classification results quickly and stably through global semantic convergence and linear discriminant analysis, meeting the inspection system's requirements for the certainty and statistical validity of diagnostic conclusions; on the other hand, it generates maintenance suggestions in natural language description flexibly and coherently through autoregressive context decoding, meeting the on-site handling requirements for the interpretability of operational details and causal logic. This synchronous generation mechanism of "feature sharing and task decoupling" effectively balances the high accuracy of the classification task and the high flexibility of the generation task while ensuring the consistency of diagnostic basis.

[0132] In some embodiments, the step of performing cross-modal feature interaction and context encoding through an attention mechanism and outputting an encoded sequence includes: Perform layer normalization on the unified input sequence.

[0133] Layer normalization is a technique that standardizes the input of neurons in a layer of a deep neural network. It adjusts the distribution of the input data of that layer to have a mean of 0 and a variance of 1, in order to alleviate the problem of internal covariate bias, accelerate model training, and improve generalization ability.

[0134] Specifically, the system takes a unified input sequence X (with dimension X) as input. (where S is the sequence length and D is the feature dimension) Perform layer normalization. This operation is performed independently on the feature vector at each position in the sequence. For a D-dimensional feature vector x, calculate its mean μ and variance. Then normalize: ,in It is a very small constant (e.g.) This is used for numerical stability. Finally, scaling and translation are performed: ,in and These are learnable affine transformation parameters. For example, suppose the dimension of the uniform input sequence is... The system performs the above calculations on 256 of the 768-dimensional feature vectors. For one of the vectors x, assume its... , Initially, it is a vector of all 1s. If the initial vector is all zeros, then the normalized vector y is calculated.

[0135] Based on the normalized unified input sequence, the query vector, key vector, and value vector are calculated through multi-head attention, and attention weighting is performed.

[0136] Multi-head attention is an extension of the attention mechanism. It projects input features into multiple different subspaces (called "heads"), calculates attention independently in each subspace, and finally merges the results to enhance the model's ability to capture information from different perspectives. The query vector, key vector, and value vector are three matrices obtained by linearly projecting the input sequence, used to calculate attention weights and generate weighted outputs, respectively. Attention weighting calculation refers to the process of calculating weights based on the similarity between the query vector and the key vector, and then using these weights to perform a weighted summation of the value vectors.

[0137] Specifically, the system will normalize the sequence With three trainable weight matrices respectively Multiply to obtain the query matrix Key matrix Sum matrix For multi-head attention, the feature dimension D is divided into h heads, and the dimension of each head is... Each head independently computes scaled dot product attention: , where i represents the i-th head. Finally, the outputs of all heads are concatenated and then subjected to a linear projection. This yields the output of multi-head attention. For example, let the Y dimension be... If the number of heads h=12, then the dimension of each head is... Calculations yielded All dimensions are (Each head). For the i-th head, calculate Get one The similarity matrix, divided by The attention weight matrix is ​​then obtained through softmax, and then... Multiply to get the output of the head The outputs of the 12 heads are concatenated into... , and then with Multiplying these together yields the final multi-head attention output.

[0138] Based on the preset attention mask, the visibility control amount of vectors at different positions in the sequence is set.

[0139] The attention mask is a matrix with the same shape as the attention weight matrix, used to set the attention weights at certain locations to a very small negative number (e.g., before softmax calculation). This makes the weight at that position almost zero after softmax, thus achieving "masking" or "visibility control".

[0140] Specifically, the system calculates At this time, a preset mask matrix (Mask) is added to the scaled similarity matrix. In the Mask matrix, the positions that are allowed to be of interest are set to 0, and the positions that need to be masked are set to a large negative number (e.g., ...). This type of mask is typically used when processing variable-length sequences, achieving causal prediction (autoregressive generation), or controlling cross-modal information flow. For example, in a cross-modal task, a uniform input sequence is composed of image features (the first 196 vectors) and text features (the last 60 vectors). A pre-defined attention mask rule could be: all image features can attend to each other and all text features; text features can attend to all image features, but only to themselves and preceding text features (achieving causality in text generation). Therefore, for the 200th position in the sequence (a text position), In the vector, for indices 0 to 255 (all image and text positions), if the index of the position is greater than 200 and belongs to the text part, the Mask value is set to -10000.0; otherwise, it is set to 0.

[0141] The results of attention-weighted computation are then subjected to residual connections, layer normalization, and feedforward network transformation in sequence.

[0142] Residual connections are structures where the input to a layer is skipped and directly added to its output, i.e., Output = F(x) + x, where F(x) is the transformation of that layer. This helps alleviate the vanishing gradient problem in deep networks, allowing the network to be trained to greater depths. Feedforward network transformations typically refer to a two-layer feedforward neural network that performs independent nonlinear transformations on the features at each location to enhance the model's expressive power.

[0143] Specifically, in one encoding layer of the Transformer, let the input be... The multi-head attention output is MHA (Multi-Head Attention). First, residual connections and layer normalization are performed: Then Z is input into a feedforward network, which typically consists of two linear transformations and an activation function: ,in , Typically, it is 4 times D. The result of FFN(Z) is again residually joined with Z and layer-normalized: For example, suppose Dimensions The output dimension of MHA is the same. . get Z is obtained by using LayerNorm. The intermediate dimension of the feedforward network. Z passes through the first linear layer. Become After ReLU activation, it passes through a second linear layer. Return This yields FFN(Z). Finally, the calculation is performed. Output .

[0144] The process iteratively executes multiple processing steps, including attention weighting calculation, residual connection, layer normalization, and feedforward network transformation, to output the encoded sequence.

[0145] Here, "iterative execution multiple times" refers to stacking the computational units of "multi-head attention - residual normalization - feedforward network - residual normalization" sequentially L times to form a deep Transformer encoder. L is the depth (number of layers) of the model. The encoded sequence refers to the final feature representation sequence obtained after the original uniform input sequence has undergone such a deep transformation of L layers, which fully integrates the contextual information within the sequence (including cross-modal information).

[0146] Specifically, the system will output the previous layer's output. As input to the next layer l Repeat the above steps. This process proceeds from l=1 to l=L. The output of the Lth layer... This is the final encoded sequence. For example, suppose the total number of layers L=12. (Unified input sequence) The result obtained after the first layer of calculation , As Input to the 2nd layer, and repeat this process until the 12th layer outputs. . The dimension is still However, each feature vector contains rich semantic information after 12 layers of deep interaction and transformation.

[0147] Therefore, according to the above implementation, the system can achieve deep, bidirectional semantic alignment and information fusion between image and text features in a unified feature space through deeply stacked Transformer encoding layers and a differentiable attention mechanism. The preset attention mask provides flexible and controllable rules for interactions between different modalities. Residual connections and layer normalization ensure the stability of deep model training. The final output encoded sequence, where each feature encapsulates the context of the entire input sequence (containing all modal information), provides a high-quality, semantically rich feature foundation for subsequent dynamic routing and accurate decision-making.

[0148] In some embodiments, low-rank linear transformations are performed in parallel on the intermediate feature representations through various low-rank adapting expert networks in a subset of the expert network, resulting in multiple feature transformation outputs, including: The intermediate feature representations are distributed in parallel to each low-rank adapting expert network in the subset of expert networks.

[0149] Parallel distribution is a computational strategy that involves simultaneously and independently sending the same input data (intermediate feature representation) to multiple processing units (in this case, activated low-rank adaptive expert networks) so that these units can begin computation concurrently, thereby improving overall processing efficiency. The expert network subset refers to the set of k low-rank adaptive expert networks currently required for computation, selected based on dynamic routing weight distribution.

[0150] Specifically, the system maintains a copy of the intermediate feature representation X in memory and creates a separate computation task (or thread) for each low-rank adapted expert network in the expert network subset. Each computation task receives the same input X but invokes its corresponding expert network's private parameters and computation graph. In deep learning frameworks that support parallel computation (such as PyTorch or TensorFlow), this can be achieved through simple function call loops or by leveraging the framework's vectorized computation features. For example, suppose the dimension of the intermediate feature representation X is... (128 feature vectors, each 1024-dimensional), the expert subset activated at the current feature location is { The system will prepare two data references for X and start two calculation processes simultaneously (or nearly simultaneously): Process one inputs X... Perform the calculation; in process two, input X. Perform the calculation.

[0151] A linear transformation is performed on the intermediate feature representations distributed to each low-rank adaptive expert network using the low-rank parameter matrix in each low-rank adaptive expert network.

[0152] Among them, the low-rank parameter matrix is ​​the core trainable parameter of the low-rank adaptive expert network, which is usually composed of a pair of matrices A and B, satisfying the following condition: The rank is much smaller than the original feature dimension D. This design enables efficient transformation of high-dimensional features with very few parameters. Linear transformation here specifically refers to linear mapping operations implemented through matrix multiplication.

[0153] Specifically, each low-rank adapting expert network e internally maintains a pair of parameters: a matrix sum matrix Where r is the preset low-rank dimension (e.g., 8), and D is the input feature dimension (e.g., 1024). For the input feature vector x (D-dimensional), the linear transformation of this expert network is calculated as follows: The entire calculation is equivalent to using a low-rank matrix. Transform x, but the number of parameters is only far fewer than full-rank matrices One parameter. A fixed scaling factor is usually also introduced. (For example (or 0.5), the final linear output is For example, for Its parameters The scaling factor α = 0.5. For the input x (1024 dimensions), first calculate... This yields an 8-dimensional intermediate vector. Then, the calculation... This results in a 1024-dimensional vector. Finally, we obtain... .

[0154] Perform a nonlinear transformation on the result of the linear transformation to generate a characteristic transformation output.

[0155] Nonlinear transformation refers to applying a nonlinear activation function to the output of a linear transformation. The purpose is to introduce nonlinear expressive power into the model, enabling the network to learn and fit more complex feature patterns. Commonly used activation functions include ReLU and GELU. The feature transformation output is the final output vector obtained after low-rank linear transformation and nonlinear activation; it represents the "knowledge increment" extracted from the original input features by the expert network.

[0156] Specifically, the system scales the linear output. Apply a non-linear activation function element by element ,Right now The choice of activation function σ depends on the model design, for example, the GELU function: . This refers to the feature transformation output generated by the low-rank adapting expert network for the current input x. For example, continuing from the previous example, for... Calculated (A 1024-dimensional vector) Element-wise apply the GELU activation function. Assume... If the first element has a value of 1.2, then after calculation using the GELU function, a new value (approximately 1.12) will be output. Performing this operation on all 1024 elements results in a 1024-dimensional vector. This is The generated feature transformation output. Perform the same process to obtain .

[0157] Therefore, according to the above implementation method, the system can utilize a parameter-efficient parallel computing architecture to simultaneously mobilize multiple lightweight expert networks with different knowledge focuses to process the same set of intermediate features. Each expert achieves fast and resource-saving linear feature mapping through its low-rank parameter matrix, and then introduces the necessary complexity through nonlinear activation, ultimately generating diverse feature enhancement schemes. This provides rich and high-quality alternative materials for subsequent refined knowledge fusion based on routing weights, and is a key computational link that enables the entire dynamic routing and feature fusion mechanism to operate efficiently and flexibly.

[0158] In some embodiments, a weighted fusion calculation is performed on multiple feature transformation outputs based on dynamic routing weights corresponding to subsets of the expert network to generate a feature fusion result, including: Obtain the dynamic routing weights corresponding to each low-rank adaptive expert network in the expert network subset.

[0159] The corresponding dynamic routing weights refer to the specific weight values ​​allocated to each low-rank adaptive expert network within the current subset of expert networks in the dynamic routing weight distribution. These weight values ​​are derived from the results calculated in previous steps and after Top-k filtering and renormalization, directly reflecting the contribution of each activated expert to generating the current feature fusion result.

[0160] Specifically, the system reads k weight values, each corresponding to a subset of the activated expert networks (e.g., containing k experts), from the data structure cached in the preceding steps. These weight values ​​are typically extracted from the original dynamic routing weight distribution and normalized to ensure that the sum of the weights is 1. For example, for a given feature location, its activated expert network subset is { }, read the corresponding renormalized weight value from the route weight cache. This means that in current converged computing, The contribution weight is 0.806. Its contribution weight is 0.194.

[0161] Each dynamic routing weight is treated as a scalar and multiplied with the feature transformation output generated by the low-rank adaptation expert network corresponding to that weight to obtain multiple weighted feature outputs.

[0162] Here, a scalar refers to a single numerical value, as opposed to a vector or matrix. In this step, each dynamic routing weight is a scalar. The multiplication operation specifically refers to scalar-vector multiplication, which involves multiplying a scalar by each element of a vector. The weighted feature output refers to the scaled feature vector obtained by multiplying the transformed feature output generated by an expert network by its corresponding dynamic routing weight.

[0163] Specifically, the system processes each expert in the expert network subset sequentially. For the j-th expert in the subset, its corresponding dynamic routing weight scalar is retrieved. and the feature transformation output vector generated by the expert. (Dimension is D). Perform element-wise multiplication: Calculation result This represents the weighted feature output contributed by that expert. This process is repeated for all k experts to obtain k weighted feature output vectors. For example, continuing from the previous example, suppose... Generated feature transformation output It is a 1024-dimensional vector. , generated It is another 1024-dimensional vector. The weight 0.806 is compared with... Multiplication: The weight 0.194 is compared with... Multiplication: Thus, we obtain two weighted feature outputs. and .

[0164] Perform a summation operation on multiple weighted feature outputs to generate a feature fusion result.

[0165] In this context, summation specifically refers to vector addition, which involves adding multiple vectors of the same dimension element-wise to obtain a new vector. The feature fusion result refers to the total feature increment vector obtained by performing vector addition on the k weighted feature outputs. It integrates the weighted knowledge contributions of all activated expert networks.

[0166] Specifically, the system outputs a vector of all k weighted features. Perform element-by-element addition. The formula is: The resulting Δ is a vector with the same dimension (D-dimensional) as the input features, representing the feature fusion result at the current feature location. For example, adding the two weighted feature outputs calculated above: This 1024-dimensional vector Δ is the final feature fusion result; it fuses... and The transformation features were weighted according to their respective importance (0.806 and 0.194).

[0167] Therefore, according to the above implementation method, the system can perform fine-grained weighted fusion of feature transformations generated by multiple expert networks based on quantified routing importance. Through a series of linear but crucial operations such as scalar scaling and vector summation, the system integrates the feature knowledge emphasized by different experts into a unified, information-rich feature increment in a controllable proportion. This ensures that the "knowledge" ultimately injected into the backbone features is arbitrated and balanced, rather than simply spliced ​​together, thus laying a solid computational foundation for generating high-quality, highly consistent fused feature representations.

[0168] In some embodiments, feature extraction processing is performed on multimodal inspection data to generate multimodal feature representations, including: Image preprocessing is performed on the images of substation equipment in the multimodal inspection data.

[0169] Image preprocessing refers to a series of standardization and enhancement operations performed on the original substation equipment images before they are input into the deep learning model. The aim is to eliminate non-task-related differences in the original data (such as size and illumination) and convert the data into the input format expected by the model, so as to improve the robustness and processing efficiency of the model.

[0170] Specifically, the system performs the following operations through an image processing pipeline: First, image resizing, scaling input images of different resolutions to a uniform fixed size (e.g., 224 pixels high × 224 pixels wide); second, pixel value normalization, linearly scaling and converting the image's pixel values ​​from the original integer range of 0 to 255 to a floating-point range consistent with the model input (e.g., ...). or Finally, optional data augmentation, such as random horizontal flipping or color dithering, can increase data diversity during the training phase. For example, the system receives an image of a transformer bushing with a resolution of 1920 pixels × 1080 pixels, taken by an inspection robot. The preprocessing module first scales it down to 224 pixels × 224 pixels using bilinear interpolation. Then, it normalizes the RGB (red, green, blue) value of each pixel by dividing it by 255.0. The interval. The final output is a single dimension. The tensor (3 color channels, 224 pixels high, 224 pixels wide).

[0171] The preprocessed image is segmented and feature mapped by a visual encoder to generate an initial visual feature sequence.

[0172] In this embodiment, the visual encoder specifically refers to the initial module in the visual transformer model responsible for converting a two-dimensional image into a sequence of one-dimensional vectors. Image segmentation is the process of cutting the entire image into multiple equally sized square patches. Feature mapping refers to mapping each image patch to a high-dimensional feature vector through a trainable linear projection layer. The initial visual feature sequence is the sequence formed by arranging the feature vectors of all image patches in spatial order.

[0173] Specifically, the system will preprocess the image (dimensions) The image is divided into N image blocks (where C is the number of channels, H is the height, and W is the width), with each block being P × P pixels in size. N equals... Then each image patch is flattened into a block of length [length missing]. The vector. Then, each flattened vector is mapped to a linear projection layer (fully connected layer) through a linear projection layer. An embedding space of N dimensions. Ultimately, this results in an embedding space containing N dimensions. The sequence of dimensional vectors is the initial visual feature sequence. A learnable classification word vector is typically added before this sequence. For example, suppose the input image... The image patch size is P=16. Therefore, the image is divided into... Each image block is flattened to... A dimensional vector. Linear projection layer weight matrix. ,set up Each 768-dimensional block vector is projected into a 768-dimensional embedding vector. Finally, a sequence of 196+1=197 vectors is generated (197 including 196 image block embeddings and 1 classification term embedding), each vector being 768-dimensional, forming the initial visual feature sequence.

[0174] Apply rotational position encoding to the initial visual feature sequence.

[0175] Rotational position encoding is a relative position encoding method that transforms each pair of feature dimensions in the feature vector using a rotation matrix, encoding the relative position information of elements in the sequence into the angle of the feature vector, thus enabling the model to perceive the sequential relationship of the sequence elements. Compared to absolute position encoding, rotational position encoding can better handle long sequences and is helpful for model extrapolation.

[0176] Specifically, the system calculates a rotation angle for each position m in the sequence. For the m-th position in the initial visual feature sequence 3D feature vector regard it as There are 363 complex pairs. For each complex pair... Where i is the dimension index, and an index is applied based on position m and frequency. The determined rotation matrix is ​​then transformed. The transformation formula can be simplified to: ,in This process incorporates positional information into the features. For example, for the sequence of 197 768-dimensional vectors mentioned above, the rotation position encoding module calculates the rotation angle for each position (0 to 196). For the feature vector at position m=10, each pair of consecutive dimensions in its 768 dimensions (such as the 0th and 1st dimensions, the 2nd and 3rd dimensions, ...) is rotated according to a preset rotation frequency. With position m=10, perform the above rotation operation to generate a new feature vector that incorporates information about the absolute position 10.

[0177] The encoded feature sequence is input into a visual transformer for deep feature extraction, and the extracted visual features are mapped to a preset hidden dimension through a projection network to obtain an image feature vector.

[0178] The visual transformer is a deep learning model based on a pure Transformer encoder architecture. It performs deep contextual semantic modeling and feature extraction on visual feature sequences with positional encoding through multi-layer self-attention mechanisms and feedforward neural networks. The projection network is a small neural network (typically a linear or multi-layer perceptron) used to scale the visual features extracted by the visual transformer. Transformed to the dimension of text features (Or, the hidden dimension D of the backbone network) is a pre-defined hidden dimension D, which is consistent with the hidden dimension D, so as to facilitate subsequent multimodal feature alignment and fusion. The image feature vector specifically refers to the sequence of visual feature representations with a unified dimension D obtained after extraction by the visual transformer and mapping by the projection network.

[0179] Specifically, the system inputs a feature sequence with applied rotational position encoding into a... A visual transformer using a layered Transformer encoder. Each layer of the encoder contains a multi-head self-attention mechanism and a feedforward neural network. After layer computation, the output vector corresponding to the classification term is usually taken as the global representation of the entire image, and its dimension is... Then, this vector is fed into the projection network. The projection network is typically a linear layer: ,calculate The result It is a D-dimensional vector, that is, the image feature vector. In some designs, the average or specific pooling result of all image patch features may also be used. For example, suppose the visual transformer contains a 12-layer encoder ( The input consists of 197 vectors of 768 dimensions. After 12 layers of computation, the output vector at the position of the categorized word (index 0) is extracted, with a dimension of 768. The preset hidden dimension D=1024. Projection network weights. .calculate This results in a 1024-dimensional vector, which is the image feature vector. The system combines this vector with the subsequent text feature vector to form a multimodal feature representation.

[0180] Therefore, according to the above implementation method, the system can standardize the original images of substation equipment and efficiently and accurately extract visual features containing deep semantic information using a visual transformer model based on rotational position encoding. Finally, through dimensional projection alignment, the visual features are mapped to a feature space unified with the text modality, providing high-quality, alignable visual input for subsequent hybrid expert multimodal large models. This is a crucial and solid visual perception foundation in the multimodal intelligent diagnostic process.

[0181] According to a second aspect, the present invention provides a fault diagnosis system for substation equipment based on a large model, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements any fault diagnosis method for substation equipment based on a large model in the embodiments of the present invention.

[0182] The specific functions and examples of each module and submodule of the device in this embodiment of the invention can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0183] According to embodiments of the present invention, the above-described method of the present invention can be applied to an electronic device and a readable storage medium.

[0184] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0185] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0186] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0187] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as a large-model-based substation fault diagnosis method. For example, in some embodiments, a large-model-based substation fault diagnosis method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the large-model-based substation fault diagnosis method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured, by any other suitable means (e.g., by means of firmware), to perform a large-model-based substation fault diagnosis method.

[0188] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0189] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0190] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0191] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0192] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0193] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0194] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0195] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this invention should be included within the scope of protection of this invention.< / bos> < / bos> < / bos> < / eos> < / bos> < / eos> < / bos>

Claims

1. A fault diagnosis method for substation equipment based on a large model, characterized in that, include: Perform feature extraction processing on multimodal inspection data to generate multimodal feature representations; The multimodal feature representation is input into a preset hybrid expert multimodal large model, which includes a backbone network, a dynamic routing gating network, and multiple low-rank adaptive expert networks. The backbone network performs forward computation on the multimodal feature representation to obtain intermediate feature representation, and the dynamic routing weight distribution is calculated based on the intermediate feature representation through the dynamic routing gating network. Based on the dynamic routing weight distribution, the corresponding low-rank adaptation expert network is activated from the plurality of low-rank adaptation expert networks to perform feature fusion on the intermediate feature representation and generate a fused feature representation; Based on the fused feature representation, fault classification results and maintenance suggestion text are generated simultaneously.

2. The method according to claim 1, characterized in that, The step of performing forward computation on the multimodal feature representation through the backbone network to obtain intermediate feature representations includes: Obtain the multimodal feature representation that includes image feature vectors and text feature vectors; Perform sequence concatenation on the image feature vector and the text feature vector to generate a unified input sequence; The unified input sequence is input into the backbone network, and cross-modal feature interaction and context encoding are performed through an attention mechanism to output the encoded sequence; The intermediate feature representation is extracted from the encoded sequence.

3. The method according to claim 2, characterized in that, The step of calculating the dynamic routing weight distribution based on the intermediate feature representation through the dynamic routing gating network includes: The intermediate feature representation is normalized to obtain a normalized intermediate feature representation used as input to the dynamic routing gating network; The routing weight calculation layer configured by the dynamic routing gating network performs linear projection on the normalized intermediate feature representation to generate a routing score vector. The routing score vector is normalized to output the dynamic routing weight distribution; The dimension of the routing scoring vector is associated with the multiple low-rank adaptation expert networks.

4. The method according to claim 3, characterized in that, The step of activating a corresponding low-rank adaptation expert network from among the multiple low-rank adaptation expert networks based on the dynamic routing weight distribution, and performing feature fusion on the intermediate feature representation to generate a fused feature representation, includes: Based on the dynamic routing weight distribution, a subset of activated expert networks is selected from the plurality of low-rank adaptation expert networks; Through each low-rank adapting expert network in the aforementioned expert network subset, low-rank linear transformations are performed in parallel on the intermediate feature representations to obtain multiple feature transformation outputs; Based on the dynamic routing weights corresponding to the subset of the expert network, a weighted fusion calculation is performed on the multiple feature transformation outputs to generate a feature fusion result. The feature fusion result is superimposed with the intermediate feature representation to output the fused feature representation.

5. The method according to claim 4, characterized in that, The process of simultaneously generating fault classification results and maintenance suggestion text based on the fused feature representation includes: The fused feature representation is then globally pooled to obtain a classification feature vector; The classification feature vector is input into a classifier for processing, the class probability distribution is output, and the classification label is determined to obtain the fault classification result. The fused feature representation is input into the text decoder, and a word sequence is generated in an autoregressive manner to obtain the maintenance suggestion text.

6. The method according to claim 2, characterized in that, The step of performing cross-modal feature interaction and context encoding through an attention mechanism to output the encoded sequence includes: The unified input sequence is subjected to layer normalization. Based on the normalized unified input sequence, the query vector, key vector, and value vector are calculated through multi-head attention, and attention weighting is performed. Based on the preset attention mask, set the visibility control amount of vectors at different positions in the sequence; The results of attention-weighted computation are sequentially processed with residual connections, layer normalization, and feedforward network transformation. The process iteratively executes multiple processing steps, including attention weighting calculation, residual connection, layer normalization, and feedforward network transformation, to output the encoded sequence.

7. The method according to claim 4, characterized in that, The intermediate feature representation is subjected to parallel low-rank linear transformations through each low-rank adapting expert network in the subset of the expert network, resulting in multiple feature transformation outputs, including: The intermediate feature representations are distributed in parallel to each low-rank adapting expert network in the subset of expert networks; A linear transformation is performed on the intermediate feature representations distributed to the network using the low-rank parameter matrices in each low-rank adaptation expert network. A nonlinear transformation is performed on the result of the linear transformation to generate the feature transformation output.

8. The method according to claim 7, characterized in that, The weighted fusion calculation is performed on the multiple feature transformation outputs based on the dynamic routing weights corresponding to the subset of the expert network to generate a feature fusion result, including: Obtain the dynamic routing weights corresponding to each low-rank adaptive expert network in the expert network subset; Each dynamic routing weight is treated as a scalar and multiplied with the feature transformation output generated by the low-rank adaptation expert network corresponding to that weight to obtain multiple weighted feature outputs. A summation operation is performed on the multiple weighted feature outputs to generate the feature fusion result.

9. The method according to claim 1, characterized in that, The step of performing feature extraction processing on the multimodal inspection data to generate multimodal feature representations includes: Image preprocessing is performed on the substation equipment images in the multimodal inspection data; The preprocessed image is segmented and feature-mapped using a visual encoder to generate an initial visual feature sequence. Apply rotational position encoding to the initial visual feature sequence; The encoded feature sequence is input into a visual transformer for deep feature extraction, and the extracted visual features are mapped to a preset hidden dimension through a projection network to obtain an image feature vector.

10. A fault diagnosis system for substation equipment based on a large model, characterized in that, It includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the method as described in any one of claims 1-9.

Images