Photovoltaic equipment fault diagnosis method, system, equipment and medium

Abstract

The invention relates to the technical field of power systems, and discloses a photovoltaic equipment fault diagnosis method, system and equipment and a medium. The method comprises the steps of obtaining a structured multi-mode fault data set based on a disclosed photovoltaic data set and collected multi-source heterogeneous data generated in the operation process of photovoltaic equipment; selecting an initial multi-modal large language model, and constructing a feature-oriented dynamic routing architecture according to a pre-training weight matrix of the initial multi-modal large language model; based on the structured multi-modal fault data set, utilizing a pre-constructed joint training loss function to carry out fine tuning training on the initial multi-modal large language model to obtain a target multi-modal large language model; and inputting the real-time operation data of the photovoltaic equipment into the target multi-mode large language model for reasoning to obtain a fault diagnosis result of the photovoltaic equipment. According to the invention, efficient and accurate diagnosis of the photovoltaic equipment fault is realized.

Description

Technical Field

[0001] This invention relates to the field of power system technology, and in particular to a method, system, equipment and medium for diagnosing photovoltaic equipment faults. Background Technology

[0002] As a crucial component of distributed energy supply in modern power systems, the reliable operation of photovoltaic (PV) equipment directly determines the efficiency and security of power generation. However, PV equipment is constantly exposed to the complex and dynamic outdoor environment, enduring external interference such as irradiance fluctuations, temperature variations, dust accumulation, and partial shading. This leads to frequent faults, including hot spots on modules, damage to inverter power modules, loose wiring in combiner boxes, and aging of transformer insulation. Such equipment failures not only reduce the output of individual units and increase maintenance costs but can also trigger chain reactions, leading to regional power outages and even fire risks, severely impacting the stable operation of the power system.

[0003] Traditional photovoltaic equipment fault diagnosis methods mainly rely on regular manual inspections and rule-based judgments based on expert experience. While these methods can identify common problems to some extent, they are insufficient to meet the real-time, stability, and consistency requirements of large-scale photovoltaic equipment clusters for diagnostic tasks.

[0004] In recent years, Multimodal Large Language Models (MLLM) have made groundbreaking progress in the field of visual-text cross-modal joint understanding, providing a new approach to solving the problem of collaborative diagnosis of multimodal data in photovoltaic equipment. MLLM possesses powerful cross-modal information fusion and general reasoning capabilities, enabling it to complete visual perception of photovoltaic equipment status, interpretation of fault areas in thermal imaging, correlation analysis of sensor data and text records, and generation of expert-level natural language diagnostic reports within a unified framework. Compared to traditional rule-based diagnostic systems, MLLM has stronger cross-scenario generalization capabilities and can effectively identify subtle and hidden specific fault patterns in photovoltaic equipment, providing a more comprehensive analytical perspective for equipment fault diagnosis.

[0005] However, directly applying MLLM to photovoltaic equipment fault diagnosis still faces key technical bottlenecks: First, there is a significant distribution difference between the monitoring data of photovoltaic equipment and the general dataset on which MLLM pre-training relies. Existing fine-tuning methods such as Low-Rank Adaptation (LoRA) use a unified modality adaptation logic, which makes it difficult to accurately capture the unique statistical distribution and structural information of different types of photovoltaic equipment and different data modalities, resulting in insufficient diagnostic accuracy of the model for equipment-specific faults. Second, directly fine-tuning instructions based on limited equipment fault labeling data can easily lead to the model losing the general cross-modal reasoning ability learned in the pre-training stage while learning knowledge of specific photovoltaic equipment faults. Especially in noisy supervised data scenarios, the model's diagnostic stability for equipment faults further decreases. Summary of the Invention

[0006] In order to overcome the limitations of existing technologies in terms of heterogeneous data fusion, domain adaptability and model robustness, and thus improve the accuracy, robustness and interpretability of photovoltaic equipment fault diagnosis, this invention provides a photovoltaic equipment fault diagnosis method, system, equipment and medium.

[0007] In a first aspect, embodiments of the present invention provide a method for diagnosing photovoltaic equipment faults, including: Based on publicly available photovoltaic datasets and multi-source heterogeneous data collected during the operation of photovoltaic equipment, a structured multimodal fault dataset is obtained. An initial multimodal large language model is selected, and a feature-guided dynamic routing architecture is constructed based on the pre-trained weight matrix of the initial multimodal large language model. Based on the structured multimodal fault dataset, the initial multimodal large language model is fine-tuned using a pre-constructed joint training loss function to obtain the target multimodal large language model. The joint training loss function is constructed based on multimodal feature alignment loss and fault diagnosis result generation loss, and the loss optimization direction of the joint training loss function is determined based on the parameter update rules of the dynamic routing architecture. The real-time operating data of the photovoltaic equipment is input into the target multimodal large language model for reasoning to obtain the fault diagnosis results of the photovoltaic equipment.

[0008] Preferably, the structured multimodal fault dataset obtained based on publicly available photovoltaic datasets and multi-source heterogeneous data collected during the operation of photovoltaic equipment includes: Based on publicly available photovoltaic datasets, multi-source heterogeneous data generated during the operation of photovoltaic equipment are collected to obtain multimodal photovoltaic data; The multimodal photovoltaic data is transformed into an initial structured diagnostic result containing fault type, cause analysis, and maintenance recommendations through chain reasoning; The initial structured diagnostic results are reviewed and revised to obtain the target structured diagnostic results; The multimodal photovoltaic data and the target structured diagnostic results are correlated to obtain a structured multimodal fault dataset.

[0009] Preferably, the step of selecting an initial multimodal large language model and constructing a feature-guided dynamic routing architecture based on the pre-trained weight matrix of the initial multimodal large language model includes: A multimodal large language model with visual-text modality alignment and semantic reasoning capabilities is selected as the base model, and the backbone network parameters of the base model are fixed to obtain the initial multimodal large language model. The pre-trained weight matrix of the initial multimodal large language model is decomposed into a low-rank matrix to obtain a shared low-rank matrix, wherein the shared low-rank matrix is ​​configured to capture the common features of multimodal photovoltaic data; Based on the differences in photovoltaic data modes and the fault types of photovoltaic equipment, several adapters are constructed, wherein each adapter is configured to capture the photovoltaic characteristics of the corresponding mode or the fault characteristics of a specific type. A routing mechanism is introduced, comprising a feature encoder and a weight predictor. The feature encoder is configured to extract features from the input multimodal photovoltaic data to obtain a photovoltaic feature vector, and the weight predictor is configured to calculate the activation weight of each adapter by combining the photovoltaic feature vector. The shared low-rank matrix, each adapter, and the routing mechanism are integrated to form a dynamic routing architecture.

[0010] Preferably, the process of constructing the joint training loss function includes: We use cosine similarity calculation to compare the similarity between photovoltaic image features and photovoltaic text features, and construct a multimodal feature alignment loss. The difference between the generated fault diagnosis result labels and the actual fault diagnosis result labels is compared using the cross-entropy loss calculation method to construct the fault diagnosis result generation loss; The joint training loss function is obtained by weighted summation of the multimodal feature alignment loss and the fault diagnosis result generation loss.

[0011] Preferably, the step of fine-tuning the initial multimodal large language model based on the structured multimodal fault dataset using a pre-constructed joint training loss function to obtain the target multimodal large language model includes: The preprocessed structured multimodal fault dataset is divided into a training set and a validation set according to a preset ratio; The training set is input into the initial multimodal large language model in batches for cross-modal feature encoding to obtain a joint feature representation that integrates visual modality encoding and text modality semantic features; The joint feature representation is input into the language decoder of the initial multimodal large language model to generate a fault diagnosis prediction result, and the joint training loss function value is calculated based on the fault diagnosis prediction result; Based on the backpropagation algorithm, the trainable parameters in the dynamic routing architecture are updated by combining the joint training loss function value; After each preset training round, the fault diagnosis performance verification index of the initial multimodal large language model is calculated using the validation set. If the fault diagnosis performance verification index does not improve for several consecutive rounds, an early stopping mechanism is triggered, and the current model parameters are saved as the target multimodal large language model.

[0012] Preferably, updating the trainable parameters in the dynamic routing architecture based on the backpropagation algorithm and the joint training loss function value includes: Calculate the partial derivative of the joint training loss function value with respect to the trainable parameters in the dynamic routing architecture; A preset optimizer is used to iteratively update the trainable parameters based on the partial derivatives and the current training parameters.

[0013] Preferably, the step of inputting real-time operating data of the photovoltaic equipment into the target multimodal large language model for inference to obtain the fault diagnosis result of the photovoltaic equipment includes: Real-time operating data of photovoltaic equipment is collected, and the preprocessed real-time operating data of photovoltaic equipment is input into the target multimodal large language model for modality type recognition and cross-modal feature fusion to generate structured fault diagnosis results; The structured fault diagnosis results are format-validated to obtain photovoltaic equipment fault diagnosis results that include fault type, cause analysis, and maintenance suggestions.

[0014] Secondly, embodiments of the present invention provide a photovoltaic equipment fault diagnosis system, comprising: The dataset construction module is used to obtain a structured multimodal fault dataset based on publicly available photovoltaic datasets and multi-source heterogeneous data collected during the operation of photovoltaic equipment. The architecture design module is used to select an initial multimodal large language model and construct a feature-oriented dynamic routing architecture based on the pre-trained weight matrix of the initial multimodal large language model. The model training module is used to fine-tune the initial multimodal large language model based on the structured multimodal fault dataset and to obtain the target multimodal large language model using a pre-built joint training loss function. The joint training loss function is constructed based on the multimodal feature alignment loss and the fault diagnosis result generation loss, and the loss optimization direction of the joint training loss function is determined based on the parameter update rules of the dynamic routing architecture. The fault diagnosis module is used to input real-time operating data of photovoltaic equipment into the target multimodal large language model for reasoning to obtain fault diagnosis results of photovoltaic equipment.

[0015] Thirdly, embodiments of the present invention provide a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the photovoltaic equipment fault diagnosis method as described above.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the photovoltaic equipment fault diagnosis method as described above.

[0017] Compared with the prior art, the photovoltaic equipment fault diagnosis method, system, equipment, and medium of this invention have the following advantages at least one point: (1) At the model architecture level, the dynamic routing architecture built on the pre-trained weights of the initial multimodal large language model, through the two-layer adaptation design of "shared low-rank matrix + dedicated adapter", not only retains the cross-modal general reasoning ability of the pre-trained model by means of the shared low-rank matrix, but also achieves specific adaptation for different modal data and fault types of photovoltaics through the adapter. Compared with the traditional single low-rank adaptation scheme, it greatly improves the feature capture accuracy of multimodal heterogeneous data of photovoltaics. (2) At the model training level, the joint training loss function based on "multimodal feature alignment loss + fault diagnosis result generation loss" combined with the parameter update rules of the dynamic routing architecture can simultaneously achieve cross-modal semantic consistency enhancement and diagnostic result accuracy guarantee during fine-tuning. At the same time, by fixing the backbone network parameters and only updating the architecture trainable parameters, the model can effectively avoid losing its general cross-modal reasoning ability, take into account the domain adaptability and generalization ability of the model, and ensure that it can still output diagnostic results stably in unseen photovoltaic fault scenarios. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart of a photovoltaic equipment fault diagnosis method according to an embodiment of the present invention; Figure 2This is a schematic diagram of the structure of a photovoltaic equipment fault diagnosis system according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a terminal device according to an embodiment of the present invention; Figure label: 01. Dataset Construction Module; 02. Architecture Design Module; 03. Model Training Module; 04. Fault Diagnosis Module; 5000. Terminal Device; 5001. Processor; 5002. Bus; 5003. Memory; 5004. Transceiver. Detailed Implementation

[0019] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0020] In the description of this invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by those skilled in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] like Figure 1 The diagram shown is a flowchart illustrating a photovoltaic equipment fault diagnosis method according to an embodiment of the present invention. (Refer to...) Figure 1 An embodiment of the present invention provides a method for diagnosing photovoltaic equipment faults, comprising the following steps: S1. Based on publicly available photovoltaic datasets and multi-source heterogeneous data collected during the operation of photovoltaic equipment, a structured multimodal fault dataset is obtained. In photovoltaic equipment fault diagnosis, existing data support has significant limitations: On the one hand, photovoltaic equipment monitoring data covers multiple modalities such as visible light, infrared, thermal imaging, and sensor logs. The data structures and feature expressions of different modalities vary significantly, and the data heterogeneity is strong due to the influence of sensor accuracy and environmental conditions. Traditional single-modal or simply fused datasets are difficult to fully cover the fault feature space. On the other hand, existing data annotations mostly focus only on basic information such as fault category or spatial location, lacking structured content such as fault cause analysis and maintenance suggestions. This makes it impossible to support the generation of automated and refined fault diagnosis reports. At the same time, purely manual annotation is costly and inefficient, making it difficult to meet the needs of large-scale dataset construction.

[0022] To overcome the aforementioned data bottlenecks and ensure the effectiveness of subsequent model training and the practicality of diagnostic results, it is necessary to construct a dedicated dataset through multi-source data integration and structured processing.

[0023] Specifically, step S1 includes: 11) Based on the publicly available photovoltaic datasets, supplement the data by collecting multi-source heterogeneous data generated during the operation of photovoltaic equipment to obtain multimodal photovoltaic data; This embodiment selects the publicly available photovoltaic defect dataset PVEL-AD as its foundation. This dataset contains a massive amount of near-infrared images of photovoltaic cells and basic defect annotations, but it suffers from limitations in modality and equipment type, being restricted to cells. To overcome these shortcomings, multi-source heterogeneous data from the actual operation of photovoltaic equipment is supplemented, including RGB images and infrared thermal imaging data of key equipment such as photovoltaic panels, transformers, and combiner boxes, as well as sensor logs and inspection text records of equipment operation status. Through data format standardization processing, the basic dataset and supplementary data are integrated to form multi-modal photovoltaic data covering multiple equipment types and multi-modal information.

[0024] 12) Transform multimodal photovoltaic data into initial structured diagnostic results containing fault types, cause analysis, and maintenance recommendations through chained reasoning; A large language model with chain-like reasoning capabilities is employed to generate structured diagnostic results based on the characteristics of multimodal photovoltaic data. For visual modal data (RGB, infrared, thermal imaging), fault-related visual features (such as abnormal temperature areas and structural damage textures) are extracted. For textual data such as sensor logs, the fluctuation patterns of electrical parameters are analyzed. Through preset domain-oriented prompt templates, the model is guided to simulate expert reasoning logic and automatically generate initial structured diagnostic results containing fault type determination, causal mechanism analysis, and operable maintenance suggestions, thus realizing the transformation of multimodal data into structured text annotation.

[0025] Specifically, this embodiment selects ChatGPT-4o as the core model for chained reasoning and designs a domain-oriented prompt template that includes multimodal feature association logic. The template explicitly embeds fault feature descriptions of visual modalities (such as temperature gradients in infrared data and structural damage locations in RGB images) and parameter fluctuation thresholds of textual data (such as voltage anomaly ranges in sensor logs). This guides the model to first determine the fault type based on features, then deduce the cause by combining the operating principles of photovoltaic equipment, and finally output operable maintenance suggestions based on operation and maintenance specifications. Through the above step-by-step reasoning, an initial structured diagnostic result with a unified format and complete content is automatically output.

[0026] 13) Review and revise the initial structured diagnostic results to obtain the target structured diagnostic results; To ensure annotation quality, a two-stage review mechanism is introduced. Specifically, in the first stage, the ChatGPT-4o large language model performs self-verification on the initial structured diagnostic results, correcting obvious terminological errors, logical contradictions, and content that does not match data characteristics. In the second stage, a team of engineers with extensive experience in photovoltaic operation and maintenance conducts manual review of the self-verified results, focusing on verifying the scientific validity of the fault cause analysis, the feasibility of the maintenance recommendations, and the consistency between the fault type and multimodal data characteristics. Any deviations are revised and improved, ultimately forming accurate and reliable target structured diagnostic results.

[0027] 14) Correlate the multimodal photovoltaic data and the target structured diagnostic results to obtain a structured multimodal fault dataset.

[0028] Based on unique sample identifiers (such as device number + data acquisition timestamp), a one-to-one correspondence is established between multimodal photovoltaic data and target structured diagnostic results. Multimodal information such as RGB images, infrared data, and sensor logs collected by the same device at the same time are bound to the corresponding target structured diagnostic results and organized according to a preset data format. This ultimately forms a fault dataset that combines the completeness of multimodal features with structured annotation. The preset data format refers to a unified file naming rule based on unique sample identifiers, a directory structure for storing multimodal data categorized by modality type, and a storage specification for structured diagnostic results containing fixed fields such as fault type, cause analysis, and maintenance recommendations.

[0029] S2. Select an initial multimodal large language model and construct a feature-guided dynamic routing architecture based on the pre-trained weight matrix of the initial multimodal large language model. While existing multimodal large language models possess basic visual-text modality alignment and semantic reasoning capabilities, they face significant bottlenecks when directly applied to photovoltaic equipment fault diagnosis scenarios. On the one hand, photovoltaic multimodal data exhibits strong heterogeneity, encompassing differences in feature representations of visual modalities such as visible light and infrared thermal imaging, as well as differences in parameter structures of text modalities such as sensor logs. Furthermore, environmental interference (such as temperature drift in infrared sensors caused by high temperatures and blurred details in visible light images due to dust coverage) further exacerbates the inconsistency of modal features. On the other hand, photovoltaic equipment is diverse (modules, inverters, combiner boxes, etc.), and the fault manifestations and feature distributions of different devices vary significantly. Traditional single low-rank adaptation schemes struggle to achieve differentiated adaptation, easily leading to the submergence of high-value, specific fault features. Additionally, cross-modal general knowledge may be forgotten during fine-tuning, affecting the model's robustness to unseen fault scenarios.

[0030] To address the aforementioned issues, this step involves selecting a suitable base model and fixing its backbone network parameters to construct a dynamic routing architecture that integrates common feature capture and specific adaptation, thereby achieving accurate adaptation and efficient inference for photovoltaic multimodal heterogeneous data.

[0031] Specifically, step S2 includes: 21) Select a multimodal large language model with visual-text modality alignment and semantic reasoning capabilities as the base model, and fix the backbone network parameters of the base model to obtain the initial multimodal large language model; Considering the requirements of photovoltaic fault diagnosis tasks for cross-modal alignment accuracy, text generation completeness, and inference efficiency, a multimodal large language model that performs well in visual-text joint understanding and semantic generation tasks is selected as the basic model. This model has been pre-trained on large-scale general data and has the core ability to deeply associate image features with text semantics.

[0032] To adapt to the characteristics of photovoltaic data, a dedicated modal preprocessing submodule is added to the basic model: It performs unified resolution normalization and core region segmentation on visual data such as visible light, infrared, and thermal imaging to eliminate background interference; and it performs domain terminology mapping on text data such as sensor logs and inspection records (converting professional terms such as "module hotspots" and "inverter overvoltage" into semantic identifiers that the model can recognize). Its mathematical expression is as follows: in, This represents the text modality preprocessing submodule. This represents a semantic embedding function. Represents a domain terminology mapping function. This refers to the raw data from sensor logs or inspection texts generated during the operation of photovoltaic equipment.

[0033] After initialization, the parameters of the Transformer backbone network of the base model are fixed, and only the adjustable space of the modal input layer and the feature mapping layer is retained to avoid the loss of general cross-modal knowledge during fine-tuning, thus obtaining the initial multimodal large language model.

[0034] 22) Perform low-rank decomposition on the pre-trained weight matrix of the initial multimodal large language model to obtain a shared low-rank matrix; The shared low-rank matrix refers to the low-dimensional parameter matrix pair obtained by performing low-rank decomposition on the pre-trained weight matrix of the initial multimodal large language model. It is configured to capture the common features of multimodal photovoltaic data. Its core function is to capture the general features of multimodal photovoltaic data (such as the common semantic structure of faults in different modal data and the basic representation logic of equipment operating status), while retaining the cross-modal general reasoning ability of the basic model, providing a foundation for subsequent specific adaptation.

[0035] Singular value decomposition is used to optimize the pre-trained weight matrix of the initial multimodal large language model. ( Perform low-rank decomposition on the model's hidden layer dimension, retaining the low-dimensional parameter matrices corresponding to the core singular values, to obtain a shared low-rank matrix pair. ,in , , For low-rank rank ( The mechanism of action of this shared low-rank matrix is ​​defined as follows: in, This represents the latent vector after fusing common features. This represents the original latent vector output by the base model.

[0036] 23) Based on the differences in photovoltaic data modes and the types of photovoltaic equipment faults, construct several adapters; An adapter refers to a lightweight feature extraction module that adopts a low-rank matrix pair structure design and is specifically adapted to specific modal data or specific fault types in the photovoltaic field. Each adapter is configured to capture the corresponding modal photovoltaic features or specific type of fault features.

[0037] Based on the modal heterogeneity of photovoltaic data (visual modalities: RGB, infrared, thermal imaging; textual modalities: sensor logs, inspection texts) and the diversity of fault types (module-related: hot spots, cracks, shading; equipment-related: inverter overvoltage, combiner box poor contact, transformer abnormal noise), a system is constructed. There are 1 adapter, each designed using a low-rank matrix pair structure, mathematically expressed as: in, Indicates the first The latent vectors output after processing by each adapter. , For the first The low-rank parameter matrix of each adapter.

[0038] In the specific design, the adapters are divided into two dimensions: "modal-fault". For example, the "infrared mode-hot spot fault" adapter focuses on capturing the features of abnormal temperature areas, while the "sensor log-inverter fault" adapter focuses on extracting the features of electrical parameter fluctuations, ensuring that each adapter is accurately adapted to the specific features of the corresponding scenario.

[0039] 24) Introduce a routing mechanism that includes a feature encoder and a weight predictor; The routing mechanism refers to a decision-making mechanism that dynamically calculates the activation weight of each adapter and assigns adaptation priority based on the characteristic attributes of the input multimodal photovoltaic data.

[0040] The feature encoder is configured to extract features from the input multimodal photovoltaic data to obtain photovoltaic feature vectors. It employs a two-layer fully connected network (FCN) + LayerNorm structure to fuse and compress the features of the input multimodal photovoltaic data after initial model encoding, outputting a dimension of [missing value]. photovoltaic eigenvectors Its expression is: in, This represents the visual modality preprocessing submodule. Represents a set of visual modal data. This represents a set of text modal data. A photovoltaic feature vector is a low-dimensional vector output by a feature encoder. It integrates the core features of both visual and text modalities, comprehensively representing the modal attributes and fault-related information of the current input data, providing a core basis for adapter activation weight calculation.

[0041] The weight predictor is configured to calculate the activation weights for each adapter by combining the photovoltaic feature vectors, using the softmax activation function based on the photovoltaic feature vectors. The activation weights of each adapter are calculated, and the mathematical expression is as follows: in, Indicates the first The activation weight of each adapter, Represents the routing weight matrix. Denotes the bias term, and satisfies: .

[0042] This routing mechanism can dynamically allocate adapter contribution based on the modal type and fault characteristic intensity of the input data, thereby achieving differentiated adaptation of the input driver.

[0043] 25) Integrate the shared low-rank matrix, each adapter, and the routing mechanism to form a dynamic routing architecture.

[0044] By sharing low-rank matrices For the initial model latent vectors Perform common feature injection to obtain Activation weights output based on routing mechanism For the output of each adapter Weighted summation is performed to obtain the specific feature enhancement result; finally, feature fusion is used to obtain the architecture output latent vector, forming a complete feature-guided dynamic routing architecture, the mathematical expression of which is: in, This indicates that the dynamic routing architecture outputs the latent vector. This indicates the results of enhanced specific features. This integrated design enables collaborative modeling of general cross-modal knowledge and photovoltaic-specific knowledge, ensuring both the model's generalization ability and enhancing its adaptation accuracy to heterogeneous photovoltaic data.

[0045] It should be noted that the core parameter update carrier of the dynamic routing architecture is a low-rank update matrix. Essentially, it is a lightweight parameter set that integrates shared low-rank parameters and low-rank parameters of the dynamic weighted adapter. This achieves both accurate adaptation to the initial multimodal large language model and ensures efficient parameter fine-tuning through the low-rank structure. The construction logic of this low-rank update matrix is ​​deeply coupled with the dynamic routing architecture, specifically manifested as follows: With shared low-rank matrix pairs Provides a low-rank parameter base for general cross-modal knowledge, using the low-rank matrices of each adapter. Low-rank parameters provide modality / fault-specific knowledge supplementation, and the activation weights are ultimately output through the routing mechanism. The low-rank parameters of the adapter are dynamically weighted to form a low-rank update matrix that precisely matches the characteristics of the input data. Its mathematical expression can be derived as follows: in, Represents the low-rank update matrix. This indicates a shared low-rank parameter contribution. This represents the contribution of the low-rank parameters of the dynamically weighted adapter.

[0046] This low-rank update matrix has three core characteristics: First, it has lightweight parameters, because both the shared low-rank matrix and the adapter use low-rank decomposition. The parameter size is much smaller than the original model weight matrix, effectively reducing the computational overhead and storage cost during the fine-tuning stage; secondly, it offers dynamic adaptability, based on the activation weight allocation logic of the routing mechanism. The contribution ratio of each low-rank parameter can be dynamically adjusted according to the modal type and fault feature intensity of the input data, so as to realize differentiated parameter updates for photovoltaic multimodal heterogeneous data; thirdly, knowledge synergy, by sharing low-rank parameter terms, the cross-modal general reasoning ability of the initial model is retained, avoiding the loss of general knowledge during fine-tuning, while the photovoltaic domain-specific feature capture is strengthened by the adapter low-rank parameter terms, so as to realize the synergistic integration of general knowledge and domain knowledge.

[0047] The low-rank update matrix provides a targeted parameter update scheme for subsequent model fine-tuning, enabling the initial multimodal large language model to adapt to the photovoltaic fault diagnosis task through efficient parameter adjustment without reconstructing the backbone network. This lays the core parameter foundation for improving the model's diagnostic accuracy, generalization ability, and inference efficiency.

[0048] S3. Based on the structured multimodal fault dataset, the initial multimodal large language model is fine-tuned and trained using a pre-constructed joint training loss function to obtain the target multimodal large language model. During the initial fine-tuning training of the multimodal large language model, traditional single loss functions that focus solely on the quality of fault diagnosis result generation struggle to guarantee the semantic consistency between multimodal data, easily leading to a disconnect between visual features and textual diagnostic results. Loss functions that only focus on multimodal feature alignment fail to specifically optimize the core performance of the fault diagnosis task, resulting in insufficient diagnostic accuracy. Furthermore, the parameter updates in the dynamic routing architecture are specific. The parameters of the basic model's backbone network are fixed, with only the shared low-rank matrix, adapter, and routing mechanism participating in parameter iteration. If the loss function's optimization direction does not match this update rule, it can easily lead to chaotic parameter update logic: either the shared low-rank matrix becomes overly biased towards a certain type of modality feature, losing cross-modal general knowledge; or the adapter fails to accurately capture photovoltaic-specific features, resulting in decreased diagnostic accuracy.

[0049] Therefore, constructing a joint training loss function that takes into account both multimodal feature collaboration and diagnostic task objectives, and anchoring the parameter update rules of the dynamic routing architecture to clarify the optimization direction, becomes the key to improving the model fine-tuning effect.

[0050] Specifically, the joint training loss function is constructed based on the multimodal feature alignment loss and the fault diagnosis result generation loss, and the loss optimization direction of the joint training loss function is determined based on the parameter update rules of the dynamic routing architecture.

[0051] The following section details the construction process of the joint training loss function: 1) The cosine similarity calculation method is used to compare the similarity between photovoltaic image features and photovoltaic text features, and a multimodal feature alignment loss is constructed; Through the visual modality preprocessing submodule and encoding layer of the dynamic routing architecture, high-dimensional feature vectors of photovoltaic images, including visible light, infrared, and thermal imaging, are extracted. Its expression is: in, Represents the image feature extraction function. Represents a set of photovoltaic image data. Indicates shared low-rank matrix parameters. This indicates the adapter parameters.

[0052] The semantic feature vector of the target structured diagnostic result is extracted through the text modality preprocessing submodule and the encoding layer. Its expression is: in, This represents the text feature extraction function. This represents a collection of structured diagnostic texts.

[0053] Normalized cosine similarity is used to calculate the semantic correlation between two types of feature vectors. To achieve multimodal feature aggregation for similar samples and feature separation for dissimilar samples, a multimodal feature alignment loss function is constructed as follows: in, This represents the multimodal feature alignment loss function. Indicates the number of samples. express and For the same fault sample, Indicates the corresponding different fault samples ( ), It represents the semantic correlation between two types of feature vectors.

[0054] The optimization objective of this loss function is to minimize the negative logarithm of the feature similarity of samples of the same class and maximize the negative logarithm of the feature similarity of samples of different classes, so as to ensure that the photovoltaic image features and text diagnostic results maintain a high degree of consistency in a unified semantic space, and provide reliable cross-modal feature support for the generation of subsequent diagnostic results.

[0055] 2) The difference between the generated fault diagnosis result labels and the actual fault diagnosis result labels is compared using the cross-entropy loss calculation method to construct the fault diagnosis result generation loss; The target structured diagnostic results are tokenized and encoded to generate a true labeled sequence. : in, This represents the first text tag obtained after the actual structured diagnostic results are tokenized. This indicates the sequence length. The sequence contains markers for core fields such as fault type, cause analysis, and maintenance recommendations.

[0056] The model generates predicted tag sequences through a language decoder. : in, The first text tag is obtained by tokenizing the predicted structured diagnostic results generated by the language decoder through the representation model.

[0057] Cross-entropy loss is used to quantify the difference between predicted and true labels. The basic loss formula is: in, Indicates the basic loss. Represents the predicted probability distribution. The weighting coefficients are used to assign higher weights to key labels such as fault type and core cause, and to assign normal weights to auxiliary descriptive labels, ensuring that the model prioritizes learning high-value diagnostic information.

[0058] To meet the needs of photovoltaic diagnostic scenarios, supplementary structured constraints are added: the completeness and logical consistency of the generated results are verified; if core fields are missing or logical contradictions exist, a penalty term is added to the basic loss. : in, This represents a structural difference measurement function. Indicates the regulating factor. The function extracts the field hierarchy and logical relationships of the structured diagnostic results, ensuring that the generated report maintains a paragraph hierarchy and logical structure that conforms to industry standards. By analyzing the paragraph structure of real reports, it penalizes structural deviations in the generated text, guiding the model to generate structured reports that meet industry standards. Furthermore, to ensure the accuracy of photovoltaic terminology, a terminology matching penalty is introduced into the loss calculation. : in, This represents the weighting coefficient of the term matching penalty term. This indicates the total number of photovoltaic-related tags contained in the predicted tag sequence. A dictionary of terms in the photovoltaic field. The indicator function is represented. The final fault diagnosis result loss function is shown below: in, This represents the loss function used to generate the fault diagnosis results. The core objective of this loss function is to minimize the distributional difference between the predicted sequence and the true sequence, while ensuring the accuracy, completeness, and logical rationality of the diagnosis results.

[0059] 3) The multimodal feature alignment loss and the fault diagnosis result generation loss are weighted and summed to obtain the joint training loss function.

[0060] To balance the dual objectives of multimodal feature alignment and fault diagnosis generation, the two types of losses are weighted and summed to construct a joint training loss function, the mathematical expression of which is: in, The weight coefficients represent the multimodal feature alignment loss. The weighting coefficients represent the weighting coefficients used to generate the loss from the fault diagnosis results, and satisfy the following conditions: .

[0061] The weight coefficients employ a dynamic adjustment mechanism to adapt to different stages of model fine-tuning. Specifically, in the early stages of training (the first 20% of iterations), this embodiment focuses on constructing multimodal semantic associations and sets... , By using a high-weighted alignment loss to guide the optimization of the shared low-rank matrix and adapter parameters, stable semantic associations are ensured between image and text features. During the mid-training phase (20%-80% of iterations), the weights of the generation loss are gradually increased and adjusted to... , The model is guided to optimize the quality of diagnostic results based on feature alignment; in the later stages of training (the last 20% of iterations), an adaptive weight adjustment strategy is adopted, dynamically updating the weights based on the descent rate of the two types of losses. and ,in The dynamic update formula is shown below: in, , These represent the recent rate of decrease for the two types of losses, respectively. This represents the balance coefficient, ensuring that the two types of losses decrease in tandem.

[0062] Meanwhile, the optimization direction of the joint training loss function follows the parameter update rules of the dynamic routing architecture, performing gradient backpropagation only on the trainable parameters of the dynamic routing architecture to avoid interference with the backbone parameters of the base model. In other words, during gradient backpropagation, the loss gradient is only directed to the trainable parameters of the dynamic routing architecture, strictly shielding gradient propagation to the backbone network parameters of the base model. This prevents the pre-trained general cross-modal reasoning knowledge in the backbone network from being interfered with by the photovoltaic domain fine-tuning process.

[0063] In the field of photovoltaic (PV) equipment fault diagnosis, while general-purpose multimodal large-scale language models possess cross-modal information processing capabilities, they lack deep adaptation to PV-specific knowledge, making it difficult to directly meet the demands of high-precision fault analysis. When faced with multimodal PV data, these models often exhibit feature interpretation biases, failing to accurately analyze subtle differences in fault-related features, or generating diagnostic results lacking the specific safety regulations and maintenance logic required for PV equipment. Structured multimodal fault datasets and joint training loss functions provide core support for model domain adaptation. Through targeted fine-tuning, the model can deeply master PV fault diagnosis knowledge, forming specialized analytical capabilities.

[0064] Therefore, fine-tuning the basic model based on a structured multimodal fault dataset and a joint training loss function is a key step in constructing a dedicated multimodal model for photovoltaic fault analysis.

[0065] Specifically, step S3 includes: 31) Divide the preprocessed structured multimodal fault dataset into a training set and a validation set according to a preset ratio; A stratified sampling strategy is adopted to divide the preprocessed structured multimodal fault dataset according to a preset ratio (such as training set: validation set = 8:2 or training set: validation set = 7:3). This ensures that the multimodal data distribution (such as the proportion of visual modality / text modality) and fault type coverage (such as the proportion of fault categories such as component hot spots and inverter overvoltage) in the training set and validation set are completely consistent, thus avoiding the distortion of validation results due to data distribution deviation.

[0066] It should be noted that the dataset was globally shuffled before the split to eliminate the impact of sample order on the training effect. After the split, the training set and validation set were batch-organized (e.g., each batch contained 32 samples), and unique identifiers were added to the samples to associate multimodal data and structured diagnostic results, providing standardized data input for subsequent batch training and performance verification.

[0067] 32) Input the training set into the initial multimodal large language model in batches for cross-modal feature encoding to obtain a joint feature representation that integrates visual modality encoding and text modality semantic features; The organized training set batch data is input into the initial multimodal large language model. First, the data is standardized through the built-in visual modality preprocessing submodule and text modality preprocessing submodule. Then, it enters the dynamic routing architecture for cross-modal feature encoding: common features of multimodal data are extracted by sharing a low-rank matrix, each adapter activates adaptive encoding for the modality type and fault features of the data, and the routing mechanism fuses specific features through dynamic weight allocation. Finally, the feature fusion layer outputs a joint feature representation that integrates high-dimensional visual modality encoding and semantic features of text modality. This feature representation retains the core correlation information of multimodal data and contains fault-specific characterization in the photovoltaic field, providing comprehensive feature support for the generation of subsequent diagnostic results.

[0068] 33) Input the joint feature representation into the language decoder of the initial multimodal large language model to generate fault diagnosis prediction results, and calculate the joint training loss function value based on the fault diagnosis prediction results; Specifically, the joint feature representation is input into the language decoder of the initial multimodal large language model. Based on the pre-trained semantic generation logic and photovoltaic domain adaptation parameters, the language decoder generates a token-level prediction sequence, i.e., the fault diagnosis prediction result, in a structured order of fault type, cause analysis, and maintenance suggestions. The language decoder is the core component in the initial multimodal large language model responsible for transforming the cross-modal joint feature representation into structured text output, and it adopts an autoregressive Transformer decoder structure.

[0069] Furthermore, the loss value is calculated based on the pre-constructed joint training loss function. Specifically, firstly, the multimodal feature alignment loss is calculated using cosine similarity to quantify the correlation deviation between visual encoding and text semantics in the joint feature representation; then, the fault diagnosis result generation loss is calculated using cross-entropy loss, structural constraint loss, and term penalty term to quantify the difference between the predicted labeled sequence and the true labeled sequence.

[0070] Finally, according to the dynamic weighting coefficient ( , The weighted summation yields the joint training loss function value, providing an optimization basis for parameter updates.

[0071] 34) Based on the backpropagation algorithm, the trainable parameters in the dynamic routing architecture are updated by combining the joint training loss function value; Specifically, step 34) includes: a) Calculate the partial derivative of the joint training loss function with respect to the trainable parameters in the dynamic routing architecture; Based on the backpropagation algorithm, starting from the joint training loss function value, the partial derivatives of the loss with respect to all trainable parameters in the dynamic routing architecture are calculated layer by layer according to the chain rule. Trainable parameters include shared low-rank matrix parameters (…). ), adapter parameters ( ) and routing mechanism parameters ( During the partial derivative calculation process, the gradient contribution of the multimodal feature alignment loss and the fault diagnosis result generation loss are synchronously associated to ensure that the gradient of each type of parameter can reflect its impact on the dual objectives of feature alignment and diagnosis generation, providing accurate directional guidance for parameter updates (if the gradient is positive, the parameter value should be decreased; if the gradient is negative, the parameter value should be increased).

[0072] b) Use a preset optimizer to iteratively update the trainable parameters based on the partial derivatives and the current training parameters.

[0073] The preset optimizer is either the Adam optimizer or the SGD optimizer, which are suitable for multimodal fine-tuning scenarios. The initial learning rate is configured (e.g., ...). ) and learning rate decay strategy (such as decaying to 0.9 times the original value every 10 rounds).

[0074] The optimizer iteratively updates the parameters based on the calculated partial derivatives, current trainable parameter values, and a preset learning rate, following an adaptive gradient descent logic. Specifically, for shared low-rank matrix parameters, the focus is on gradient optimization based on the multimodal feature alignment loss to ensure the capture of common multimodal features; for adapter parameters and routing mechanism parameters, gradient adjustments to the loss are prioritized based on fault diagnosis results to enhance modality-specific adaptation and activation weight allocation accuracy. During parameter updates, only the trainable parameters of the dynamic routing architecture are adjusted, while the parameters of the basic model backbone network remain fixed to avoid the loss of general cross-modal knowledge.

[0075] It should be noted that during the parameter update process, gradient clipping is performed on the adapter parameters to limit the absolute value of the gradient within a certain range, thereby avoiding gradient explosion and affecting the model training effect.

[0076] 35) After each preset training round, the fault diagnosis performance verification index of the initial multimodal large language model is calculated using the validation set. If the fault diagnosis performance verification index does not improve for several consecutive rounds, the early stopping mechanism is triggered, and the current model parameters are saved as the target multimodal large language model.

[0077] In this embodiment, the preset training rounds are set to 5-10 rounds. After each round of training is completed, the validation set is completely input into the model for performance evaluation. The performance validation indicators for fault diagnosis include fault diagnosis accuracy (correctness of fault type determination), report completeness (percentage of core fields without missing data), terminology standardization (correctness of photovoltaic terminology usage), and feature alignment accuracy (mean of visual-text feature similarity).

[0078] A sliding window monitoring mechanism is adopted. If the fault diagnosis performance verification index on the validation set does not improve significantly for 3-5 consecutive rounds (the improvement is lower than the preset threshold, such as 0.5%), the early stopping mechanism is triggered to terminate the training and avoid the model from overfitting to the training set. At the same time, the trainable parameters of the current dynamic routing architecture and the fixed parameters of the basic model are saved and integrated to form a target multimodal large language model with cross-modal reasoning capabilities and adaptability to the photovoltaic field.

[0079] S4. Input the real-time operating data of the photovoltaic equipment into the target multimodal large language model for reasoning to obtain the fault diagnosis results of the photovoltaic equipment.

[0080] Specifically, step S4 includes: 41) Collect real-time operating data of photovoltaic equipment, and input the preprocessed real-time operating data of photovoltaic equipment into the target multimodal large language model for modality type recognition and cross-modal feature fusion to generate structured fault diagnosis results; It collects multimodal data during the real-time operation of photovoltaic equipment, including visible light images, infrared thermal imaging data, thermal imaging heat maps, and real-time sensor log data such as voltage, current, and temperature from key equipment such as photovoltaic panels, inverters, and combiner boxes.

[0081] The collected real-time data undergoes the same preprocessing operations as during the training phase: image data is normalized to a uniform resolution (to adapt to the input size of the target multimodal large language model) and segmented into core regions to eliminate background interference; text data such as sensor logs are formatted and outliers are removed, transforming them into a semantic format that the model can parse.

[0082] The preprocessed real-time data is input into the target multimodal large language model. The model automatically determines the modality type of each data point through the modality recognition module in the dynamic routing architecture, and dynamically activates the corresponding adapter based on modality attributes and data features. Simultaneously, it combines a shared low-rank matrix to complete cross-modal feature fusion, mapping the multimodal data to a unified semantic space. Finally, it generates a structured fault diagnosis result containing preliminary fault type, potential causes, and basic maintenance directions according to a preset structured template. The modality recognition module refers to the built-in functional module in the dynamic routing architecture that can automatically determine the modality type of the data (such as visible light, infrared, sensor logs, etc.).

[0083] 42) Perform format verification on the structured fault diagnosis results to obtain photovoltaic equipment fault diagnosis results that include fault type, cause analysis and maintenance suggestions.

[0084] Based on the diagnostic report standards in the photovoltaic operation and maintenance field, a pre-set format verification template is provided. The template clearly defines the required fields, expression standards, and logical relationship requirements for the three core fields: fault type, cause analysis, and maintenance recommendations. For example, the cause analysis needs to be consistent with the physical mechanism of the fault type, and the maintenance recommendations need to be operable.

[0085] First, check whether the core fields are complete and whether the format meets the template requirements. If there are problems such as missing fields or disordered format, trigger the model's secondary inference. Then, focus on verifying the standardization of fault type terminology, the scientific nature of cause analysis, and the relevance of maintenance suggestions to ensure that the diagnostic results are logically consistent and free of professional expression errors.

[0086] After verification, the output is a standard format, complete content, and photovoltaic equipment fault diagnosis result that can directly support operation and maintenance decisions.

[0087] This invention discloses a photovoltaic equipment fault diagnosis method. Based on a dynamic routing architecture constructed from the pre-trained weights of an initial multimodal large language model, it employs a two-layer adaptation design of "shared low-rank matrix + dedicated adapter." This design retains the cross-modal general reasoning capability of the pre-trained model through the shared low-rank matrix, while the adapter achieves specific adaptation for different photovoltaic modalities and fault types. Compared to traditional single low-rank adaptation schemes, this significantly improves the feature capture accuracy for heterogeneous multimodal photovoltaic data. A joint training loss function based on "multimodal feature alignment loss + fault diagnosis result generation loss," combined with the parameter update rules of the dynamic routing architecture, can simultaneously enhance cross-modal semantic consistency and ensure diagnostic accuracy during fine-tuning. Furthermore, by fixing the backbone network parameters and updating only the trainable parameters of the architecture, the method effectively avoids the model losing its general cross-modal reasoning capability, balancing the model's domain adaptability and generalization ability, ensuring stable output of diagnostic results even in unseen photovoltaic fault scenarios.

[0088] like Figure 2 The diagram shown is a structural schematic of a photovoltaic equipment fault diagnosis system according to an embodiment of the present invention. (Refer to...) Figure 2 An embodiment of the present invention provides a photovoltaic equipment fault diagnosis system, comprising: Dataset construction module 01 is used to obtain a structured multimodal fault dataset based on publicly available photovoltaic datasets and multi-source heterogeneous data collected during the operation of photovoltaic equipment. Architecture design module 02 is used to select an initial multimodal large language model and construct a feature-oriented dynamic routing architecture based on the pre-trained weight matrix of the initial multimodal large language model. Model training module 03 is used to fine-tune the initial multimodal large language model based on the structured multimodal fault dataset and obtain the target multimodal large language model by using a pre-built joint training loss function. The joint training loss function is constructed based on the multimodal feature alignment loss and the fault diagnosis result generation loss, and the loss optimization direction of the joint training loss function is determined based on the parameter update rules of the dynamic routing architecture. The fault diagnosis module 04 is used to input real-time operating data of photovoltaic equipment into the target multimodal large language model for reasoning to obtain fault diagnosis results of photovoltaic equipment.

[0089] It should be noted that each module in the aforementioned photovoltaic equipment fault diagnosis system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module. For specific limitations regarding the photovoltaic equipment fault diagnosis system, please refer to the limitations of the photovoltaic equipment fault diagnosis method described above; both have the same function and role, and will not be repeated here.

[0090] This invention also provides a terminal device, which includes: Processor, memory, and bus; The bus is used to connect the processor and the memory; The memory is used to store operation instructions; The processor is configured to execute operations corresponding to the photovoltaic equipment fault diagnosis method described above by invoking the operation instructions.

[0091] In one alternative embodiment, a terminal device is provided, such as Figure 3 As shown, Figure 3 The terminal device 5000 shown includes a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, for example, via a bus 5002. Optionally, the terminal device 5000 may also include a transceiver 5004. It should be noted that in practical applications, the transceiver 5004 is not limited to one type, and the structure of this terminal device 5000 does not constitute a limitation on the embodiments of the present invention.

[0092] Processor 5001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 5001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0093] Bus 5002 may include a path for transmitting information between the aforementioned components. Bus 5002 may be a PCI bus or an EISA bus, etc. Bus 5002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0094] The memory 5003 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0095] The memory 5003 is used to store application code that executes the present invention, and its execution is controlled by the processor 5001. The processor 5001 is used to execute the application code stored in the memory 5003 to implement the content shown in any of the foregoing method embodiments.

[0096] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the photovoltaic equipment fault diagnosis method described above.

[0097] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0098] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] In summary, the present invention provides a photovoltaic equipment fault diagnosis method, system, device, and medium. Based on a dynamic routing architecture constructed from the pre-trained weights of an initial multimodal large language model, it employs a two-layer adaptation design of "shared low-rank matrix + dedicated adapter." This design retains the cross-modal general reasoning capability of the pre-trained model through the shared low-rank matrix while achieving specific adaptation for different photovoltaic modalities and fault types through the adapter. Compared to traditional single low-rank adaptation schemes, this significantly improves the feature capture accuracy for multimodal heterogeneous photovoltaic data. The joint training loss function, constructed based on "multimodal feature alignment loss + fault diagnosis result generation loss," combined with the parameter update rules of the dynamic routing architecture, simultaneously enhances cross-modal semantic consistency and ensures the accuracy of diagnostic results during fine-tuning. Furthermore, by fixing the backbone network parameters and updating only the trainable parameters of the architecture, the loss of the model's general cross-modal reasoning capability is effectively avoided, balancing the model's domain adaptability and generalization ability, ensuring stable output of diagnostic results even in unseen photovoltaic fault scenarios.

[0102] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0103] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A method for diagnosing faults in photovoltaic equipment, characterized in that, include: Based on publicly available photovoltaic datasets and multi-source heterogeneous data collected during the operation of photovoltaic equipment, a structured multimodal fault dataset is obtained. An initial multimodal large language model is selected, and a feature-guided dynamic routing architecture is constructed based on the pre-trained weight matrix of the initial multimodal large language model. Based on the structured multimodal fault dataset, the initial multimodal large language model is fine-tuned using a pre-constructed joint training loss function to obtain the target multimodal large language model. The joint training loss function is constructed based on multimodal feature alignment loss and fault diagnosis result generation loss, and the loss optimization direction of the joint training loss function is determined based on the parameter update rules of the dynamic routing architecture. The real-time operating data of the photovoltaic equipment is input into the target multimodal large language model for reasoning to obtain the fault diagnosis results of the photovoltaic equipment.

2. The photovoltaic equipment fault diagnosis method according to claim 1, characterized in that, The structured multimodal fault dataset, obtained based on publicly available photovoltaic datasets and multi-source heterogeneous data collected during the operation of photovoltaic equipment, includes: Based on publicly available photovoltaic datasets, multi-source heterogeneous data generated during the operation of photovoltaic equipment are collected to obtain multimodal photovoltaic data; The multimodal photovoltaic data is transformed into an initial structured diagnostic result containing fault type, cause analysis, and maintenance recommendations through chain reasoning; The initial structured diagnostic results are reviewed and revised to obtain the target structured diagnostic results; The multimodal photovoltaic data and the target structured diagnostic results are correlated to obtain a structured multimodal fault dataset.

3. The photovoltaic equipment fault diagnosis method according to claim 1, characterized in that, The step of selecting an initial multimodal large language model and constructing a feature-guided dynamic routing architecture based on the pre-trained weight matrix of the initial multimodal large language model includes: A multimodal large language model with visual-text modality alignment and semantic reasoning capabilities is selected as the base model, and the backbone network parameters of the base model are fixed to obtain the initial multimodal large language model. The pre-trained weight matrix of the initial multimodal large language model is decomposed into a low-rank matrix to obtain a shared low-rank matrix, wherein the shared low-rank matrix is ​​configured to capture the common features of multimodal photovoltaic data; Based on the differences in photovoltaic data modes and the fault types of photovoltaic equipment, several adapters are constructed, wherein each adapter is configured to capture the photovoltaic characteristics of the corresponding mode or the fault characteristics of a specific type. A routing mechanism is introduced, comprising a feature encoder and a weight predictor. The feature encoder is configured to extract features from the input multimodal photovoltaic data to obtain a photovoltaic feature vector, and the weight predictor is configured to calculate the activation weight of each adapter by combining the photovoltaic feature vector. The shared low-rank matrix, each adapter, and the routing mechanism are integrated to form a dynamic routing architecture.

4. The photovoltaic equipment fault diagnosis method according to claim 1, characterized in that, The process of constructing the joint training loss function includes: We use cosine similarity calculation to compare the similarity between photovoltaic image features and photovoltaic text features, and construct a multimodal feature alignment loss. The difference between the generated fault diagnosis result labels and the actual fault diagnosis result labels is compared using the cross-entropy loss calculation method to construct the fault diagnosis result generation loss; The joint training loss function is obtained by weighted summation of the multimodal feature alignment loss and the fault diagnosis result generation loss.

5. The photovoltaic equipment fault diagnosis method according to claim 1, characterized in that, The step of fine-tuning the initial multimodal large language model based on the structured multimodal fault dataset using a pre-constructed joint training loss function to obtain the target multimodal large language model includes: The preprocessed structured multimodal fault dataset is divided into a training set and a validation set according to a preset ratio; The training set is input into the initial multimodal large language model in batches for cross-modal feature encoding to obtain a joint feature representation that integrates visual modality encoding and text modality semantic features; The joint feature representation is input into the language decoder of the initial multimodal large language model to generate a fault diagnosis prediction result, and the joint training loss function value is calculated based on the fault diagnosis prediction result; Based on the backpropagation algorithm, the trainable parameters in the dynamic routing architecture are updated by combining the joint training loss function value; After each preset training round, the fault diagnosis performance verification index of the initial multimodal large language model is calculated using the validation set. If the fault diagnosis performance verification index does not improve for several consecutive rounds, an early stopping mechanism is triggered, and the current model parameters are saved as the target multimodal large language model.

6. The photovoltaic equipment fault diagnosis method according to claim 5, characterized in that, The process of updating the trainable parameters in the dynamic routing architecture based on the backpropagation algorithm and the joint training loss function value includes: Calculate the partial derivative of the joint training loss function value with respect to the trainable parameters in the dynamic routing architecture; A preset optimizer is used to iteratively update the trainable parameters based on the partial derivatives and the current training parameters.

7. The photovoltaic equipment fault diagnosis method according to claim 1, characterized in that, The step of inputting real-time operating data of photovoltaic equipment into the target multimodal large language model for reasoning to obtain photovoltaic equipment fault diagnosis results includes: Real-time operating data of photovoltaic equipment is collected, and the preprocessed real-time operating data of photovoltaic equipment is input into the target multimodal large language model for modality type recognition and cross-modal feature fusion to generate structured fault diagnosis results; The structured fault diagnosis results are format-validated to obtain photovoltaic equipment fault diagnosis results that include fault type, cause analysis, and maintenance suggestions.

8. A photovoltaic equipment fault diagnosis system, characterized in that, include: The dataset construction module is used to obtain a structured multimodal fault dataset based on publicly available photovoltaic datasets and multi-source heterogeneous data collected during the operation of photovoltaic equipment. The architecture design module is used to select an initial multimodal large language model and construct a feature-oriented dynamic routing architecture based on the pre-trained weight matrix of the initial multimodal large language model. The model training module is used to fine-tune the initial multimodal large language model based on the structured multimodal fault dataset and to obtain the target multimodal large language model using a pre-built joint training loss function. The joint training loss function is constructed based on the multimodal feature alignment loss and the fault diagnosis result generation loss, and the loss optimization direction of the joint training loss function is determined based on the parameter update rules of the dynamic routing architecture. The fault diagnosis module is used to input real-time operating data of photovoltaic equipment into the target multimodal large language model for reasoning to obtain fault diagnosis results of photovoltaic equipment.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the photovoltaic equipment fault diagnosis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the photovoltaic equipment fault diagnosis method as described in any one of claims 1 to 7.

Images