Photovoltaic fault diagnosis method based on multi-modal big language

Abstract

The invention discloses a photovoltaic fault diagnosis method based on a multi-modal big language, which comprises the following steps: injecting professional knowledge related to photovoltaic fault diagnosis into a big language model to enable the photovoltaic fault diagnosis to learn and understand the content of a photovoltaic operation and maintenance industry, analyzing an inspection thermal imaging picture and professional experience of an expert by using the multi-modal big language model, and obtaining a fault diagnosis result through cross-modal cross data. The hidden complex fault problem in the photovoltaic module is analyzed more comprehensively, in addition, the database is updated periodically, model parameters are updated periodically, the diagnosis capacity of the complex fault and the hidden fault of the photovoltaic module is improved, and finally accurate fault recognition is formed.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of photovoltaic system operation and fault diagnosis, and relates to a multi-modal large language model and incremental learning for ensuring the stability, power generation efficiency and equipment life of a photovoltaic power station. BACKGROUND

[0002] With the continuous increase of photovoltaic installed capacity, the photovoltaic power station industry has entered a new stage of parallel development of scale and high quality. However, in the actual operation process of photovoltaic power stations, their operation and maintenance, power generation efficiency, and power station safety are all facing complex and diverse problems.

[0003] Chinese patent CN120237806A discloses a remote photovoltaic power generation operation and maintenance management and control system based on cloud technology. Through digital twin simulation of virtual models of photovoltaic components, combiner boxes, inverters and transformer devices, multi-person collaborative remote operation is realized, an intention reasoning conflict management system is established, a multi-level conflict resolution system is deployed, and augmented reality remote collaboration technology is integrated. However, this method requires experts to accurately guide on-site personnel, and it does not consider whether the on-site personnel can effectively find the characteristics of hidden faults and the consistency of communication and understanding between the two parties.

[0004] Chinese patent CN116805060A discloses a photovoltaic fault diagnosis method based on IV curve machine learning recognition. By collecting photovoltaic component current and voltage data, preprocessing is performed to obtain a data set, feature engineering is constructed including feature discovery and feature extraction, a classification algorithm is used for data enhancement, and a machine learning algorithm is used to identify the type of photovoltaic component fault. However, this method only uses IV curve features as the basis for judgment and cannot accurately lock the photovoltaic fault category.

[0005] In addition, traditional manual inspection relies on the experience of technical personnel, and full-scale inspection of large power stations requires several days, resulting in a large number of missed inspections. Only analyzing electrical parameters or a single data source of infrared images cannot comprehensively consider multi-modal information such as the environment and acoustics, and the missed diagnosis of complex faults will be more, therefore, there is an urgent need for a photovoltaic fault diagnosis method that can record inspection information comprehensively and integrate multiple types of information. SUMMARY

[0006] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides a photovoltaic fault diagnosis method related to a multi-modal large language model, which injects photovoltaic fault diagnosis related professional knowledge into the large language model, makes it learn and understand the photovoltaic operation and maintenance industry content, analyzes the inspection thermal imaging pictures and expert professional experience by using the multi-modal large language model, analyzes the hidden complex fault problems in the photovoltaic module through cross-modal cross-data, in addition, regularly updates the database, periodically updates the model parameters, improves the diagnosis ability of the complex fault and implicit fault of the photovoltaic module, and finally forms precise fault identification.

[0007] The present application is realized by the following technical solutions:

[0008] A photovoltaic fault diagnosis method based on a multi-modal large language, the method comprising: S1, constructing a photovoltaic operation and maintenance knowledge base

[0009] The photovoltaic operation and maintenance knowledge base is constructed by acquiring the historical photovoltaic module inspection thermal imaging pictures of the photovoltaic power station and the corresponding maintenance logs; S2, constructing a multi-modal photovoltaic operation and maintenance large model Based on the open source language large model, increase the text expert module and the image expert module, so as to construct the multi-modal photovoltaic operation and maintenance large model; S3, model parameter migration and specific scene retraining

[0010] The parameters in the original model which are the same as the basic structure are copied by using the hierarchical parameter migration strategy, and the bottom layer parameters are frozen; the text expert module and the image expert module are fine-tuned, and the general language ability is retained while adapting to the photovoltaic operation and maintenance task.

[0011] Further, the specific method of step S1 is:

[0012] S11, acquiring the historical photovoltaic module inspection thermal imaging pictures of the photovoltaic power station and the corresponding maintenance logs, and establishing a rule base;

[0013] S12, adopt multi-modal processing technology, after median filter denoising and normalization of thermal imaging map, use U-Net to segment abnormal area and extract morphological features and texture features; maintenance log is extracted by extracting entity relationship triplets; expert guidance text is embedded into vector by text coding; S13, based on steps S11 and S12, build a photovoltaic operation and maintenance knowledge base of "image-log-guidance". Further, the specific method of step S2 is: S21, based on DeepSeek-7B open source model architecture, reconfigure its hybrid expert model for photovoltaic operation and maintenance scene, add two types of vertical expert modules of text expert and image expert based on the general expert layer; S22, the gating neuron dynamically selects experts based on the type of input data, and shields irrelevant experts to reduce computational redundancy; S23, through the power dedicated adaptation layer, map the GLCM texture feature vector of thermal imaging and the semantic vector of maintenance log to the same hidden space, and use contrast learning to optimize feature correlation. Further, the specific method of step S3 is: based on the parameter structure of DeepSeek-7B open source model, retain its general semantic understanding layer, add divide-and-conquer text expert and image expert based on the original hybrid expert model; through hierarchical parameter migration strategy, copy the parameters same as the basic structure in the original model, and freeze the bottom layer parameters, only fine-tune the text expert module and image expert module, to retain the general language ability while adapting to photovoltaic operation and maintenance tasks; wherein, the text expert module uses the maintenance log and expert rules of photovoltaic operation and maintenance knowledge base for instruction fine-tuning, designs task prompt words, and injects domain knowledge through Adapter module; the image expert module introduces cross-modal alignment mechanism, uses contrast loss to map the U-Net segmentation features of thermal imaging and the fault description vector of text expert module to a unified hidden space, and realizes correlation reasoning;

[0014] The gating network dynamically allocates expert weights through sparse activation mask, and combines hierarchical learning rate strategy to optimize computational efficiency and reduce reasoning energy consumption.

[0015] Further, the dynamic screening logic of the gating neuron is: the input data is calculated by the gating weight W g and bias b g， to get the expert score g(x), select the Top k expert with the highest score to activate, and shield the rest of the experts: Further, the method further comprises: S4, periodic incremental learning

[0016] Update the knowledge base every 200 new maintenance cases, automatically annotate the abnormal area of thermal imaging map and pass the expert review

[0017] The photovoltaic operation and maintenance knowledge base is injected, and a "image-log-guidance" triple is constructed. Furthermore, step S4 further comprises: lightweight gate fine-tuning: every month, 10% of high-value samples are randomly extracted from the historical library by using a sparse experience replay technology, mixed with new data for training, only updating the gate network parameters, and constraining the activation weight distribution of the text expert module and the image expert module;

[0018] Full parameter deep update: full model retraining is performed every 3 months, combined with updated variation samples of hidden cracks and hot spots, to expand the original original fault types; and regularization constraint is used to freeze key parameters; the fault types learned by the large model are increased, and finally a multi-modal large language model capable of accurately diagnosing photovoltaic component faults is obtained.

[0019] Compared with the prior art, the present application has the following technical effects:

[0020] 1. The present application integrates multi-modal data such as thermal imaging images, maintenance logs and expert experience in photovoltaic operation and maintenance, overcomes the limitations of traditional methods which rely on only a single data source, can more comprehensively identify complex faults and hidden faults of photovoltaic components, and significantly reduces the missed detection rate; 2. The text expert and image expert modules are customized for photovoltaic operation and maintenance scenarios, and the relevant experts are dynamically activated through the gate neuron, which greatly reduces redundant calculations, so that the model can be efficiently trained and deployed on a consumer-grade GPU; 3. Through the power-specific adaptation layer and contrastive learning, the texture features of thermal imaging and the semantic features of maintenance logs are mapped to a unified hidden space, realizing accurate association of "image searching logs" and improving diagnostic explainability; 4. The knowledge base is updated periodically and dynamic weighted sampling is used to preferentially learn complex fault patterns; combined with sparse experience replay and hierarchical fine-tuning strategies, catastrophic forgetting is avoided, and the model continuously improves the diagnostic accuracy in long-term operation. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The implementation process of the overall model architecture of the present application;

[0022] Figure 2 The principle diagram for selecting experts by the gate neuron in the present application. DETAILED DESCRIPTION

[0023] The technical solutions of the embodiments of the present application will be described below in conjunction with the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0024] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "linking" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection or can communicate with each other; it can be directly connected, or indirectly connected through intermediate medium, or the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0025] In the present application, unless otherwise explicitly specified and limited, the "upper" or "lower" of the first feature to the second feature can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, the "upper", "above" and "on" of the first feature to the second feature includes that the first feature is directly above and obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The "below", "under" and "under" of the first feature to the second feature includes that the first feature is directly below and obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0026] Those skilled in the art can understand that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and unless defined as such, should not be interpreted in an idealized or overly formal sense.

[0027] As Figure 1 shown, the present application proposes a photovoltaic fault diagnosis method based on multi-modal large language, which includes the following steps:

[0028] S1, constructing photovoltaic operation and maintenance knowledge base

[0029] Building a photovoltaic operation and maintenance knowledge base requires the integration of multi-source data and intelligent technology to form a closed-loop system of "image-log-guidance". First, for the specific power station's historical thermal imaging map (annotated hot spot, hidden crack, etc. regional coordinates) and maintenance log (structured extraction of fault type, treatment measures, root cause, etc. fields), through expert interviews, the experience is converted into a rule base. Second, multi-modal processing technology is used: after median filter denoising and normalization of the thermal imaging map, U-Net is used to segment the abnormal area and extract morphological features and texture features; maintenance logs extract entity relationship triples (such as "[component hidden crack, cause, hot spot]"); expert text is converted into vector embeddings. The knowledge base needs to design a dynamic updating mechanism: when a new fault case is added, a temporary node is automatically generated and expert review is triggered, and the confidence of the solution is optimized based on the operation and maintenance score feedback.

[0030] S2, build a multi-modal photovoltaic operation and maintenance large model based on deepseek

[0031] Based on the DeepSeek-7B open source model architecture, a hybrid expert model (MoE) is reconstructed for the photovoltaic operation and maintenance scenario, adding two types of vertical expert modules, text and image experts, to the general expert layer. The text expert focuses on processing professional text data features such as maintenance logs and expert rules, and achieves semantic analysis through word vector embedding; the image expert is responsible for analyzing thermal imaging features, using a lightweight ViT model to extract morphological features (such as the area / perimeter ratio of the hot spot segmented by U-Net) and texture features (such as the contrast / entropy of the gray level co-occurrence matrix). The gating neuron design uses a sparse activation mechanism to dynamically select experts based on input data types: for example, when the input is "thermal imaging + expert guidance", the gating network activates the image expert (processing hot spot features) and the text expert (analyzing fault correlation rules), while irrelevant experts (such as experts for non-related faults, and faults with different manifestations correspond to different expert neurons) are screened out, significantly reducing computational redundancy. Cross-modal alignment is achieved through a power-specific adaptation layer: GLCM texture feature vectors of thermal imaging and semantic vectors of maintenance logs are mapped to the same hidden space, and contrast learning loss is used to optimize feature similarity, ensuring high correlation between "local high temperature image features" and "hot spot fault description" in the vector space.

[0032] ;

[0033] where W g is the gating weight, b g is the bias term, Top k only selects the top K experts, and g(x) is the expert score.

[0034] S3, model parameter migration and specific scenario retraining

[0035] Based on the parameter structure of the DeepSeek-7B open source model, the general semantic understanding layer (such as the Transformer encoder) is retained, and the divide-and-conquer text and image experts are added on the basis of the original mixed expert layer (MoE). Through the hierarchical parameter migration strategy, the parameters with the same basic structure as the original model are copied, and the bottom layer parameters are frozen, and only the text and image experts are fine-tuned (the learning rate is set to 1 / 10 of the base model) to retain the general language ability while adapting to the photovoltaic operation and maintenance task. The image expert module uses a lightweight ViT architecture and is initialized through a progressive domain adaptation strategy.

[0036] Text expert module: Use the maintenance log and expert rules of the photovoltaic operation and maintenance knowledge base to fine-tune the instructions, design task prompts such as "generate maintenance steps according to coordinates (X, Y) hot spots and 30% current drop", and inject domain knowledge through the Adapter module (only train new parameters, fix original weights) to avoid catastrophic forgetting.

[0037] Image expert module: Introduce a cross-modal alignment mechanism, use contrastive loss (InfoNCE) to map the U-Net segmentation features of thermal imaging (such as abnormal area area / temperature gradient) and the fault description vector of the text expert to a unified hidden space, and realize "image searching log" correlation reasoning.

[0038] As shown in Figure 2 , the gating network dynamically allocates expert weights through sparse activation masks (such as thermal imaging + current data activating dual experts, and pure text logs only activating text experts), and combines a hierarchical learning rate strategy (gating network learning rate 1e-3, expert module learning rate 1e-5) to optimize computing efficiency and reduce inference energy consumption.

[0039] Based on the parameter order of magnitude of the DeepSeek-7B open source model, the model can be trained in a consumer-level GPU and can be deployed in practice.

[0040] S4, Periodic incremental learning gradually improves the model's ability to diagnose complex faults

[0041] Every 200 sets of maintenance cases (including thermal imaging, current fluctuation logs, and expert correction opinions) trigger knowledge base updates. Adopt an automated labeling process: pre-label abnormal areas (such as hot spot coordinates and hidden crack patterns) on new thermal imaging using a pre-trained ViT model, then inject the knowledge base after expert review, and simultaneously build "image-log-guidance" triplets. Introduce dynamic weighted sampling: give higher weights (weight coefficient ≥ 1.5) to implicit fault samples (such as PID effects) to ensure that the model learns low-frequency complex fault patterns first.

[0042] Lightweight gating fine-tuning: every month, 10% of high-value samples (such as misdiagnosis cases) are randomly selected from the historical library and mixed with new data for training, only updating the gating network parameters (learning rate 1e-3), and constraining the activation weight distribution of text / image experts (such as image expert weight ≥ 0.8 when diagnosing hot spots) to avoid catastrophic forgetting.

[0043] Full-parameter deep update: every 3 months, full model retraining is performed, combined with updated hidden crack and hot spot variant samples (such as different angle thermal imaging, occlusion scene), expanding the original original fault type; regularize constraint is used to freeze key parameters (such as text expert bottom layer Transformer layer), protect core knowledge from being covered. Continuously increase the fault types learned by the large model, and finally obtain a multi-modal large language model that can accurately diagnose photovoltaic component faults.

[0044] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A photovoltaic fault diagnosis method based on multi-modal large language, characterized by, The method comprises the following steps: S1, constructing a photovoltaic operation and maintenance knowledge base; S2, constructing a multi-modal photovoltaic operation and maintenance large model; S3, model parameter migration and specific scene retraining; The specific method of the step S1 is as follows:

2. The photovoltaic fault diagnosis method based on multi-modal large language according to claim 1, characterized in that, S11, acquiring historical photovoltaic component inspection thermal imaging graphs and corresponding maintenance logs of a photovoltaic power station, and establishing a rule base; S12, adopting a multi-modal processing technology, carrying out median filter denoising and normalization on the thermal imaging graphs, segmenting abnormal areas by using a U-Net, and extracting morphological features and texture features; extracting entity relationship triplets from the maintenance logs; and generating vector embedding by text encoding on the expert guidance text; S13, constructing a "image-log-guidance" photovoltaic operation and maintenance knowledge base based on the steps S11 and S12. The specific method of the step S2 is as follows: S21, reconstructing a hybrid expert model for a photovoltaic operation and maintenance scene based on a DeepSeek-7B open source model architecture, adding two types of vertical expert modules, namely a text expert and an image expert, to a general expert layer of the hybrid expert model; S22, dynamically screening experts based on input data types by using a gating neuron, and shielding irrelevant experts to reduce calculation redundancy; S23, mapping GLCM texture feature vectors of the thermal imaging and semantic vectors of the maintenance logs to the same hidden space by using a power special adaptation layer, and optimizing feature correlation by using contrastive learning.

3. The photovoltaic fault diagnosis method based on multi-modal large language according to claim 2, characterized in that, The specific method of the step S3 is as follows: based on the parameter structure of the DeepSeek-7B open source model, retaining a general semantic understanding layer, adding a divide-and-conquer text expert and an image expert to the original hybrid expert model, copying parameters in the original model which are the same as the basic structure by using a hierarchical parameter migration strategy, and freezing the bottom layer parameters, and only fine-tuning the text expert module and the image expert module to retain general language ability while adapting to photovoltaic operation and maintenance tasks; wherein, the expert module uses the maintenance logs and expert rules of the photovoltaic operation and maintenance knowledge base to fine-tune instructions, designs task prompt words, and injects domain knowledge through an Adapter module; the image expert module introduces a cross-modal alignment mechanism, maps U-Net segmentation features of the thermal imaging and fault description vectors of the text expert module to a unified hidden space by using a contrastive loss, and realizes correlation reasoning; 4. The photovoltaic fault diagnosis method based on multi-modal large language according to claim 3, characterized in that, The gating network dynamically allocates expert weights by using a sparse activation mask, optimizes calculation efficiency by using a hierarchical learning rate strategy, and reduces reasoning energy consumption. Further comprising:

5. The photovoltaic fault diagnosis method based on multi-modal large language according to claim 3, characterized in that, The dynamic screening logic of the gating neuron is: input data is gated by weight W g and bias b g， Calculate expert score g(x), select Top k The expert with the highest score activates, and the rest of the experts are masked: ​ 6. The photovoltaic fault diagnosis method based on multi-modal large language according to claim 2, characterized in that, S4, periodic incremental learning Every 200 new maintenance cases are accumulated to update the knowledge base, and the abnormal areas of the thermal imaging graphs are automatically annotated and audited by experts ​ The photovoltaic operation and maintenance knowledge base is injected, and a "image-log-guidance" triple is constructed. Dynamic weighted sampling is introduced to preferentially learn low-frequency complex fault modes.

7. The photovoltaic fault diagnosis method based on multi-modal large language according to claim 6, characterized in that, The step S4 further includes: lightweight gating fine-tuning: every month, 10% of high-value samples are randomly extracted from the historical database by using a sparse experience replay technique, mixed with new data for training, only updating the gating network parameters, and constraining the activation weight distribution of the text expert module and the image expert module; Full-parameter deep update: every 3 months, full model retraining is performed, combined with updated variation samples of hidden cracks and hot spots, to expand the original original fault types; regularization constraints are used to freeze key parameters; the fault types learned by the large model are increased, and finally a multi-modal large language model capable of accurately diagnosing photovoltaic component faults is obtained.

Images