Photovoltaic module fault cross-scene diagnosis method based on multi-mode transfer learning

Abstract

The invention discloses a photovoltaic module fault cross-scene diagnosis method based on multi-mode transfer learning, and relates to the technical field of photovoltaic operation and maintenance. Aiming at the problem of low fault diagnosis precision in different operation scenes with different illumination, temperatures, component models and the like, the method comprises the following steps: acquiring multi-modal data of images, infrared thermal imaging, electrical parameters and environmental parameters, performing preprocessing and time-space alignment, extracting multi-modal features through improved ResNet-50, a U-Net encoder, a bidirectional LSTM (Long Short Term Memory) and a full-connection network, and determining the fault diagnosis accuracy of the multi-modal data through the improved ResNet-50, a U-Net encoder, a bidirectional LSTM and a full-connection network. A modal attention fusion mechanism is adopted to fuse features, a domain self-adaptive transfer learning model is combined, classification loss, adversarial loss and MMD loss are used for weighting a training model, and accurate diagnosis of fault types and positions of hot spots, hidden cracks and the like in a cross-scene mode is achieved. The method effectively reduces the influence of scene difference, improves the fault diagnosis robustness and precision, and is suitable for the operation and maintenance of a complex photovoltaic power station.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of photovoltaic energy technology and artificial intelligence. Specifically, it relates to a cross-scenario diagnosis method for photovoltaic module faults based on multimodal transfer learning. It is particularly suitable for photovoltaic power plants with different operating environments and different module types, enabling efficient and accurate cross-scenario detection and classification of photovoltaic module faults. Background Technology

[0002] As the global energy structure shifts towards cleaner energy sources, photovoltaic (PV) power generation, as a crucial component of renewable energy, is experiencing rapid and sustained growth in installed capacity. PV modules, as the core component of PV power generation systems, are constantly exposed to complex outdoor environments such as high temperatures, high humidity, strong ultraviolet radiation, and dust storms, making them prone to various faults such as hot spots, microcracks, junction box failures, and PID degradation. Statistics show that PV module failures can lead to a 10% to 30% loss in power generation, and in severe cases, even cause safety accidents such as fires. Therefore, achieving early and accurate diagnosis of PV module failures is of great significance for ensuring the safe and stable operation of power plants and improving power generation efficiency.

[0003] In existing technologies, photovoltaic module fault diagnosis methods are mainly divided into two categories: traditional detection methods and artificial intelligence-based diagnostic methods. 1. Traditional detection methods Manual inspection: Relies on on-site observation by maintenance personnel or detection by handheld devices, which has problems such as low efficiency, high cost, strong subjectivity, and high rate of missed detection, and is especially unsuitable for large photovoltaic power plants.

[0004] Single-sensor detection: such as detecting hot spot faults using an infrared thermal imager, or determining faults by measuring electrical parameters using a multimeter. These methods rely on only a single physical quantity, are easily affected by environmental interference, and cannot comprehensively reflect the fault characteristics.

[0005] 2. Artificial intelligence-based diagnostic methods In recent years, machine learning technology has been applied to photovoltaic module fault diagnosis, mainly divided into single-mode data-driven methods and finite-scenario-dependent methods: Single-modal data-driven methods, such as convolutional neural network (CNN) diagnostic models based on visible light images and temporal neural network (RNN) models based on electrical parameters, have the following limitations: Information limitations: Visible light images are easily affected by illumination angle and shadow occlusion, making it difficult to distinguish between "false faults" and real hot spots; Infrared thermal imaging is not sensitive enough to faults such as early microcracks and PID attenuation; Electrical parameters are affected by factors such as ambient temperature and irradiance, resulting in low diagnostic accuracy when used alone.

[0006] Poor cross-scene adaptability: Model training relies on a large amount of labeled data in a specific scene. When the scene changes, the feature distribution shift causes the model's generalization ability to drop sharply. Data needs to be collected again and the model needs to be trained, which is costly and time-consuming.

[0007] Multimodal data fusion methods: A few studies have attempted to fuse two modalities of data, but they have not solved the problem of cross-scene transfer. Moreover, the fusion strategies are mostly simple splicing or weighted summation, without considering the complementarity and redundancy between modalities, resulting in low quality of fused features.

[0008] Summary of core technical problems in existing technologies In summary, existing photovoltaic module fault diagnosis technologies face the following pressing issues that need to be addressed: Single-modal data is incomplete: single-modal data cannot fully characterize complex fault features, resulting in low diagnostic accuracy and a high misjudgment rate; Poor cross-scene adaptability: Traditional models rely on labeled data for specific scenes. When scene changes such as lighting, temperature, and component model differences occur, the diagnostic performance will significantly decrease due to feature distribution shift. Data dependency: In actual operation and maintenance, due to the high cost of manual annotation and the difficulty in obtaining fault samples, fault annotation data in the target scenario is scarce, making it difficult to apply traditional supervised learning methods.

[0009] Therefore, developing a photovoltaic module fault diagnosis method that can integrate the advantages of multimodal data, overcome scenario limitations, and reduce dependence on labeled data has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0010] To address the problems of one-sidedness of single-modal data information, poor cross-scenario adaptability, and dependence on labeled data in existing photovoltaic module fault diagnosis technologies, the purpose of this invention is to provide a cross-scenario fault diagnosis method for photovoltaic modules based on multimodal transfer learning. By integrating multimodal data features with a domain-adaptive transfer learning mechanism, this method achieves efficient and accurate fault diagnosis of photovoltaic modules under different operating scenarios, reduces dependence on labeled data of the target scenario, and improves the generalization ability and practicality of the model.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: A method for cross-scenario fault diagnosis of photovoltaic modules based on multimodal transfer learning includes the following steps: S1. Multimodal data acquisition and preprocessing Annotated multimodal data of photovoltaic (PV) modules in the source scenario and unannotated / partially labeled multimodal data in the target scenario were collected. The multimodal data included visible light image modal data of the PV module surface, infrared thermal imaging modal data such as temperature distribution information, electrical parameter modal data such as open-circuit voltage / short-circuit current, and environmental parameter modal data such as ambient temperature, irradiance, and wind speed. The collected multimodal data underwent preprocessing: Gaussian noise was removed from the image data using a non-local mean denoising algorithm; temperature calibration was performed on the infrared thermal imaging data to convert it into actual temperature values; outlier removal and Z-score standardization were performed on the electrical parameter data; missing values ​​were imputed using nearest-neighbor interpolation for the environmental parameter data; and spatiotemporal alignment of the multimodal data was achieved through timestamp synchronization. Finally, standardized source-domain and target-domain multimodal datasets were obtained.

[0012] S2. Multimodal Feature Extraction Construct a multimodal feature extraction network and design dedicated subnetworks for different modalities of data: Image modal feature extraction: An improved ResNet-50 network is used to extract spatial features such as surface texture and defect contours in visible light images; Infrared thermal imaging modal feature extraction: The encoder part of the U-Net network is used to extract thermal features such as temperature gradient and abnormal region distribution in the infrared thermal image; Electrical parameter modal feature extraction: A bidirectional LSTM network is used to process the time series of electrical parameters with lengths of 50 to 200 and extract the temporal variation trend features; Environmental parameter modal feature extraction: A three-layer fully connected neural network with 128, 64 and 32 neurons was used to extract the impact features of environmental factors on component performance.

[0013] The above sub-networks are used to extract features from the source and target domain multimodal datasets, respectively, to obtain source domain multimodal features and target domain multimodal features.

[0014] S3. Construction and Training of Cross-Scene Transfer Learning Models Construct a domain-adaptive transfer learning model, which includes a feature fusion module, a domain alignment module, and a fault classification module: Feature fusion module: It adopts a modal attention fusion mechanism, calculates attention weights based on the mutual information entropy of each modal feature and fault label, and performs weighted summation on the multimodal features to obtain source domain fusion features and target domain fusion features; Domain alignment module: By using a domain discriminator and adversarial training unit, the difference in domain distribution between the source domain fusion features and the target domain fusion features is minimized, thereby achieving cross-scene feature alignment; Fault classification module: Employs an improved Transformer classifier with a classification head containing four encoder layers and a hybrid attention mechanism. Training is performed based on domain-aligned source domain features and labeled data. Simultaneously, multiple loss functions are introduced for collaborative optimization: the total loss function... ,in Cross-entropy classification loss, To counteract the loss of domain alignment, The maximum mean difference mode alignment loss is used. Through the above training process, a fault diagnosis model with cross-scenario generalization ability is obtained.

[0015] S4. Cross-scenario diagnosis of photovoltaic module faults After preprocessing, feature extraction, and fusion, the multimodal data of the photovoltaic modules to be diagnosed in the target scenario is input into the trained cross-scenario fault diagnosis model, which outputs fault types such as hot spots, microcracks, and junction box faults, as well as the spatial coordinates of the fault location through image modal features.

[0016] Due to the adoption of the above solution, the beneficial effects of the present invention are as follows: 1. Significantly improved diagnostic accuracy: By fusing data from four modalities—image, infrared, electrical, and environmental—a comprehensive characterization of fault features is achieved, avoiding misjudgments caused by the incompleteness of single-modal data. For example, visible light images locate the fault position, infrared thermal imaging verifies temperature anomalies, and electrical parameters reflect the degree of performance degradation. Multimodal collaboration improves diagnostic accuracy by 15% to 30%.

[0017] 2. Strong cross-scene adaptability: Through the domain alignment module and multi-loss function optimization, the model can achieve feature distribution alignment between source scenes with sufficient labeled data and target scenes with unlabeled / limited labeled data. This solves the problem of poor generalization ability of traditional models under scene changes such as illumination difference ≥200W / m², temperature difference ≥15℃, and different component models. The cross-scene diagnostic accuracy retention rate is ≥85%.

[0018] 3. Reduced dependence on labeled data: This invention uses a transfer learning mechanism to collaboratively train labeled data from the source scene and unlabeled data from the target scene, reducing the need for labeled data from the target scene. The amount of labeled data can be reduced by 60% to 80%, significantly reducing the cost of manual labeling. It is suitable for newly commissioned power plants lacking fault samples or old power plants with complex scenarios.

[0019] 4. Comprehensive fault type coverage: It can simultaneously diagnose a variety of common faults such as hot spots, microcracks, junction box faults, glass breakage, and PID attenuation, and achieve precise fault location through image modal features with a positioning error of ≤5mm, providing maintenance personnel with intuitive fault repair basis.

[0020] 5. High industrial applicability: The methodology is clear, data acquisition can be achieved through existing sensors such as cameras, infrared thermal imagers, and electrical parameter acquisition devices, and model training and inference can be deployed based on general deep learning frameworks such as PyTorch and TensorFlow. No special hardware is required, making it easy to promote and apply in large-scale photovoltaic power plants.

[0021] In summary, this invention, through the technological innovation of "multimodal data fusion + cross-scenario transfer learning," effectively solves the limitations of existing photovoltaic module fault diagnosis methods, and provides efficient and reliable technical support for the intelligent operation and maintenance of photovoltaic power plants. Attached Figure Description

[0022] Figure 1 This is an overall flowchart of the photovoltaic module fault cross-scenario diagnosis method based on multimodal transfer learning of the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal feature extraction network of the present invention; Figure 3 A schematic diagram of the architecture of a cross-scenario transfer learning model. Detailed Implementation

[0023] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. The embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0024] This embodiment discloses a cross-scenario fault diagnosis method for photovoltaic modules based on multimodal transfer learning, aiming to solve the problem of low fault diagnosis accuracy of photovoltaic modules under different operating scenarios such as light intensity, ambient temperature, and module model differences. By fusing multimodal data of images, infrared thermal imaging, electrical parameters, and environmental parameters, and combining domain adaptive transfer learning technology, efficient training and accurate diagnosis of the cross-scenario fault diagnosis model are achieved. The specific implementation of each step is described in detail below.

[0025] like Figure 1 As shown, a cross-scenario fault diagnosis method for photovoltaic modules based on multimodal transfer learning includes the following steps: S1. Multimodal data acquisition and preprocessing 1. Multimodal data acquisition The following equipment was used to collect four types of modal data, with specific parameter settings as follows: Image modal data: A Basler acA2500-14uc industrial camera was used, with a resolution of 512×512 pixels, a frame rate of 25fps, to capture visible light images of the photovoltaic module surface, covering the entire surface of the module, with a shooting distance of 5m and a lens focal length of 16mm.

[0026] Infrared thermal imaging modal data: The infrared thermal imager model FLIR T650sc was used, with a resolution of 640×512 pixels, a temperature measurement range of -20℃ to 100℃, a sampling frequency of 10fps, and outputting a pseudo-color thermal image containing temperature distribution.

[0027] Electrical parameter modal data: collected by Huawei SUN2000-6KTL-M0 photovoltaic string monitoring device, including open circuit voltage (Voc), short circuit current (Isc), maximum power point voltage (Vmp), and maximum power point current (Imp), with a sampling frequency of 1Hz and a time series length of 50~200.

[0028] Environmental parameter modal data: collected using a Davis Vantage Pro2 weather station, including ambient temperature with an accuracy of ±0.5℃, irradiance with an accuracy of ±10W / m², wind speed with an accuracy of ±0.3m / s, and a sampling frequency of 0.5Hz.

[0029] Data labeling: In the standard test environment of 25℃, irradiance of 1000W / m², and the source scenario of LONGi Hi-MO4 module, professionals labeled the collected data with five types of faults, including hot spots and microcracks, as well as normal state fault labels. In the high-temperature scenario of 45℃, irradiance of 800W / m², and the target scenario of Jinko Kc200T module, only unlabeled data was collected.

[0030] 2. Multimodal data preprocessing The collected raw data is preprocessed as follows to obtain a standardized source / target domain multimodal dataset: (1) Single-mode data preprocessing Image modal denoising: A nonlocal mean denoising algorithm is used to remove Gaussian noise from visible light images while preserving the surface texture details of components; Algorithm parameters: search window 7×7, similarity window 5×5, Gaussian kernel standard deviation 1.2.

[0031] Infrared thermal imaging temperature calibration: The 0~255 infrared image pixel values ​​are converted into actual temperature values ​​using the built-in blackbody calibration module of the thermal imager. The formula is as follows:

[0032] in , .

[0033] Electrical parameter standardization: Time-series data such as open-circuit voltage and short-circuit current are standardized using Z-score.

[0034] in μ The mean of the training set, σ The standard deviation of the training set; through 3 σ Criteria for removing outliers, i.e., exceeding The data.

[0035] Imputation of missing environmental parameters: For missing values ​​in environmental temperature and irradiance data that account for less than 5%, the nearest time interpolation method is used. For example, if the missing value is at time t, the average value of the values ​​at time t-1 and t+1 is taken.

[0036] (2) Spatiotemporal alignment of multimodal data The following steps are used to achieve time synchronization and spatial mapping of multimodal data: Time alignment / timestamp synchronization: a. Standardize the system time for all data acquisition devices, with millisecond-level accuracy, to ensure consistent data acquisition starting points; b. Timestamp alignment for data from different sampling frequencies: High frame rate images / infrared data: Retain frames with the same timestamp as electrical / environmental parameters, and take 1 image frame and 1 infrared image frame every 1 second; Low sampling rate electrical / environmental parameters: amended to 1Hz using linear interpolation to ensure that each time point, such as t=1s, 2s, ..., Ns, contains four modal data. c. Time synchronization error is controlled within ±0.1 seconds.

[0037] Spatial alignment / coordinate mapping: Using the pixel coordinate system of the image mode as a reference, establish the spatial correspondence between the infrared thermal imaging mode and the image mode: a. The camera intrinsic parameter matrix K and distortion coefficients, including focal length and principal point coordinates, are obtained using the Zhang Zhengyou calibration method; b. Using the extrinsic parameters of the infrared thermal imager and camera, namely the rotation matrix R and the translation vector t, the 256×256 resolution infrared thermal image is mapped to a 512×512 image coordinate system through perspective transformation:

[0038] in( u , v ) represents the image pixel coordinates, ( () represents the three-dimensional coordinates of the infrared thermal image pixels; Electrical and environmental parameters are global features that do not require spatial alignment and are bound to image / infrared data only by timestamps.

[0039] S2. Construction of Multimodal Feature Extraction Network Four sub-networks are constructed to extract multimodal features, with the specific structure as follows: 1. Image Modal Convolutional Feature Extraction Subnetwork An improved ResNet-50 network is used, with the following structure: Figure 2 As shown, the specific configuration is as follows: Input: Preprocessed visible light image, with dimensions uniformly adjusted to 512×512×3; Network layer design: Initial convolutional layer: 7×7 convolution (64 filters, stride 2) → BatchNorm → ReLU → 3×3 max pooling (stride 2); Residual blocks 1-4: contain 3, 4, 6, and 3 residual blocks respectively, and each residual block consists of two 3×3 convolutional layers; The attention mechanism module, added after the third residual block, adopts the SENet structure channel attention mechanism, which outputs the channel weights through global average pooling → fully connected layer (neurons 128 → 64) → Sigmoid activation, and weights the feature maps accordingly. Output: Image spatial features of dimension 16×16×2048, where 16×16 corresponds to the spatial resolution of the image and 2048 is the number of channels.

[0040] 2. Infrared thermal imaging modal feature extraction subnetwork The encoder portion of the U-Net network is used to extract temperature gradient features: Input: Spatially aligned infrared thermal image, with dimensions of 256×256×1; Network layer design: Downsampling path: 3 convolutional blocks, each containing 2 3×3 convolutions → BatchNorm → ReLU, followed by 22×2 max pooling with stride after each layer; Feature dimension changes: 256×256×64 → 128×128×128 → 64×64×256 → 32×32×512; Output: 32×32×512 infrared thermal features.

[0041] 3. Electrical Parameter Temporal Feature Extraction Subnetwork A bidirectional LSTM network is used to capture the dynamic trends of electrical parameters: Input: Standardized electrical parameter time series, length 50~200, this example takes 100, feature dimension 4: open circuit voltage, short circuit current, maximum power point voltage, maximum power point current; Network layer design: (1) Bidirectional LSTM layer: 2 layers, 128 hidden units per layer, dropout=0.3; (2) Output layer: Dimensions are compressed through a fully connected layer (128→64); Output: Time-series trend features, with 64 dimensions.

[0042] 4. Environmental Parameter Feature Extraction Subnetwork A 3-layer fully connected neural network is used: Input: Environmental parameter vector, dimension 3: ambient temperature, irradiance, wind speed; Network layer design: FC1(128)→ReLU→FC2(64)→ReLU→FC3(32) Output: Environmental impact features, 32 dimensions.

[0043] S3. Construction and Training of Cross-Scenario Transfer Learning Models 1. Feature fusion module / modal attention fusion mechanism Calculate the attention weights for each modality feature to achieve dynamic weighted fusion: Attention weight calculation: The weights are determined based on the mutual information entropy (MI) between modal features and fault labels, using the following formula:

[0044] in For the first i Modal features ( i =1: Image, 2: Infrared, 3: Electrical, 4: Environment). Y For fault labels such as hot spots and normal conditions, mutual information entropy (MI) is used. , Y This measures the relevance between features and labels; the higher the value, the higher the weight. Weighted fusion: Flatten the features of each modality and sum them according to their weights.

[0045] in The flattened feature vectors are as follows: image features are flattened to 16×16×2048=524288 dimensions, infrared features are flattened to 32×32×512=524288 dimensions, electrical features are 64 dimensions, and environmental features are 32 dimensions. They are then fused after being uniformly compressed to 1024 dimensions through a fully connected layer. Output: Source / target domain fused features, dimension 1024.

[0046] 2. Domain Alignment Module: Domain Discriminator + Adversarial Training Minimize the distribution difference between the source and target domains through adversarial training: Domain discriminator structure: 3-layer fully connected network, input is 1024-dimensional fused features, output is the domain classification probability (source domain / target domain):

[0047] The process of combat training: A fixed feature fusion module and classifier are used to train a domain discriminator to distinguish source / target domain features, i.e., binary classification cross-entropy loss. With a fixed domain discriminator, the training feature fusion module generates features that the domain discriminator cannot distinguish, thus minimizing the domain classification accuracy. Iterative training was conducted until the accuracy of the domain discriminator approached 50%.

[0048] 3. Fault classification module, using an improved Transformer classifier. A Transformer classifier with four encoder layers is used to achieve joint diagnosis of fault type and location. Input: 1024-dimensional fused features after neighborhood alignment; Encoder layer design: Each encoder layer contains: (1) Hybrid attention mechanism: multi-head self-attention (number of heads = 8, dimension = 128) + modal cross-attention (cross-correlation of image / infrared spatial features and electrical / environmental temporal features); (2) Feedforward Network (FFN): 2 layers of fully connected (1024→2048→1024) + ReLU activation; Classification head: 1 fully connected layer (1024→6) + Softmax, outputting 6 types of fault probabilities, including 5 types of faults and 1 type of normal state.

[0049] 4. Model training parameter settings Loss function: Total loss ,in: Classification loss Cross-entropy loss is used to calculate the difference between the predicted fault type and the true label. Domain alignment loss Adversarial loss, also known as binary cross-entropy, is expressed by the formula: ( Features of the source domain For target domain features, D (for the domain discriminator) Modal alignment loss Maximum mean difference (MMD) loss measures the distribution distance of modal features between the source and target domains; Weighting coefficients: (satisfy ).

[0050] Training configuration: Optimizer: Adam, learning rate 0.001, weight decay 1e-5; Batch size: 32, number of iterations: 200 rounds, early stopping strategy: stop if the accuracy of the validation set does not improve for 10 consecutive rounds; Hardware environment: NVIDIA RTX 3090 GPU, CUDA 11.3 framework.

[0051] S4. Cross-scenario diagnosis of photovoltaic module faults 1. Diagnostic Process The unlabeled multimodal data of photovoltaic modules to be diagnosed in the target scenario will be processed according to the following steps: (1) Multimodal data preprocessing, including denoising, standardization, and spatiotemporal alignment; (2) Input the multimodal feature extraction network to construct the multimodal feature extraction network to obtain the multimodal features of the target domain; (3) Input the cross-scene transfer learning model that has been trained, and output the fault diagnosis results.

[0052] 2. Fault diagnosis result output Fault type determination: Based on the 6 probability vectors output by the classifier, the category with the highest probability is taken as the diagnostic result. The specific determination thresholds are shown in Table 1. Table 1

[0053] Fault location (spatial coordinate calculation): Based on the mapping relationship between the spatial resolution of the 16×16 image modal features and the 512×512 original image, the specific steps are as follows: (1) Extract the spatial region with the highest attention weight from the image features (16×16×2048) (e.g., the feature map with the largest activation value in the i-th row and j-th column). (2) Calculate the coordinate range in the original image:

[0054]

[0055] Where 32 is the scaling ratio between the feature map and the original image: 512 / 16=32; (3) Output the coordinates of the fault location, such as “hot spot fault: (96, 64)-(128, 96)” (unit: pixels).

[0056] 3. Cross-scenario diagnostic effect verification Validation was conducted under the following typical cross-scenario conditions: source scenario: irradiance 1000W / m², temperature 25℃, module model A. Scenario 1 (Illumination Difference): Target scene irradiance 700W / m² (difference 300W / m² ≥ 200W / m²), diagnostic accuracy 92.3%; Scenario 2 (Temperature Difference): Target scene temperature 45℃ (difference ≥ 15℃ ≥ 20℃), diagnostic accuracy 90.7%; Scenario 3 (Component Model Difference): Target scenario component model B (manufacturer different from A), diagnostic accuracy rate 88.5%.

[0057] This embodiment describes in detail the multimodal data acquisition and preprocessing (including spatiotemporal alignment methods), feature extraction network structure, transfer learning model training details, and fault diagnosis result output method. It achieves efficient and accurate fault diagnosis of photovoltaic modules under different operating scenarios, reduces dependence on labeled data for the target scenario, and improves the model's generalization ability and practicality. Those skilled in the art can reproduce this method based on the above parameter settings and steps, demonstrating significant inventiveness and industrial value.

Claims

1. A method for photovoltaic module fault cross-scene diagnosis based on multi-modal transfer learning, characterized in that, The method comprises the following steps: S1. Multimodal data acquisition and preprocessing: acquiring labeled multimodal data of photovoltaic modules under a source scene and unlabeled / less labeled multimodal data under a target scene, the multimodal data comprising image modal data, infrared thermal imaging modal data, electrical parameter modal data and environmental parameter modal data; and preprocessing the acquired multimodal data to obtain a standardized source domain multimodal data set and a target domain multimodal data set; S2. Multimodal feature extraction: constructing a multimodal feature extraction network, and performing feature extraction on the source domain multimodal data set and the target domain multimodal data set respectively to obtain source domain multimodal features and target domain multimodal features; the multimodal feature extraction network comprises a convolution feature extraction subnetwork for image modal data, an infrared feature extraction subnetwork for infrared thermal imaging modal data, a time series feature extraction subnetwork for electrical parameter modal data, and an environmental feature extraction subnetwork for environmental parameter modal data; S3. Cross-scene transfer learning model construction and training: constructing a transfer learning model based on domain adaptation, the transfer learning model comprising a feature fusion module, a domain alignment module and a fault classification module; inputting the source domain multimodal features into the feature fusion module for multimodal feature fusion to obtain source domain fusion features; inputting the target domain multimodal features into the feature fusion module for multimodal feature fusion to obtain target domain fusion features; aligning the source domain fusion features and the target domain fusion features in domain distribution through the domain alignment module to obtain domain-aligned source domain features and target domain features; training the fault classification module based on the domain-aligned source domain features and source scene fault labeled labels to obtain a cross-scene fault diagnosis model; S4. Cross-scene diagnosis of photovoltaic module faults: inputting multimodal data of a photovoltaic module to be diagnosed under a target scene into the cross-scene fault diagnosis model after preprocessing in step S1, feature extraction in step S2 and feature fusion in step S3, and outputting a fault diagnosis result.

2. The method of claim 1, wherein, In step S1, the image modal data is a visible light image of the surface of the photovoltaic module, with a resolution of 256x256-1024x1024 pixels; the infrared thermal imaging modal data is an infrared thermal image of the photovoltaic module, containing temperature distribution information, with a temperature range of -20℃-100℃; the electrical parameter modal data includes open-circuit voltage, short-circuit current, maximum power point voltage and maximum power point current of the photovoltaic module string; and the environmental parameter modal data includes environmental temperature, irradiance and wind speed.

3. The method of claim 1, wherein, In step S1, the preprocessing comprises: denoising the image modal data by using a non-local mean denoising algorithm to remove Gaussian noise; temperature calibration of the infrared thermal imaging modal data to convert pixel values into actual temperature values; outlier rejection and Z-score standardization of the electrical parameter modal data; missing value interpolation of the environmental parameter modal data by using a neighboring time point interpolation method; and time stamp synchronization to achieve spatio-temporal alignment of the multimodal data.

4. The method of claim 1, wherein, In step S2, the convolution feature extraction sub-network adopts an improved ResNet-50 network, and an attention mechanism module is added after the third residual block; the infrared feature extraction sub-network adopts an encoder part of a U-Net network to extract temperature gradient features of an infrared thermal image; the time sequence feature extraction sub-network adopts a bidirectional LSTM network, and the input is an electrical parameter time sequence with a length of 50-200; and the environment feature extraction sub-network adopts a 3-layer fully connected neural network, and the number of neurons of each layer is 128, 64 and 32 respectively.

5. The method of claim 1, wherein, In step S3, the feature fusion module adopts a modal attention fusion mechanism, specifically: attention weights of the modal features are calculated, the attention weights are determined based on mutual information entropy of the modal features and the fault label; and the modal features are weighted and summed according to the attention weights to obtain multi-modal fusion features.

6. The method of claim 1, wherein, In step S3, the domain alignment module includes a domain discriminator and an adversarial training unit; the domain discriminator is used for domain classification of the input source domain / target domain fusion features, and the adversarial training unit realizes domain distribution alignment of the source domain fusion features and the target domain fusion features by minimizing the classification accuracy of the domain discriminator; and the domain discriminator adopts a three-layer fully connected neural network, and the activation function is LeakyReLU.

7. The method of claim 1, wherein, In step S3, the training loss function L of the transfer learning model is a weighted sum of a classification loss , a domain alignment loss , and a modal alignment loss , i.e. , where α, β, γ are weight coefficients, and α+β+γ=1; the classification loss uses a cross-entropy loss, the domain alignment loss uses an adversarial loss, and the modal alignment loss uses a maximum mean discrepancy loss.

8. The method of claim 1, wherein, In step S4, the fault diagnosis result includes a fault type and a fault position of the photovoltaic module; the fault type includes at least one of a hot spot fault, a hidden crack fault, a junction box fault, a glass breakage fault and a PID attenuation fault; and the fault position is located by a spatial coordinate in the image modal feature.

9. The method of claim 1, wherein, The source scene and the target scene are different photovoltaic module operation scenes, and include at least one of the following scene differences: different light intensity scenes, different environmental temperature scenes, different component model scenes and different installation angle scenes.

10. The method of claim 1, wherein, In step S3, the fault classification module adopts an improved Transformer classifier, including four encoder layers and one classification head, and the attention mechanism of the encoder layer adopts a hybrid attention mechanism combining multi-head self-attention and modal cross-attention.

Images