Photovoltaic module intelligent fault detection method and system based on multi-modal deep learning

Abstract

The application relates to the photovoltaic operation and maintenance technical field and provides a photovoltaic module intelligent fault detection method based on multi-modal deep learning, which comprises the following steps: acquiring visible light image data, infrared thermal imaging data and electrical performance data of a photovoltaic module; respectively extracting features of the data to obtain visible light features, infrared thermal features and electrical features; performing fusion processing on the three features based on a cross-modal attention fusion mechanism to obtain multi-modal fusion features; inputting the multi-modal fusion features into a sparse gated hybrid expert network for fault reasoning to obtain a fault recognition result; and performing fault type determination and severity assessment based on the fault recognition result to obtain a fault detection result. The technical problems of low fault detection precision and poor adaptability to new faults in the prior art are solved.

Description

Technical Field

[0001] This application relates to the field of photovoltaic operation and maintenance technology, and in particular to a method and system for intelligent fault detection of photovoltaic modules based on multimodal deep learning. Background Technology

[0002] With the continuous increase in the installed capacity of photovoltaic power plants, monitoring the operational status of photovoltaic modules is of great significance for ensuring the power generation efficiency and asset safety of the power plants. Currently, photovoltaic module fault detection mainly relies on manual inspection or automatic detection technology based on a single mode. Manual inspection is inefficient and easily affected by subjective factors. Single-mode data often cannot fully characterize the complex fault characteristics of photovoltaic modules. For example, relying solely on visible light images is insufficient to detect internal hidden faults such as hot spots and PID effects, and the technology has poor adaptability to new fault types, resulting in low fault detection accuracy and a high false alarm rate. Therefore, there is an urgent need for a method to solve the technical problems of low fault detection accuracy and poor adaptability to new fault types in existing technologies. Summary of the Invention

[0003] This application provides a method and system for intelligent fault detection of photovoltaic modules based on multimodal deep learning, which solves the technical problems of low fault detection accuracy and poor adaptability to new faults in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for intelligent fault detection of photovoltaic modules based on multimodal deep learning is provided, comprising: acquiring visible light image data, infrared thermal imaging data, and electrical performance data of photovoltaic modules; performing module region segmentation, texture feature extraction, and structural feature extraction on the visible light image data to obtain visible light features; performing temperature calibration and thermal anomaly feature extraction on the infrared thermal imaging data to obtain infrared thermal features; extracting electrical performance anomaly features from the electrical performance data to obtain electrical features; fusing the visible light features, infrared thermal features, and electrical features based on a cross-modal attention fusion mechanism to obtain multimodal fused features; inputting the multimodal fused features into a sparse gated hybrid expert network, selecting a target expert network for fault reasoning, and obtaining fault identification results; the target expert network includes a thermal fault expert network, an electrical fault expert network, a structural fault expert network, and a contamination fault expert network; and determining the fault type and assessing the fault severity based on the fault identification results to obtain photovoltaic module fault detection results.

[0005] In conjunction with the first aspect mentioned above, in one possible implementation, after obtaining the photovoltaic module fault detection results, the method further includes: acquiring few-sample fault data corresponding to the newly added fault type, the few-sample fault data including visible light image data, infrared thermal imaging data, and electrical performance data corresponding to the newly added fault type; constructing a feature sample set for the newly added fault type based on the few-sample fault data, and updating the parameters of the target expert network and the gated network in the sparse gated hybrid expert network through transfer learning; and performing feature matching and fault identification for the newly added fault type based on the updated sparse gated hybrid expert network.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, visible light image data is segmented into component regions, and texture and structural features are extracted to obtain visible light features. This includes: segmenting the photovoltaic component region in the visible light image data using a component segmentation network to obtain a component region mask image; filtering the visible light image data based on the component region mask image to obtain a target component image; inputting the target component image into a texture feature extraction network to extract texture information from the target component image through convolutional features to obtain texture features; inputting the visible light image data into a structural feature extraction network to perform structural analysis on edge, crack, and deformation information in the visible light image data to obtain structural features; and performing feature concatenation and dimension mapping on the texture and structural features to obtain visible light features.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, infrared thermal imaging data is calibrated for temperature and thermal anomaly features are extracted to obtain infrared thermal features. This includes: performing temperature conversion processing on the pixel grayscale values ​​in the infrared thermal imaging data using preset temperature calibration parameters to obtain a temperature matrix; performing temperature normalization processing on the temperature matrix to obtain a normalized thermal image; segmenting the hotspot regions in the normalized thermal image using a Thermal-UNet thermal anomaly segmentation network to obtain hotspot features; performing gradient analysis on the normalized thermal image based on the Sobel and Laplacian operators to obtain temperature gradient features; extracting the thermal distribution statistical features of the temperature matrix, including temperature mean, temperature standard deviation, temperature skewness, and temperature kurtosis; and fusing the hotspot features, temperature gradient features, and thermal distribution statistical features to obtain the infrared thermal features.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, electrical performance anomaly features are extracted from the electrical performance data to obtain electrical features. This includes: extracting the operating current data, operating voltage data, and output power data of the photovoltaic module within a preset time period based on the electrical performance data, and constructing electrical time-series data; performing outlier removal and missing value compensation processing on the electrical time-series data based on preset data cleaning rules to obtain standardized electrical time-series data; calculating current fluctuation features, voltage offset features, and power attenuation features based on the standardized electrical time-series data to obtain electrical statistical features; inputting the standardized electrical time-series data into the IV-Curve-Transformer electrical performance analysis network to perform feature learning on the current-voltage correlation and time-series change relationships in the standardized electrical time-series data to obtain electrical time-series features; and fusing the electrical statistical features and electrical time-series features to obtain the electrical features.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, based on a cross-modal attention fusion mechanism, visible light features, infrared thermal features, and electrical features are fused to obtain multimodal fusion features. This includes: performing feature encoding and dimension mapping on visible light features, infrared thermal features, and electrical features to obtain a modal feature vector of a unified dimension; constructing query vectors, key vectors, and value vectors based on the unified-dimensional modal feature vectors, and calculating the association weights between each modality through scaling dot products to obtain cross-modal attention weights; performing weighted fusion on visible light features, infrared thermal features, and electrical features based on the cross-modal attention weights to obtain cross-modal association features; performing feature enhancement processing on the features within each modality in the unified-dimensional modal feature vector based on a self-attention mechanism to obtain modal enhancement features; and performing residual fusion and feature concatenation processing on the cross-modal association features and modal enhancement features to obtain multimodal fusion features.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, the multimodal fusion features are input into a sparse gated hybrid expert network, and a target expert network is selected for fault reasoning to obtain the fault identification result. This includes: linearly mapping and weighting the multimodal fusion features through the gating network of the sparse gated hybrid expert network to obtain the expert activation weights corresponding to each expert network; sorting and filtering the expert activation weights based on a preset TopK sparse selection rule to obtain the target activation weights of the target expert network; selecting the target expert network from thermal fault expert networks, electrical fault expert networks, structural fault expert networks, and contamination fault expert networks according to the target activation weights; performing fault reasoning on the thermal anomaly features, electrical performance anomaly features, structural anomaly features, and contamination anomaly features in the multimodal fusion features through the target expert network to obtain expert fault results; and performing weighted fusion processing on each expert fault result based on the target activation weights to obtain the fault identification result.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, the photovoltaic module fault detection result is obtained by determining the fault type and assessing the fault severity based on the fault identification result. This includes: calculating the fault classification probability corresponding to each fault type based on the expert fault results and corresponding target activation weights in the fault identification result, and obtaining the fault confidence score; performing fault category matching based on the fault confidence score to obtain the target fault type; extracting thermal anomaly features, electrical performance anomaly features, structural anomaly features, and contamination anomaly features from the fault identification result, and constructing fault severity assessment features; inputting the fault severity assessment features into the severity assessment model, performing regression analysis on the impact of photovoltaic module faults, and obtaining the fault severity value; and determining the photovoltaic module fault detection result based on the target fault type and the fault severity value.

[0012] Secondly, a photovoltaic module intelligent fault detection system based on multimodal deep learning is provided. The system includes: a data acquisition module, a visible light feature extraction module, an infrared feature extraction module, an electrical feature extraction module, a feature fusion module, a fault identification module, and a fault discrimination module. The data acquisition module acquires visible light image data, infrared thermal imaging data, and electrical performance data of the photovoltaic module. The visible light feature extraction module performs module region segmentation, texture feature extraction, and structural feature extraction on the visible light image data to obtain visible light features. The infrared feature extraction module performs temperature calibration and thermal anomaly feature extraction on the infrared thermal imaging data to obtain infrared thermal features. The electrical feature extraction module performs... The system comprises four modules: an extraction module for extracting electrical performance anomalies from electrical performance data to obtain electrical features; a feature fusion module for fusing visible light features, infrared thermal features, and electrical features based on a cross-modal attention fusion mechanism to obtain multimodal fused features; a fault identification module for inputting the multimodal fused features into a sparse gated hybrid expert network, selecting a target expert network for fault reasoning, and obtaining fault identification results; the target expert networks include thermal fault expert networks, electrical fault expert networks, structural fault expert networks, and contamination fault expert networks; and a fault discrimination module for determining the fault type and assessing the severity of the fault based on the fault identification results to obtain photovoltaic module fault detection results.

[0013] In conjunction with the second aspect above, in one possible implementation, the visible light feature extraction module is further configured to: segment the photovoltaic module region in the visible light image data using a component segmentation network to obtain a component region mask image; perform background filtering on the visible light image data based on the component region mask image to obtain a target component image; input the target component image into a texture feature extraction network to extract texture information from the target component image through convolutional feature extraction to obtain texture features; input the visible light image data into a structural feature extraction network to perform structural analysis on edge information, crack information, and deformation information in the visible light image data to obtain structural features; and perform feature concatenation and dimension mapping on the texture features and structural features to obtain visible light features.

[0014] This application provides a method and system for intelligent fault detection of photovoltaic modules based on multimodal deep learning. By fusing visible light images, infrared thermal imaging, and electrical performance data, it utilizes a cross-modal attention mechanism to mine complementary features across multiple modalities, thus solving the problem of missed detections caused by incomplete representation of single-modal information. Through a sparse-gated hybrid expert network, it designs dedicated expert networks for different fault types to perform accurate reasoning, solving the problem of difficult identification of coupled multi-fault systems and improving fault detection accuracy. Through a few-sample meta-learning framework, it achieves rapid adaptation to novel faults and reduces model iteration costs.

[0015] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0016] Figure 1 A system architecture diagram of a photovoltaic module intelligent fault detection system based on multimodal deep learning provided in this application embodiment; Figure 2 A flowchart illustrating a photovoltaic module intelligent fault detection method based on multimodal deep learning, provided in an embodiment of this application; Figure 3A flowchart illustrating another intelligent fault detection method for photovoltaic modules based on multimodal deep learning provided in this application embodiment; Figure 4 A flowchart illustrating another intelligent fault detection method for photovoltaic modules based on multimodal deep learning provided in this application embodiment; Figure 5 A flowchart illustrating another intelligent fault detection method for photovoltaic modules based on multimodal deep learning provided in this application embodiment; Figure 6 This is a flowchart illustrating another intelligent fault detection method for photovoltaic modules based on multimodal deep learning, provided in an embodiment of this application. Detailed Implementation

[0017] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0018] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0019] The intelligent fault detection method for photovoltaic modules based on multimodal deep learning provided in this application embodiment can be applied to, for example... Figure 1 The photovoltaic module intelligent fault detection system shown is based on multimodal deep learning. Figure 1 As shown, the system includes: a data acquisition module 101, a visible light feature extraction module 102, an infrared feature extraction module 103, an electrical feature extraction module 104, a feature fusion module 105, a fault identification module 106, and a fault discrimination module 107. The system includes the following modules: a data acquisition module 101, used to acquire visible light image data, infrared thermal imaging data, and electrical performance data of the photovoltaic module; a visible light feature extraction module 102, used to perform component region segmentation, texture feature extraction, and structural feature extraction on the visible light image data to obtain visible light features; an infrared feature extraction module 103, used to perform temperature calibration and thermal anomaly feature extraction on the infrared thermal imaging data to obtain infrared thermal features; an electrical feature extraction module 104, used to extract electrical performance anomaly features from the electrical performance data to obtain electrical features; a feature fusion module 105, used to fuse visible light features, infrared thermal features, and electrical features based on a cross-modal attention fusion mechanism to obtain multimodal fused features; a fault identification module 106, used to input the multimodal fused features into a sparse gated hybrid expert network, select a target expert network for fault reasoning, and obtain fault identification results; the target expert networks include thermal fault expert networks, electrical fault expert networks, structural fault expert networks, and contamination fault expert networks; and a fault discrimination module 107, used to determine the fault type and assess the fault severity based on the fault identification results to obtain photovoltaic module fault detection results.

[0020] To address the technical problems of low fault detection accuracy and poor adaptability to novel faults in existing technologies, this application provides an intelligent fault detection method for photovoltaic modules based on multimodal deep learning. Figure 2 This is a flowchart illustrating the intelligent fault detection method for photovoltaic modules based on multimodal deep learning provided in the embodiments of this application, as shown below. Figure 2 As shown, the method includes: S201. Acquire visible light image data, infrared thermal imaging data, and electrical performance data of photovoltaic modules.

[0021] Visible light image data is used to characterize the texture, color, and physical structure information of the photovoltaic module surface, and is usually collected by a high-resolution RGB camera mounted on a drone or a fixed monitoring camera; infrared thermal imaging data is used to characterize the temperature distribution of the photovoltaic module and is collected by an infrared thermal imager; electrical performance data is used to characterize the power generation performance of the photovoltaic module, including time-series data such as operating current, operating voltage, and output power, and is obtained by a module-level optimizer or data acquisition device.

[0022] In this embodiment, the system synchronously acquires data from the three modalities mentioned above through a preset acquisition interface. Specifically, visible light image data can be a single image or a keyframe from a video stream, infrared thermal imaging data includes the mapping relationship between pixel grayscale values ​​and temperature, and electrical performance data is continuous time-series data.

[0023] It should be noted that, in order to ensure the time alignment of the data, the system will calibrate the timestamps of the collected data to ensure that the three modal data correspond to the component state at the same time.

[0024] S202. Perform component region segmentation, texture feature extraction, and structural feature extraction on the visible light image data to obtain visible light features.

[0025] Among them, visible light features are deep feature vectors used to characterize photovoltaic modules in the visual dimension. Since the original image often contains background interference such as weeds, ground, and supports, direct feature extraction will introduce noise, so segmentation processing is required first.

[0026] In this embodiment, the step specifically includes: First, the photovoltaic module region in the visible light image data is segmented using a module segmentation network to obtain a module region mask image. Specifically, the module segmentation network uses a UNet++ network, which, through a nested dense skip connection structure, can more accurately capture module edge information, achieving a segmentation accuracy of over 99%. Next, the visible light image data is filtered for background based on the module region mask image to obtain the target module image, i.e., the mask and the original image are subjected to a Hadamard product operation to remove background pixels. Subsequently, the target module image is input into a texture feature extraction network to extract texture features from the texture information in the target module image through convolutional features. The texture feature extraction network uses a ResNet50 network, utilizing its residual structure to extract surface texture features such as snail trails, contamination, and damage. Simultaneously, the visible light image data is input into a structural feature extraction network to perform structural analysis on the edge information, crack information, and deformation information in the visible light image data to obtain structural features. The structural feature extraction network employs the PV-EdgeNet network, which is specifically optimized for the edge and crack characteristics of photovoltaic modules, effectively identifying structural faults such as hidden cracks and module delamination. Finally, the texture and structural features are concatenated and mapped to a unified dimensional space to obtain visible light features. Feature concatenation uses the Concatenate operation, while dimensional mapping is performed through fully connected layers to map the feature vectors to a unified dimensional space.

[0027] It should be noted that this step extracts and merges texture features and structural features separately because texture features focus on surface dirt and material changes, while structural features focus on geometric shape and physical damage. The two complement each other and can more comprehensively characterize the state of the component.

[0028] Based on the above steps, this embodiment effectively eliminates background interference and accurately captures the texture details and structural morphology of the component surface through segmentation denoising and dual-path feature extraction, thereby improving the expressive power and robustness of visible light features.

[0029] S203. Perform temperature calibration and thermal anomaly feature extraction on the infrared thermal imaging data to obtain infrared thermal features.

[0030] Infrared thermal features are deep feature vectors used to characterize abnormal temperature distribution in photovoltaic modules. Since the raw data output by infrared thermal imagers is usually pixel grayscale values ​​rather than actual physical temperatures, and is easily affected by ambient temperature, calibration and normalization processing are required.

[0031] In this embodiment, the step specifically includes: First, based on the pixel grayscale values ​​in the infrared thermal imaging data, a temperature conversion process is performed on the pixel grayscale values ​​using preset temperature calibration parameters to obtain a temperature matrix. The temperature conversion process satisfies the following formula: ,in, coordinates The actual temperature at that location coordinates The pixel grayscale value at that location, This is the temperature scaling factor. This is the temperature offset coefficient. and The temperature was determined through factory calibration experiments. Next, temperature normalization was performed based on the temperature matrix to obtain a normalized thermal image. Normalization eliminated overall environmental temperature differences and highlighted localized abnormal temperature differences. Subsequently, the Thermal-UNet thermal anomaly segmentation network was used to segment hotspot regions in the normalized thermal image, obtaining hotspot features. The Thermal-UNet network is optimized for the single-channel characteristics of thermal images, enabling accurate segmentation of hotspot regions. Simultaneously, gradient analysis was performed on the normalized thermal image using the Sobel and Laplacian operators to obtain temperature gradient features, used to identify hidden faults such as uneven temperature distribution and abnormal heat dissipation at edges. Furthermore, the thermal distribution statistical features of the temperature matrix were extracted, including temperature mean, temperature standard deviation, temperature skewness, and temperature kurtosis, used to quantify the global thermal distribution state. Finally, the hotspot features, temperature gradient features, and thermal distribution statistical features were fused to obtain infrared thermal features. The fusion process can employ feature stitching or attention weighting.

[0032] It should be noted that this step not only extracts the hot spot features in the spatial dimension, but also extracts gradient features and statistical features, quantifying the thermal state of the component from local details to global distribution in all aspects, effectively avoiding the omission of early latent thermal faults by single hot spot segmentation.

[0033] Based on the above steps, this embodiment eliminates equipment and environmental errors through physical calibration and multi-dimensional feature extraction, accurately quantifies the thermal state of components, and provides reliable data support for thermal fault identification.

[0034] S204. Extract electrical performance anomaly features from electrical performance data to obtain electrical features.

[0035] Among them, electrical features are deep feature vectors used to characterize the variation patterns of the electrical performance of photovoltaic modules. Electrical performance data is usually time series data, containing parameters such as current, voltage, and power, which are difficult to effectively learn from their inherent relationships when directly input into the network.

[0036] In this embodiment, based on electrical performance data, the operating current, operating voltage, and output power data of the photovoltaic module within a preset time period are extracted, and electrical time-series data is constructed. The preset time period can be set according to actual needs, such as the past 24 hours or one week. Then, based on preset data cleaning rules, outlier removal and missing value compensation are performed on the electrical time-series data to obtain standardized electrical time-series data. Outlier removal can use the 3σ principle or box plot method, and missing value compensation can use linear interpolation or mean filling. Subsequently, based on the standardized electrical time-series data, current fluctuation characteristics, voltage offset characteristics, and power decay characteristics are calculated to obtain electrical statistical characteristics. These statistical characteristics can intuitively reflect the power generation performance stability of the module. At the same time, the standardized electrical time-series data is input into the IV-Curve-Transformer electrical performance analysis network to perform feature learning on the current-voltage correlation and time-series change relationship in the standardized electrical time-series data to obtain electrical time-series features. The IV-Curve-Transformer network uses a self-attention mechanism to capture long-distance dependencies between electrical parameters and mine deep features of electrical faults such as PID decay and bypass diode failure. Finally, the electrical statistical features and electrical time-series features are fused to obtain the electrical features.

[0037] It should be noted that this step combines manually designed statistical features with time-series features extracted by deep learning, which not only preserves expert knowledge but also uncovers hidden patterns in the data, thereby improving the accuracy of electrical fault identification.

[0038] Based on the above steps, this step, through data cleaning and dual-path feature learning, deeply explores the inherent correlation and temporal variation patterns of electrical data, thereby improving the identification accuracy of electrical faults such as PID attenuation and diode failure.

[0039] S205. Based on the cross-modal attention fusion mechanism, visible light features, infrared thermal features, and electrical features are fused to obtain multimodal fused features.

[0040] Among them, the cross-modal attention fusion mechanism is used to solve the problems of large differences in feature dimensions, information redundancy and insufficient complementarity among different modalities. It achieves deep fusion of feature level and decision level by adaptively learning the association weights between modalities.

[0041] In this embodiment, the system performs feature encoding and dimension mapping on visible light features, infrared thermal features, and electrical features to obtain a unified-dimensional modal feature vector. Subsequently, a query vector, key vector, and value vector are constructed based on the unified-dimensional modal feature vector, and the association weights between each modality are calculated by scaling the dot product to obtain cross-modal attention weights. Then, the visible light features, infrared thermal features, and electrical features are weighted and fused based on the cross-modal attention weights to obtain cross-modal association features. Simultaneously, feature enhancement processing is performed on the intra-modal features within the unified-dimensional modal feature vector based on a self-attention mechanism to obtain modal enhancement features. Finally, residual fusion and feature concatenation processing are performed on the cross-modal association features and modal enhancement features to obtain multimodal fusion features.

[0042] Based on the above steps, this embodiment adaptively fuses multimodal information through an attention mechanism, maximizing information utilization and solving the problem of missed detection caused by incomplete representation of single modal information.

[0043] S206. Input the multimodal fusion features into a sparse gated hybrid expert network, select the target expert network for fault reasoning, and obtain the fault identification result.

[0044] The sparse-gated hybrid expert network comprises multiple expert networks and a gating network, used for specialized reasoning for different types of faults, improving the model's ability to identify complex faults. The expert networks include thermal fault expert networks, electrical fault expert networks, structural fault expert networks, and contamination fault expert networks.

[0045] In this embodiment, the system uses a sparsely gated hybrid expert network to perform linear mapping and weight calculation on the multimodal fusion features, obtaining the expert activation weights corresponding to each expert network. Next, based on a preset TopK sparse selection rule, the expert activation weights are sorted and selected to obtain the target activation weights of the target expert network. For example, setting K=2 activates only the two expert networks with the highest weights to reduce computational overhead. Based on the target activation weights, a target expert network is selected from thermal fault expert networks, electrical fault expert networks, structural fault expert networks, and contamination fault expert networks. Through the target expert network, fault inference is performed on the thermal anomaly features, electrical performance anomaly features, structural anomaly features, and contamination anomaly features in the multimodal fusion features, obtaining expert fault results. Finally, the expert fault results are weighted and fused based on the target activation weights to obtain the fault identification result.

[0046] Based on the above steps, this step activates only the relevant expert network through a sparse gating mechanism, which reduces computational overhead while ensuring recognition accuracy and achieves accurate adaptation to multiple types of faults.

[0047] S207. Based on the fault identification results, determine the fault type and assess the fault severity to obtain the photovoltaic module fault detection results.

[0048] The fault detection results include not only the fault category but also a quantified severity value, providing a precise basis for operation and maintenance decisions.

[0049] In this embodiment, the system calculates the fault classification probability for each fault type based on the expert fault results and corresponding target activation weights in the fault identification results, thus obtaining the fault confidence level. Next, fault category matching is performed based on the fault confidence level to obtain the target fault type. Simultaneously, thermal anomaly features, electrical performance anomaly features, structural anomaly features, and contamination anomaly features are extracted from the fault identification results, and fault severity assessment features are constructed. These fault severity assessment features are input into a severity assessment model to perform regression analysis on the impact of photovoltaic module faults, obtaining a fault severity value. Finally, based on the target fault type and the fault severity value, the photovoltaic module fault detection result is determined.

[0050] This application embodiment acquires multimodal data and extracts features separately, fully explores complementary information between different modalities by utilizing a cross-modal attention fusion mechanism, and combines a sparse gated hybrid expert network to perform specialized reasoning for different fault types. This solves the technical problems of low fault detection accuracy and poor adaptability to new faults in the prior art, and achieves high-precision and high-robust fault detection.

[0051] In one possible implementation, combining Figure 2 ,like Figure 3 As shown, after S207, the method provided in this application embodiment further includes the following S301 to S303: S301. Obtain the few-sample fault data corresponding to the newly added fault type. The few-sample fault data includes visible light image data, infrared thermal imaging data and electrical performance data corresponding to the newly added fault type.

[0052] Among them, newly added fault types refer to fault categories that did not appear during the model training phase but were discovered for the first time in actual operation and maintenance, such as special texture changes caused by the aging of new materials or unknown electrical characteristic anomalies caused by the introduction of new equipment. Few-sample fault data refers to a small number of labeled samples collected for this newly added fault type, usually between 1 and 10.

[0053] In this embodiment, when maintenance personnel discover a suspected new type of fault during inspection, they can manually label the fault type through a human-computer interaction interface. The system automatically retrieves visible light images, infrared thermal imaging data, and electrical performance data before and after the fault occurred to construct a few-sample fault dataset. The system preprocesses the collected data, including image cropping, temporal alignment, and data augmentation operations such as random rotation and color jitter, to expand sample diversity and improve the robustness of subsequent model training.

[0054] It should be noted that, due to the adoption of the contrastive learning pre-training framework, the model already has a powerful ability to extract general features. Therefore, only a very small number of samples are needed to complete the adaptation to new categories, which effectively solves the problems of poor adaptability to new faults and high annotation costs of traditional deep learning models.

[0055] S302. Construct a feature sample set for new fault types based on few sample fault data, and update the parameters of the target expert network and the gated network in the sparse gated hybrid expert network through transfer learning.

[0056] The feature sample set refers to the set of deep feature vectors obtained by inputting a small number of fault data samples into the feature extraction network. Transfer learning refers to a training strategy that uses the existing knowledge of a pre-trained model to fine-tune it for a new task.

[0057] In this embodiment, the system first inputs a small sample of fault data into the feature extraction network of Embodiment 1 to extract corresponding visible light features, infrared thermal features, and electrical features, and fuses them into a multimodal feature vector to construct a feature sample set. Next, a prototype network algorithm is used to calculate the feature prototype of the newly added fault type. The mean of all sample feature vectors under this category is calculated as the prototype representation of that category. Subsequently, the parameters of the sparse gated hybrid expert network are updated through transfer learning. To maintain the model's ability to identify existing fault types and avoid catastrophic forgetting, the system adopts a parameter freezing strategy, updating only the linear mapping layer parameters of the gated network and the output layer parameters of the target expert network most relevant to the newly added fault type, while freezing the parameters of the backbone network and other expert networks. For example, if the newly added fault is a special type of thermal fault, only the relevant parameters of the thermal fault expert network and the gated network are updated.

[0058] It should be noted that by freezing the backbone network and only fine-tuning specific parameters, not only is the consumption of computing resources and training time significantly reduced, but the stability of the model on old categories is also guaranteed, enabling the model to quickly adapt to new things and retain old knowledge for a long time.

[0059] S303. Based on the updated sparse gated hybrid expert network, feature matching and fault identification are performed on the newly added fault types.

[0060] Feature matching refers to calculating the similarity between the features of the sample to be tested and the prototype features of each category, while fault identification is based on the similarity to determine the fault category of the sample to be tested.

[0061] In this embodiment, when a new test sample is input into the updated sparse gated hybrid expert network, the system first extracts its multimodal fusion features, and then calculates the Euclidean distance between these features and the feature prototypes of each category (including existing and newly added categories). Using a distance metric function, the test sample is classified into the fault category corresponding to the nearest feature prototype. The system calculates the distance between the test sample's features and the feature prototype of the newly added fault type. If this distance is less than a preset threshold or is the minimum value among all categories, the sample is determined to belong to the newly added fault type. Simultaneously, the gated network adaptively adjusts the activation weights of the expert network based on the input features, ensuring that the newly added fault type can be correctly routed to the most matching expert network for refined inference.

[0062] As an example, assuming the newly added fault type is "diode micro-short circuit," the system only needs to collect three infrared and visible light images of this fault, along with the corresponding electrical data. The above steps are then used to construct a feature prototype and update the model. During subsequent inspections, when encountering similar faults, the updated model can accurately identify the fault.

[0063] This application's embodiments, by introducing few-shot learning and transfer learning mechanisms, enable the model to adapt to novel faults quickly and at low cost, thereby improving the scalability of the fault detection system.

[0064] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S205 can be implemented by the following S401 to S405, which are explained in detail below: S401. Perform feature encoding and dimension mapping on visible light features, infrared thermal features, and electrical features to obtain a modal feature vector with a unified dimension.

[0065] Feature encoding is used to map feature vectors from different modalities to the same feature space, while dimensionality mapping is used to eliminate the impact of differences in feature dimensionality between different modalities on the fusion process.

[0066] In this embodiment, since visible light features, infrared thermal features, and electrical features originate from different feature extraction networks, their feature dimensions and distribution spaces differ significantly. For example, visible light features may focus on texture and structural information, resulting in higher dimensionality; electrical features may focus on temporal variations, resulting in lower dimensionality. The system performs linear transformations on these three types of features through fully connected layers, mapping them to a unified dimensional space, for example, a unified 1024-dimensional feature vector. For visible light features, linear transformation matrix one is used for mapping transformation; for infrared thermal features, linear transformation matrix two is used; and for electrical features, linear transformation matrix three is used. After mapping, the three modal features have the same dimension, facilitating subsequent attention calculation.

[0067] It should be noted that the feature encoding process can also introduce positional encoding or modality type encoding to retain the positional information or modal attribute information of the features, further improving the model's ability to distinguish the source of the features.

[0068] S402. Construct query vector, key vector, and value vector based on modal feature vectors of a unified dimension, and calculate the association weights between modalities by scaling the dot product to obtain cross-modal attention weights.

[0069] Among them, query vector, key vector, and value vector are the core concepts in the attention mechanism. The query vector is used to represent the query intent of the current modality, the key vector is used to represent the index information of other modalities, and the value vector is used to represent the content information of other modalities.

[0070] In this embodiment, the system constructs Query, Key, and Value vectors for each modal feature. Taking the visible light modality as an example, the mapped feature vectors are respectively mapped to the weight matrix. Multiplying them together yields the Query vector for the visible light modes. Key vector and Value vector Similarly, the construction of infrared thermal and electrical modes... as well as Subsequently, cross-modal attention weights are calculated. When calculating the attention weights of the visible light mode on the infrared thermal mode, the visible light query vector is used. Key vector with infrared heat The scaling dot product operation is performed. Through the attention calculation formula, the system can adaptively learn the degree of correlation between different modalities. For example, when crack features appear in a visible light image, the model may assign higher weights to the corresponding temperature anomaly features in the infrared thermal features.

[0071] It should be noted that the attention mechanism is not limited to the scaled dot product form, and can also be replaced by additive attention or other variants, as long as adaptive learning of intermodal weights can be achieved. This embodiment does not impose any restrictions on this.

[0072] S403. Based on cross-modal attention weights, visible light features, infrared thermal features, and electrical features are weighted and fused to obtain cross-modal correlation features.

[0073] Among them, cross-modal correlation features refer to feature representations that integrate complementary information from other modes, which can more comprehensively characterize the fault state of photovoltaic modules.

[0074] In this embodiment, the system performs a weighted summation of the Value vectors of other modalities based on the calculated attention weights to obtain the cross-modal association features of the current modality. These cross-modal association features not only integrate complementary information from other modalities but also retain key features of the current modality through residual connections, achieving bidirectional information flow and enhancement.

[0075] S404. Based on the self-attention mechanism, feature enhancement processing is performed on the internal features of each modality in the modal feature vector of the same dimension to obtain modal enhancement features.

[0076] Modality enhancement features refer to feature representations that enhance key features within a modality through a self-attention mechanism. Cross-modality association features focus on mining complementary information between modalities, while modality enhancement features focus on mining contextual relationships within a modality.

[0077] In this embodiment, the system, while fusing cross-modal information, introduces a self-attention mechanism to enhance features within a single modality. For visible light feature vectors, the system calculates the self-attention weights between each feature element within the vector, strengthening the correlation between texture and structural features through weighted summation, such as enhancing the contrast between crack edges and surrounding textures. Similarly, self-attention enhancement is applied to infrared thermal and electrical features respectively. Through the self-attention mechanism, the model can more accurately capture key fault features within a modality and suppress irrelevant noise interference.

[0078] It should be noted that cross-modal correlation features and modality enhancement features are complementary. The former solves the problem of "how different modalities complement each other", while the latter solves the problem of "how to enhance within a single modality". Combining the two can maximize the utilization of multimodal information.

[0079] S405. Perform residual fusion and feature splicing on cross-modal correlation features and modal enhancement features to obtain multimodal fusion features.

[0080] Among them, residual fusion is used to fuse the enhanced features with the original features, and feature concatenation is used to merge feature vectors from different modalities into a single long vector.

[0081] In this embodiment, the system performs residual connections between cross-modal correlation features and modal enhancement features to further fuse inter-modal and intra-modal feature information. Subsequently, the fused features of the three modalities are concatenated, for example, visible light fusion features, infrared thermal fusion features, and electrical fusion features are concatenated dimensionally to form a multimodal fusion feature vector containing global information. This feature vector will serve as input to a subsequent sparse gated hybrid expert network for fault reasoning.

[0082] This application's embodiments adaptively learn the correlation weights between modalities through a cross-modal attention mechanism, and enhance the internal features of modalities by combining a self-attention mechanism, thereby achieving deep fusion between the feature level and the decision level, maximizing the utilization rate of multimodal information, and effectively solving the problem of missed detection caused by incomplete representation of single modal information.

[0083] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 5 As shown, the above S206 can be implemented by the following S501 to S505, which are explained in detail below: S501. Through the gating network of the sparse gated hybrid expert network, the multimodal fusion features are linearly mapped and weights are calculated to obtain the expert activation weights corresponding to each expert network.

[0084] The gating network is the core scheduling component of the sparse gating hybrid expert network, used to adaptively determine which expert networks to activate based on the characteristics of the input features. The expert activation weights characterize the degree of matching or contribution of each expert network to the current input sample.

[0085] In this embodiment, the gated network receives multimodal fusion features as input and performs feature mapping and weight calculation through a simple linear layer. The gated network contains a learnable weight matrix W and a bias vector b. Through a linear transformation, it maps the input features into an N-dimensional vector, where N is the number of expert networks; in this embodiment, N=4. This linear mapping process can be understood as projecting the high-dimensional fusion features into the decision space of the expert networks. Subsequently, the mapped vector is subjected to Softmax normalization, ensuring that the sum of all expert activation weights is 1, thereby obtaining the initial activation probability distribution corresponding to each expert network.

[0086] It should be noted that the design of gated networks follows the principle of "lightweighting". Its number of parameters is much smaller than that of expert networks, which can complete routing decisions with extremely low computational overhead and ensure the overall inference efficiency of the system.

[0087] S502. Based on the preset TopK sparse filtering rules, the expert activation weights are sorted and filtered to obtain the target activation weights of the target expert network.

[0088] The TopK sparse selection rule refers to a selection strategy that retains only the top K expert networks with the highest activation weights, and resets the weights of the remaining expert networks to zero. The target activation weights are the non-zero weights retained after sparsification.

[0089] In this embodiment, to reduce computational resource consumption and improve inference speed, the system introduces a TopK mechanism to sparsify the output of the gating network. K=2 is set, meaning that only the two expert networks with the highest weights are activated in each inference iteration. The system sorts the four expert activation weights in descending order, retains the top two weights, and sets the remaining two weights to zero. This sparse selection mechanism ensures that for any input sample, the system only needs to compute the forward propagation process of two expert networks. Compared to the full expert activation mode, the computational load is reduced by half, improving the real-time performance of inference on edge computing devices.

[0090] It should be noted that the value of K can be adjusted according to the actual hardware computing power and accuracy requirements. For example, K=3 or K=4 can be set on cloud servers with sufficient computing power, while K=1 can be set on edge devices with limited computing power. This embodiment does not impose any restrictions on this.

[0091] S503. Based on the target activation weight, select the target expert network from the thermal fault expert network, electrical fault expert network, structural fault expert network, and pollution fault expert network.

[0092] Among them, the target expert network refers to the expert network selected by the gating network and participating in the inference of the current sample. The thermal fault expert network is specifically used to handle faults such as hot spots and abnormal temperatures; the electrical fault expert network is specifically used to handle faults such as PID decay and diode failure; the structural fault expert network is specifically used to handle faults such as microcracks and damage; and the contamination fault expert network is specifically used to handle faults such as dust obstruction and bird droppings.

[0093] In this embodiment, the system selects the appropriate network from four expert networks as the target expert network based on the index corresponding to the non-zero weight. For example, if the input sample is a typical hot spot fault, the weights calculated by the gated network may show that the thermal fault expert network and the fouling fault expert network have the highest weights, and the system selects these two networks as the target expert networks. This division of labor simulates the mechanism of human expert consultation, with experts from different fields responsible for diagnosing different types of faults, thereby ensuring the professionalism and accuracy of the diagnosis.

[0094] As an example, a thermal fault expert network can adopt a Temperature-CNN+LSTM architecture, where CNN is used to extract spatial features from infrared thermal images and LSTM is used to capture temporal temperature variation features, thereby accurately identifying faults such as hot spots and heat dissipation failures; an electrical fault expert network can be based on an IV-Curve-Transformer architecture, integrating a photovoltaic electrical physics model to specifically identify electrical faults such as PID attenuation and series mismatch; a structural fault expert network can adopt a Fracture-CNN+Graph-NN architecture, modeling the topological relationships of component structures to identify faults such as microcracks and glass breakage; and a contamination fault expert network can adopt a multi-scale CNN architecture, integrating visible light and infrared features to identify occlusion-related faults.

[0095] S504. Through the target expert network, fault reasoning is performed on the thermal anomaly features, electrical performance anomaly features, structural anomaly features, and fouling anomaly features in the multimodal fusion features to obtain expert fault results.

[0096] Among them, the expert fault result refers to the fault category probability vector or feature vector output by each target expert network.

[0097] In this embodiment, the selected target expert networks receive multimodal fusion features in parallel and perform forward propagation calculations independently. Since the multimodal fusion features contain information from multiple modalities such as visible light, infrared, and electrical, each expert network automatically extracts a subset of features relevant to its own professional domain based on its network structure. For example, the thermal fault expert network focuses on feature channels related to temperature distribution in the fusion features, suppressing interference from irrelevant features such as texture and color, thereby outputting a probability distribution for the thermal fault category. Each target expert network independently outputs its own inference results, providing a foundation for subsequent weighted fusion.

[0098] It should be noted that through the specialized design of the expert network, the model can learn more discriminative feature representations for different fault types, avoiding the problem of feature confusion when a single model faces multiple types of faults, and significantly improving the recognition accuracy in complex fault scenarios.

[0099] S505. Based on the target activation weight, the fault results of each expert are weighted and fused to obtain the fault identification result.

[0100] Weighted fusion processing refers to linearly combining the outputs of each expert network based on the weights of the gating network outputs.

[0101] In this embodiment, the system performs a weighted summation of the expert fault results output by each target expert network and their corresponding target activation weights to obtain the final fault identification result. The fault identification result includes not only the fault category determination but also the confidence score for each type of fault.

[0102] This application embodiment achieves dynamic collaborative reasoning among multiple experts through a sparse gated hybrid expert network. It leverages the professional advantages of expert networks to improve fault identification accuracy while reducing computational overhead through a sparse activation mechanism, thus solving the problem that a single model cannot simultaneously handle multiple types of fault identification and real-time reasoning.

[0103] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 6 As shown, the above S207 can be specifically implemented through the following S601 to S605, which are explained in detail below: S601. Based on the expert fault results and the corresponding target activation weights in the fault identification results, calculate the fault classification probability corresponding to each fault type to obtain the fault confidence.

[0104] Among them, fault confidence refers to the probability value that the system determines that the photovoltaic module belongs to a certain specific fault type, which is used to quantify the credibility of the fault identification result.

[0105] In this embodiment, the system first extracts the fault identification results output by the sparse gated hybrid expert network. This result includes the expert fault results output by the activated target expert network (typically the logits or probability distributions of various fault types) and the target activation weights output by the gated network. The system weights and fuses the expert fault results with the corresponding target activation weights to calculate the final fault classification probability. Assuming the two activated expert networks are a thermal fault expert network and a structural fault expert network, their output fault probability vectors are respectively... and The corresponding target activation weights are respectively and The final probability distribution is calculated using a weighted summation formula. The category corresponding to the highest probability value in this distribution is the initially determined fault type, and this probability value is the fault confidence level. It should be noted that the calculation of the fault confidence level does not rely solely on the output of a single expert, but rather integrates the opinions of multiple experts. This multi-expert collaborative judgment method effectively reduces the risk of misjudgment by a single model and improves the robustness of fault classification.

[0106] Based on the above steps, this embodiment achieves accurate quantification of fault confidence by weighted fusion of the reasoning results of multiple experts, providing a reliable numerical basis for subsequent fault determination.

[0107] S602. Based on the fault confidence level, perform fault category matching to obtain the target fault type.

[0108] Among them, the target fault type refers to the fault category label finally determined by the system, such as hot spot, microcrack, PID decay, etc.

[0109] In this embodiment, the system sets a confidence threshold mechanism to determine the confidence level of a fault. If the fault confidence level is higher than a preset confidence threshold (e.g., 0.85), the corresponding fault category is directly identified as the target fault type; if the fault confidence level is lower than the threshold, it is marked as a "suspected fault" or "low-confidence fault," prompting maintenance personnel to conduct manual review. This threshold determination mechanism can effectively filter out low-quality identification results and ensure the accuracy of the output fault type.

[0110] S603. Extract thermal anomaly features, electrical performance anomaly features, structural anomaly features, and contamination anomaly features from the fault identification results, and construct fault severity assessment features.

[0111] Among them, the fault severity assessment feature refers to the feature vector used to quantify the degree of impact of faults on component performance. Its construction process requires comprehensive consideration of the physical properties and electrical effects of the fault.

[0112] In this embodiment, the system extracts specific features related to fault severity from multimodal fusion features or intermediate layer features of expert networks. For hot spot faults, the system extracts thermal anomaly features such as the highest temperature, average temperature, and hot spot area ratio of the hot spot region; for microcrack faults, the system extracts structural anomaly features such as crack length, width, and orientation; for PID attenuation faults, the system extracts electrical performance anomaly features such as power attenuation rate and current offset. The system standardizes these multi-dimensional features and concatenates them into a unified fault severity assessment feature vector.

[0113] It should be noted that different types of faults correspond to different combinations of severity assessment features. The system adaptively selects relevant feature subsets to construct the severity assessment based on the target fault type, thereby improving the relevance and accuracy of the severity assessment.

[0114] Based on the above steps, this step extracts multi-dimensional fault features and constructs a comprehensive severity assessment feature vector, providing data support for subsequent regression analysis.

[0115] S604. Input the fault severity assessment features into the severity assessment model, perform regression analysis on the impact of photovoltaic module faults, and obtain the fault severity value.

[0116] The severity assessment model is a regression model used to map fault characteristics to specific severity values. The fault severity value is usually a continuous number, ranging from 0 to 1 or 1 to 10, with higher values ​​indicating more severe faults.

[0117] In this embodiment, the severity assessment model can employ regression algorithms such as Multilayer Perceptron (MLP) or Support Vector Regression (SVR). The model establishes a mapping relationship between feature vectors and severity values ​​by learning from a large number of fault samples labeled with severity levels. To improve the model's training efficiency and prediction accuracy, the system uses the Smooth L1 loss function for training. The Smooth L1 loss function uses a squared term when the error is small and a linear term when the error is large. Compared to mean squared error (MSE), it is more robust to outliers and can prevent individual extreme samples from affecting the overall performance of the model.

[0118] It should be noted that the prediction of severity values ​​depends not only on the geometric characteristics of the fault (such as crack length) but also on the physical effects of the fault (such as the magnitude of temperature anomalies). This multi-dimensional regression analysis can more realistically reflect the actual harm of the fault to the component performance.

[0119] Based on the above steps, this step quantifies the severity of the fault through a regression model, achieving a leap from qualitative classification to quantitative assessment, and providing more refined data support for operation and maintenance decisions.

[0120] S605. Determine the photovoltaic module fault detection results based on the target fault type and fault severity value.

[0121] The fault detection result is a structured data set containing a wealth of information, including not only the fault category but also a quantified severity value.

[0122] In this embodiment, the system encapsulates the target fault type and fault severity value to generate a final fault detection report. This report may include fault type labels (e.g., "hotspot"), fault confidence (e.g., 0.92), fault severity value (e.g., "Severity Level: 8.5 / 10"), fault location coordinates, and other information. Maintenance personnel can formulate differentiated maintenance strategies based on the severity value; for example, emergency shutdowns can be performed for faults with a severity value higher than 8, while periodic maintenance can be scheduled for faults with a severity value lower than 3.

[0123] This application embodiment achieves accurate fault type determination and quantitative assessment of fault hazard by calculating fault confidence, constructing severity assessment features, and predicting severity values ​​through regression analysis. It solves the problem that traditional detection methods can only output fault categories but cannot quantify the degree of fault hazard, thus improving the decision-making efficiency and accuracy of photovoltaic operation and maintenance.

[0124] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0125] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A method for intelligent fault detection of photovoltaic modules based on multimodal deep learning, characterized in that, include: Acquire visible light image data, infrared thermal imaging data, and electrical performance data of photovoltaic modules; The visible light image data is segmented into component regions, and texture and structural features are extracted to obtain visible light features. The infrared thermal imaging data is calibrated for temperature and thermal anomaly features are extracted to obtain infrared thermal features; The electrical performance data is subjected to electrical performance anomaly feature extraction to obtain electrical features; Based on the cross-modal attention fusion mechanism, the visible light features, the infrared thermal features, and the electrical features are fused to obtain multimodal fused features; The multimodal fusion features are input into a sparse gated hybrid expert network, and a target expert network is selected for fault reasoning to obtain the fault identification result. The target expert network includes a thermal fault expert network, an electrical fault expert network, a structural fault expert network, and a pollution fault expert network. Based on the fault identification results, the fault type is determined and the fault severity is assessed to obtain the photovoltaic module fault detection results.

2. The method according to claim 1, characterized in that, After obtaining the photovoltaic module fault detection results, the method further includes: Acquire few sample fault data corresponding to the newly added fault type, including visible light image data, infrared thermal imaging data and electrical performance data corresponding to the newly added fault type; Based on the few-sample fault data, a feature sample set for new fault types is constructed, and the parameters of the target expert network and the gated network in the sparse gated hybrid expert network are updated through transfer learning. Feature matching and fault identification of new fault types are performed based on the updated sparse gated hybrid expert network.

3. The method according to claim 1, characterized in that, The visible light image data is segmented into component regions, and texture and structural features are extracted to obtain visible light features, including: The photovoltaic module region in the visible light image data is segmented using a component segmentation network to obtain a component region mask image; The visible light image data is filtered for background based on the component region mask image to obtain the target component image; The target component image is input into a texture feature extraction network, and convolutional feature extraction is performed on the texture information in the target component image to obtain texture features; The visible light image data is input into a structural feature extraction network to perform structural analysis on the edge information, crack information and deformation information in the visible light image data, and obtain structural features. The texture features and structural features are spliced ​​and mapped to obtain visible light features.

4. The method according to claim 1, characterized in that, The process of temperature calibration and thermal anomaly feature extraction of the infrared thermal imaging data to obtain infrared thermal features includes: Based on the pixel grayscale values ​​in infrared thermal imaging data, a temperature matrix is ​​obtained by performing temperature conversion processing on the pixel grayscale values ​​through preset temperature calibration parameters. Temperature normalization is performed based on the temperature matrix to obtain a normalized thermal image. The hot spot region in the normalized thermal image is segmented using the Thermal-UNet thermal anomaly segmentation network to obtain hot spot features; Gradient analysis was performed on the normalized thermal image based on the Sobel and Laplacian operators to obtain temperature gradient features. Extract the thermal distribution statistical features of the temperature matrix, which include temperature mean, temperature standard deviation, temperature skewness, and temperature kurtosis. The hot spot features, the temperature gradient features, and the thermal distribution statistical features are fused together to obtain infrared thermal features.

5. The method according to claim 1, characterized in that, The step of extracting electrical performance anomaly features from the electrical performance data to obtain electrical features includes: Based on electrical performance data, the operating current data, operating voltage data, and output power data of photovoltaic modules within a preset time period are extracted, and electrical time series data are constructed. Based on preset data cleaning rules, outlier removal and missing value compensation are performed on the electrical time series data to obtain standardized electrical time series data. Based on the standardized electrical timing data, current fluctuation characteristics, voltage offset characteristics, and power attenuation characteristics are calculated to obtain electrical statistical characteristics. The standardized electrical timing data is input into the IV-Curve-Transformer electrical performance analysis network to perform feature learning on the current-voltage correlation and timing change relationship in the standardized electrical timing data, thereby obtaining electrical timing features. The electrical statistical features and the electrical timing features are fused to obtain the electrical features.

6. The method according to claim 1, characterized in that, The cross-modal attention fusion mechanism fuses the visible light features, the infrared thermal features, and the electrical features to obtain multimodal fused features, including: The visible light features, the infrared thermal features, and the electrical features are subjected to feature encoding and dimension mapping to obtain a modal feature vector with a unified dimension. Based on the modal feature vector of the unified dimension, a query vector, a key vector, and a value vector are constructed, and the correlation weight between each modality is calculated by scaling the dot product to obtain the cross-modal attention weight; Based on the cross-modal attention weights, the visible light features, the infrared thermal features, and the electrical features are weighted and fused to obtain cross-modal correlation features; Based on the self-attention mechanism, feature enhancement processing is performed on the internal features of each modality in the unified dimension modal feature vector to obtain modality-enhanced features; The cross-modal correlation features and the modal enhancement features are subjected to residual fusion and feature splicing to obtain multimodal fusion features.

7. The method according to claim 1, characterized in that, The process of inputting the multimodal fused features into a sparse gated hybrid expert network, selecting a target expert network for fault reasoning, and obtaining fault identification results includes: By using a sparse gated hybrid expert network, the multimodal fusion features are linearly mapped and weights are calculated to obtain the expert activation weights corresponding to each expert network. Based on the preset TopK sparse filtering rules, the expert activation weights are sorted and filtered to obtain the target activation weights of the target expert network. Based on the target activation weight, a target expert network is selected from the thermal fault expert network, electrical fault expert network, structural fault expert network, and pollution fault expert network. Through the target expert network, fault reasoning is performed on the thermal anomaly features, electrical performance anomaly features, structural anomaly features, and fouling anomaly features in the multimodal fusion features to obtain expert fault results; The expert fault results are weighted and fused based on the target activation weights to obtain the fault identification results.

8. The method according to claim 1, characterized in that, The process of determining the fault type and assessing the fault severity based on the fault identification results to obtain photovoltaic module fault detection results includes: Based on the expert fault results and the corresponding target activation weights in the fault identification results, the fault classification probability corresponding to each fault type is calculated to obtain the fault confidence. Based on the fault confidence level, fault category matching is performed to obtain the target fault type; Thermal anomaly features, electrical performance anomaly features, structural anomaly features, and fouling anomaly features are extracted from the fault identification results, and fault severity assessment features are constructed. The fault severity assessment features are input into the severity assessment model, and regression analysis is performed on the impact of photovoltaic module faults to obtain fault severity values. Based on the target fault type and the fault severity value, the photovoltaic module fault detection result is determined.

9. A photovoltaic module intelligent fault detection system based on multimodal deep learning, characterized in that, The system includes: a data acquisition module, a visible light feature extraction module, an infrared feature extraction module, an electrical feature extraction module, a feature fusion module, a fault identification module, and a fault discrimination module; The data acquisition module is used to acquire visible light image data, infrared thermal imaging data, and electrical performance data of the photovoltaic module. The visible light feature extraction module is used to perform component region segmentation, texture feature extraction, and structural feature extraction on the visible light image data to obtain visible light features. The infrared feature extraction module is used to perform temperature calibration and thermal anomaly feature extraction on the infrared thermal imaging data to obtain infrared thermal features. The electrical feature extraction module is used to extract electrical performance anomaly features from the electrical performance data to obtain electrical features; The feature fusion module is used to fuse the visible light features, the infrared thermal features, and the electrical features based on a cross-modal attention fusion mechanism to obtain multimodal fused features; The fault identification module is used to input the multimodal fusion features into a sparse gated hybrid expert network, select a target expert network for fault reasoning, and obtain the fault identification result; the target expert network includes a thermal fault expert network, an electrical fault expert network, a structural fault expert network, and a contamination fault expert network. The fault identification module is used to determine the fault type and assess the severity of the fault based on the fault identification results, so as to obtain the fault detection results of the photovoltaic module.

10. The system according to claim 9, characterized in that, The visible light feature extraction module is also used for: The photovoltaic module region in the visible light image data is segmented using a component segmentation network to obtain a component region mask image; The visible light image data is filtered for background based on the component region mask image to obtain the target component image; The target component image is input into a texture feature extraction network, and convolutional feature extraction is performed on the texture information in the target component image to obtain texture features; The visible light image data is input into a structural feature extraction network to perform structural analysis on the edge information, crack information and deformation information in the visible light image data, and obtain structural features. The texture features and structural features are spliced ​​and mapped to obtain visible light features.

Images