Photovoltaic module surface defect online detection method based on multi-modal image fusion

Abstract

The application discloses a photovoltaic module surface defect online detection method based on multi-modal image fusion, and relates to the technical field of photovoltaic module detection.The application collects real-time data through installation of a visible light line array camera, a short-wave infrared camera, an EL line array camera and a laser speckle imaging module, and performs dynamic window median filtering on optical images, adopts time sliding average combined with dynamic threshold segmentation on infrared and EL images, adopts image-guided outlier filtering on 3D point cloud data, determines pixel defect types based on gray scale-temperature correlation and calculates fusion weights, respectively inputs the optical, infrared, EL images and 3D point cloud into a GhostNet, a lightweight ViT, a CNN and a PointNet-Lite to extract features, inputs the features into a channel attention module after global average pooling and splicing to generate modal weights, and obtains defect types through a full connection layer after weighted fusion.The application effectively improves the accuracy of defect detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of photovoltaic module detection, in particular to a photovoltaic module surface defect online detection method based on multi-modal image fusion. BACKGROUND

[0002] At present, the quality of photovoltaic modules is directly related to the power generation efficiency, and at the same time, surface defects such as hot spots, cracks and broken grids are prone to occur during the production, transportation and use of photovoltaic modules, which will cause the performance of the modules to decline.

[0003] The traditional manual detection method is not only low in efficiency and poor in accuracy, but also easily affected by subjective factors. The detection method based on single modal image is difficult to detect the defects inside the module, such as relying only on infrared image detection, which cannot obtain the accurate texture and shape information of the defects.

[0004] With the development of multi-modal image fusion technology, defect detection by integrating visible light, infrared, electroluminescence (EL) and other multi-source data has become a research hotspot. It can obtain module information from multiple dimensions such as optics, thermotics and electricity, and make up for the shortcomings of single modal. However, the existing technology has key defects in the preprocessing link of multi-modal information, and does not design differentiated preprocessing methods according to the physical characteristics of different modal data, which leads to inaccurate feature extraction and ultimately affects the accuracy of defect detection. SUMMARY

[0005] In view of the shortcomings of the prior art, the present application provides a photovoltaic module surface defect online detection method based on multi-modal image fusion, which solves the problem of feature extraction redundancy caused by single modal detection difficulty and multi-modal preprocessing inaccuracy.

[0006] To achieve the above purpose, the present application realizes the following technical scheme: a photovoltaic module surface defect online detection method based on multi-modal image fusion, comprising:

[0007] Step 1, install a visible light line array camera and match a polarized light source and a ring LED, a short-wave infrared camera, an electroluminescence (EL) line array camera and a laser speckle imaging module, synchronize the trigger signals of each sensor with a precise clock, and generate a space-time calibration matrix by using Zhang's calibration method combined with a dynamic distortion compensation algorithm;

[0008] Step 2, adopt a dynamic window size median filter for the collected surface optical image, adopt a time sliding average combined with a dynamic threshold segmentation for the infrared image and the EL image, and adopt an image-guided outlier filtering for the 3D point cloud data;

[0009] Step 3, based on the gray-scale-temperature correlation, determine the reflection false defects or potential real defects according to the pixels of the optical and infrared images, and calculate the pixel fusion weight w.

[0010] Step 4, the optical image is input into the GhostNet to extract features Fvis, the infrared image is input into the lightweight ViT to generate features Fir, the EL image is input into the CNN to extract features Fel, and the 3D point cloud is input into the PointNet-Lite to extract features F3D; the four modal features are globally averaged and pooled to obtain Gconcat, which is input into the channel attention module to generate modal weights, and the weighted fusion obtains Gattended; the Gattended is mapped to the defect category dimension K through a fully connected layer, and the Softmax outputs a probability distribution P; the P is combined with the dynamic threshold Tk to screen the defect categories that meet the standard.

[0011] As a further scheme of the present application, the specific steps for generating the space-time calibration matrix by using Zhang's calibration method combined with a dynamic distortion compensation algorithm are as follows:

[0012] A checkerboard calibration board is used to collect 20 groups of images at different positions in the field of view, and the camera intrinsic and extrinsic parameters are calculated by Zhang's calibration method;

[0013] For the camera pose change caused by the production line vibration, the compensation algorithm is triggered automatically once every 100 frames of images, the extrinsic matrix is updated in real time through feature point matching, and the pixel-level alignment accuracy is ensured to be less than 0.3 pixels;

[0014] Taking the visible light image as the reference, the transformation matrix of the infrared image, the EL image and the 3D point cloud is optimized by the mutual information maximization algorithm, so that the projection error of the same defect in each modal image is less than 1 pixel.

[0015] As a further scheme of the present application, the specific steps for automatically adjusting the filtering window according to the edge density of the local region are as follows:

[0016] The gradient amplitude of each pixel neighborhood is calculated, and the number Ne of pixels with a gradient amplitude greater than a threshold Tg is counted in a 7x7 pre-window centered on the current pixel;

[0017] The edge density ED is calculated according to the formula ED=Ne / 49;

[0018] The image is divided into three types of regions by the edge density ED, and three types of filtering windows are matched, i.e. .

[0019] As a further scheme of the present application, the mean value of the continuous 5 frames of infrared or EL images is calculated to suppress high-frequency noise and retain slowly changing thermal anomalies or electrical signal anomalies, and the threshold is automatically calculated using the Otsu algorithm according to the gray scale distribution of the mean value image to segment the abnormal region.

[0020] As a further scheme of the present application, the specific steps for using image-guided outlier filtering on the 3D point cloud data are as follows:

[0021] Projecting the 3D point cloud to the visible light image to obtain the 2D coordinates of each point;

[0022] Performing Canny edge detection on the visible light image, marking the edge region, and retaining outliers of the point cloud in the edge region;

[0023] Using statistical outlier filtering on the point cloud in the non-edge region to remove points exceeding 2 times the standard deviation.

[0024] As a further scheme of the present application, the specific steps for determining the reflection light false defect or potential real defect according to the optical and infrared image pixels are as follows:

[0025] For each pixel (x, y) in the visible light image, if its gray value G(x, y) > Gmax and the corresponding infrared temperature T(x, y) ≤ Tvis+1℃, it is determined as a reflection light false defect, and the visible light feature weight of the pixel is reduced in fusion;

[0026] If the pixel gray value G(x, y) < Gmin in the visible light and the infrared temperature T(x, y) ≥ Tvis+2℃, it is determined as a potential real defect, and the bimodal feature weight of the pixel is enhanced in fusion, wherein Gmax and Gmin are pixel gray value thresholds, and Tvis is the global mean temperature of the infrared image.

[0027] As a further scheme of the present application, the pixel fusion weight is calculated according to the formula

[0028] As a further scheme of the present application, Gmax takes the first 5% quantile of the gray value of the visible light image, and Gmin takes min{the last 5% quantile, μg-30}, wherein μg is the global mean gray value and 30 is an empirical coefficient.

[0029] As a further scheme of the present application, for the visible light feature Fvis and the infrared feature Fir, the weight mask w is expanded in the channel dimension to W1 ∈ RH×W×C1 and W2 ∈ RH×W×C2, to obtain the weighted features Fvis'=Fvis⊙W1 and Fir'=Fir⊙W2, wherein ⊙ represents element-wise multiplication, H is the feature map height, W is the feature map width, C1 and C2 are the number of feature channels.

[0030] As a further scheme of the present application, the specific steps for generating the modality weight according to the input channel attention module are as follows:

[0031] Inputting Gconcat into the channel attention module, reducing the dimension to Ctotal / r through the first full connection layer FC1, and introducing nonlinearity through ReLU activation;

[0032] ​The channel attention weight A corresponding to the attention weight of each modality Gvis, Gir, Gel and G3D is generated by recovering to the original dimension through the second full connection layer FC2, and the Sigmoid activation generates the channel attention weight A.

[0033] The global average of the weight of each modality is obtained to obtain the modality level importance coefficient qvis, qir, qel and q3D, and after the normalization of the feature vectors of each modality, the fusion feature is calculated according to the formula Gattended=(Gvis*qvis+Gir*qir+Gel*qel+G3D*q3D) / (qvis+qir+qel+q3D), wherein Ctotal=C1+C2+C3+D, and r is the compression ratio.

[0034] The present application provides a photovoltaic module surface defect online detection method based on multi-modal image fusion, which has the following beneficial effects compared with the prior art:

[0035] (1) The present application fuses visible light, infrared, electroluminescence and 3D point cloud data, which can comprehensively obtain the information of the surface and interior of the photovoltaic module from multiple angles, greatly improving the accuracy and integrity of defect detection compared with single modality detection.

[0036] (2) The present application adopts a dynamic preprocessing method, which adaptively adjusts the processing parameters according to the data characteristics, effectively removes the noise of different modalities, and retains the defect features, providing high-quality data for subsequent feature extraction and improving the robustness of the detection system.

[0037] (3) The present application can effectively reduce redundant information, highlight potential defect features and enhance the recognition ability of the detection model for different defect types through the cross-modal coarse matching and feature fusion strategy. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The present application provides a photovoltaic module surface defect online detection method based on multi-modal image fusion, which has the following beneficial effects compared with the prior art: DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] As Figure 1 , the present application provides a photovoltaic module surface defect online detection method based on multi-modal image fusion, which includes:

[0041] Step 1, real-time acquisition of surface optical images, infrared images, EL images, and 3D point cloud data of photovoltaic modules by visible light line array camera, combined with polarized light source and ring-shaped LED, short-wave infrared camera, electroluminescence EL line array camera, and laser speckle imaging module, and precise clock synchronization of the trigger signals of each sensor to ensure that the time difference of multi-modal data acquisition of the same component position is within the specified time, ensuring the real-time and consistency of the data, at the same time, Zhang's calibration method combined with dynamic distortion compensation algorithm is used to generate a space-time calibration matrix to calibrate the internal and external parameters of each camera, realizing pixel-level alignment;

[0042] The specific reason for choosing a visible light line array camera combined with a polarized light source and a ring-shaped LED is that the tempered glass on the surface of the photovoltaic module is prone to mirror reflection, resulting in strong light spots in the image, which obscures the true defect features. By adjusting the polarization direction of the polarized light source, the reflection can be effectively filtered, and the contrast of the surface texture, such as cracks and grid lines, can be enhanced. For example, in the detection of a certain crystalline silicon module, after using the polarized light source, the gray value standard deviation of the reflection area is greatly reduced, and the gradient amplitude of the defect edge is improved.

[0043] The ring-shaped LED provides multi-angle coaxial illumination, which can eliminate the shadow effect. Traditional single-sided light source can form shadows at defects such as cracks and depressions, leading to misjudgment. Ring-shaped illumination makes the defect edges clearly visible.

[0044] The line array camera realizes high-speed imaging through line-by-line scanning, and adjusts the exposure time dynamically to ensure that the image is not blurred under motion. For example, at a conveyor belt speed of 1.5 m / s, a clear image with a pixel resolution of 0.1 mm can still be acquired.

[0045] The specific reason for choosing a short-wave infrared camera is that based on Planck's law, objects with a temperature higher than absolute zero emit infrared radiation. Internal defects in photovoltaic modules can cause local thermal resistance to increase, such as hidden cracks and cell failure, resulting in temperature anomalies.

[0046] Short-wave infrared can penetrate the glass surface and detect defects in the underlying cells, with a very high detection rate for obscured hot spots. At the same time, if a refrigeration-type InGaAs detector is used, the temperature resolution is 0.01°C, which can identify early hot spots and meet the real-time detection needs of high-speed production lines.

[0047] The specific reason for choosing an electroluminescence EL line array camera is that a forward bias voltage, typically 10-20V, is applied to the module, and near-infrared light is generated by the internal electron transition of the cell. The light intensity in the defect area is weakened.

[0048] It can detect cell micro-cracks, broken grid lines, and other defects that cannot be identified under visible light. A near-infrared dedicated lens and a high-sensitivity InGaAs sensor are used, with a gray scale contrast ratio of 8:1 between the defect area and the normal area.

[0049] The specific reason for selecting the laser speckle imaging module is that the phase difference of reflected light is calculated by projecting structured light to the surface of the component using binocular vision principle, and the three-dimensional topography is reconstructed;

[0050] Its vertical resolution can reach 5 μm, and it can detect the topographic defects such as bumps and depressions on the surface of the component, such as solder bumps and battery piece fragmentation. Near-infrared laser is used, which is not affected by environmental light and can still maintain measurement accuracy in a strong light workshop environment. At the same time, the single-frame point cloud generation time is less than 10 ms, meeting the online detection requirements;

[0051] The precise clock synchronization can select Microsemi SyncServer S650, which has a built-in rubidium atomic clock with a time synchronization error of less than 100 ns, ensuring that the sampling time difference of multiple sensors on the same physical location is less than 1 μs, avoiding fusion errors caused by time misalignment. For example, when detecting moving components, if the time synchronization error is greater than 10 μs, the spatial registration deviation of visible light and infrared images will be greater than 0.5 mm;

[0052] Dual-redundant power supply and anti-vibration design are used, with an average failure-free time of more than 200,000 hours in an industrial environment, ensuring long-term stable operation. At the same time, the built-in GNSS receiver automatically switches to GPS time service when the network is interrupted, ensuring uninterrupted operation of the system;

[0053] The specific process of generating the space-time calibration matrix through Zhang's calibration method combined with dynamic distortion compensation algorithm is as follows:

[0054] Use a checkerboard calibration board to collect 20 groups of images at different positions in the field of view, and calculate the camera intrinsic parameters (focal length, principal point, distortion coefficient) and extrinsic parameters (rotation matrix, translation vector) through Zhang's calibration method;

[0055] For camera pose changes caused by production line vibration, the compensation algorithm is automatically triggered once every 100 frames of images, and the extrinsic matrix is updated in real time through feature point matching, such as ORB features, to ensure that the pixel-level alignment accuracy is less than 0.3 pixels;

[0056] Taking the visible light image as the reference, the transformation matrix of infrared, EL and 3D point cloud is optimized through the mutual information maximization algorithm, such as Parzen window estimation, so that the projection error of the same defect in each modal image is less than 1 pixel.

[0057] Step 2, different dynamic preprocessing methods are used for the collected surface optical image, infrared image, EL image and 3D point cloud data respectively;

[0058] For optical images, median filtering with dynamic window size is adopted, the range of dynamic window is 3x3~7x7, the filtering window is automatically adjusted according to the edge density of local area, the specific steps are as follows:

[0059] The gradient amplitude of each pixel neighborhood is calculated, the number Ne of pixels with gradient amplitude greater than threshold Tg is counted in the 7x7 pre-window with the current pixel as the center, Tg is automatically adjusted according to the overall contrast of the image, such as increasing to 2 times of the average in strong light environment, to avoid edge missing or misjudgment caused by fixed threshold, the threshold is usually set to 1.5 times of the average gradient of the image;

[0060] The edge density is calculated according to the formula ED=Ne / 49;

[0061] The image is divided into 3 categories of regions by edge density ED, and 3 kinds of filtering windows are matched, that is, ;

[0062] There are a large number of grid lines, frames and other regular edges on the surface of photovoltaic modules, and traditional fixed window median filtering is easy to blur the edge details, such as the fine fracture of broken grid defects, while the adaptive method can maintain high resolution in strong edge area and effectively denoise in smooth area;

[0063] For example, in an optical image containing more details, the area with high edge density, such as the edge of the module and the vicinity of the grid line, uses a 3x3 filtering window, which can effectively remove noise and retain details; the area with low edge density, such as a large area of blank area, uses a 7x7 filtering window to improve the filtering efficiency;

[0064] For infrared images and EL images, time sliding average combined with dynamic threshold segmentation is adopted, and the specific operation steps are as follows:

[0065] The average of 5 consecutive infrared or EL images is calculated to suppress high-frequency noise, such as random thermal fluctuations caused by environmental light flicker and equipment vibration, and to retain slowly changing thermal anomalies or electrical signal anomalies, such as sustained temperature rise caused by hidden cracks and EL signal attenuation caused by broken grid;

[0066] According to the gray scale distribution of the average image, the threshold is automatically calculated using Otsu algorithm, which replaces the traditional fixed threshold here, and the abnormal area is segmented, such as the temperature of hot spot higher than the average plus 2σ, and the EL signal lower than the average minus 1.5σ, wherein σ represents the standard deviation of the pixel value in the local area of the image;

[0067] The changes of thermal signal and electrical signal have time continuity, such as the temperature rise caused by the aging of sealing ring is a gradual process, while noise is mostly high-frequency random signal, time dimension filtering can effectively distinguish real defects from noise, and dynamic threshold adapts to the initial temperature and signal difference of different modules, such as the EL baseline of new and old modules is different;

[0068] For 3D point cloud data, image-guided outlier filtering is adopted, and the specific operation is as follows:

[0069] Project the 3D point cloud to the visible light image to obtain the 2D coordinates of each point;

[0070] Perform Canny edge detection on the visible light image to mark the edge region, such as the component frame, the grid line edge, and the defect contour;

[0071] Keep the outliers of the point cloud in the edge region, which may correspond to defect protrusions or depressions, and use statistical outlier filtering for the point cloud in the non-edge region to remove points exceeding 2 times the standard deviation;

[0072] The 3D point cloud contains a large amount of non-defect noise, such as dust particles and outliers caused by equipment vibration, especially in smooth areas of the component, such as the glass surface, more than 90% of the point cloud belongs to background noise and is irrelevant to defects;

[0073] The surface defects of the photovoltaic component are usually accompanied by edge features, which are represented as gray level mutations in the visible light image, while the noise points are mostly distributed in the non-edge region, such as dust. By guiding the visible light edge, the defect-related point cloud can be retained and irrelevant noise can be removed.

[0074] Step 3, cross-modal coordination of the preprocessed optical image, infrared image, EL image, and 3D point cloud data is needed. Based on the gray-temperature correlation, rough matching is performed. Direct fusion of multi-modal data without verification will introduce a large amount of redundant information, such as invalid features in the reflection area, increasing the computational load of the fusion model;

[0075] At the same time, it can also solve the problem of single-mode pseudo-defects. There are a large number of single-mode abnormalities on the surface of the photovoltaic component, but they are not real defects. For example, grid reflection: in the visible light image, it appears as a high gray level, similar to a crack, but in the infrared image, there is no temperature anomaly, and the metal grid conducts heat quickly, with no local temperature rise; surface stains: in the visible light image, it appears as a low gray level patch, but in the EL image, the electrical signal is normal, and it is not an internal defect of the cell;

[0076] The specific implementation steps are as follows:

[0077] For each pixel (x, y) in the visible light image, if its gray value G(x, y) > threshold Gmax and the corresponding infrared temperature T(x, y) ≤ Tvis+1℃, it is determined as a reflection pseudo-defect, and the visible light feature weight of the pixel is reduced in the fusion, for example, the grid reflection area appears as a bright line in the visible light, but the infrared temperature is consistent with the surrounding, and through the correlation verification, it can be avoided to misjudge it as a hidden crack, which usually accompanies temperature anomaly;

[0078] If the pixel gray value G(x, y) in visible light is less than the threshold Gmin, and the infrared temperature T(x, y) is greater than or equal to Tvis+2℃, it is determined as a potential real defect, and the bimodal feature weight of the pixel is enhanced in fusion, for example, a hidden crack appears as a fine dark line in visible light, and the corresponding area temperature in infrared is 35℃, and the surrounding temperature is 30℃, and the subsequent detection model is more likely to capture the feature through weight enhancement;

[0079] Through the space-time calibration matrix, it is ensured that the pixels (x, y) of the visible light and infrared images strictly correspond to the same physical position;

[0080] Gmax takes the first 5% quantile of the gray value of the visible light image, which adapts to the change of ambient light;

[0081] Gmin takes min (the last 5% quantile, μg-30), the last 5% quantile covers the 5% darkest area in the image, while photovoltaic defects (such as hidden cracks and broken grids) usually occupy 1% to 3% of the pixels, so it is ensured that Gmin can capture these extremely dark areas, while excluding noise; the defect prior constraint is introduced, if Gmin> global mean gray μg-30, then Gmin is forced to be the global mean gray μg-30, which avoids the quantile being too high in the extreme strong light environment, and missing the defect dark line, wherein μg is the global mean gray, and 30 is an empirical coefficient;

[0082] Tvis takes the global mean temperature of the infrared image, which reflects the current working condition of the component;

[0083] For each pixel (x, y), the weight w(x, y) is calculated, and the specific formula is .

[0084] Step 4, input the preprocessed optical image into the GhostNet, extract the shallow texture feature Fvis∈R H ×W×C1 , including edge, grid line and other optical details, input the preprocessed infrared image into the lightweight ViT, generate the global heat distribution feature Fir∈R H×W×C2 , input the preprocessed EL image into the CNN to extract the electrical signal abnormal feature Fel∈R H×W×C3 , and input the preprocessed 3D point cloud data into the PointNet-Lite to extract the topographic feature F3D∈R N×D , wherein H is the feature map height, W is the feature map width, C1, C2 and C3 are the feature channel numbers, N is the number of points in the point cloud data, and D is the feature dimension of each point;

[0085] The core value of visible light images lies in surface texture details, such as crack edges and broken grid lines. High-resolution features need to be preserved in shallow networks while meeting the real-time requirements of online detection. GhostNet, as a lightweight CNN model, generates redundant features through the Ghost module. While maintaining feature expressive power, the computational cost is also effectively controlled. It is adapted to the high-resolution input of visible light images and enhances the extraction of low-frequency texture features such as edges and corners through depthwise separable convolution and linear transformation.

[0086] Defect characterization in infrared images relies on global thermal distribution patterns, such as the temperature gradient of hot spots and the continuous temperature rise of hidden cracks. It is necessary to capture the temperature correlation between long-distance pixels. The self-attention mechanism of Vision Transformer (ViT) is good at modeling global context, but traditional ViT has high computational complexity. Lightweight ViT effectively reduces the computational load while maintaining global feature extraction capabilities by compressing the self-attention window and reducing the embedding dimension, thus adapting to the low resolution and single-channel characteristics of infrared images.

[0087] Defects in EL images manifest as local electrical signal anomalies, such as attenuation of luminous intensity at broken grids and uneven luminous emission in microcracks. These are local gray-level difference detections. CNN convolutional layers have a natural advantage in detecting local spatial features, such as connected regions and gray-level abrupt changes. Moreover, they are simple in structure and fast in training, making them suitable for high-frequency dynamic acquisition of EL images.

[0088] 3D point cloud data is unstructured data, which traditional CNNs cannot process directly. PointNet-Lite, as a lightweight point cloud processing network, maintains the permutation invariance of point clouds through symmetric functions, while removing redundant computing nodes, such as reducing the number of fully connected layers, to adapt to the large-scale point cloud data generated by the laser speckle imaging module.

[0089] For visible light feature Fvis and infrared feature Fir, extend the weight mask w to W1∈R in the channel dimension. H×W×C1 and W2∈R H×W×C2 We obtain Fvis'=Fvis⊙W1, Fir'=Fir⊙W2, where ⊙ represents element-wise multiplication, which reduces the characteristic value of the reflective area and increases the potential defect area.

[0090] The weighted masks w for the visible light mode and the infrared mode need to be expanded into three-dimensional tensors with the same number of feature channels as their respective modes, namely W1 and W2 above. The visible light feature channel C1 contains optical features such as edges and textures, and each channel corresponds to edge detection in different directions and scales. The infrared feature channel C2 contains thermal features such as temperature gradients and hot spots, and each channel corresponds to the temperature response of different regions, such as temperature rise in the central region and temperature uniformity in the edge region. Even if the 2D matrix of the weighted mask w is the same, the physical meanings of the optical channels and the thermal channels are completely different and need to be weighted independently. For example, reflection only suppresses the visible light channel and does not affect the infrared temperature channel.

[0091] Global average pooling (GAP) is performed on the features of the four modalities, i.e., Gvis=GAP(Fvis'), Gir=GAP(Fir'), Gel=GAP(Fel), G3D=GAP(F3D), to obtain the one-dimensional feature vectors of the four modalities, and then concatenated into a unified feature vector Gconcat=[Gvis,Gir,Gel,G3D].

[0092] For image class features H×W×C, by averaging the spatial dimension H×W, it is compressed into 1×1×C, i.e., a one-dimensional vector C, while retaining channel-level global statistical information, such as the overall intensity of each channel feature, and removing redundant spatial location details. The specific location of the defect may not affect the type judgment, but the global distribution is more important.

[0093] For point cloud features N×D, average the number of points N and compress it to 1×D, transforming the unordered point set into a global statistical vector and avoiding feature fluctuations caused by the point cloud arrangement order.

[0094] For example, the texture feature Fvis'∈RH×W×C1 extracted from a visible light image becomes Vvis∈RC1 after GAP, where each element is the average value of all pixels in the corresponding channel, reflecting the overall intensity of the texture of that channel, such as edges and grid lines.

[0095] In defect detection, local reflections and noise points may interfere with the judgment. GAP weakens the influence of local outliers and strengthens global features through averaging operations, such as the overall temperature anomaly of the defect area in the infrared image and the global distribution of electrical signal anomalies in the EL image.

[0096] Compared to fully connected layer dimensionality reduction, GAP has no parameters, low computational complexity, and avoids overfitting caused by too many weights in fully connected layers.

[0097] The Gconcat input channel attention module is reduced to Ctotal / r through the first fully connected layer FC1, and nonlinearity is introduced by ReLU activation, where Ctotal = C1 + C2 + C3 + D, and r is the compression ratio;

[0098] The original dimension is restored through the second fully connected layer FC2, and channel attention weights A are generated by sigmoid activation. The attention weights of each modality Gvis, Gir, Gel, G3D, Avis, Air, Ael, A3D are extracted accordingly. For example, Avis is the first C1 dimension of A.

[0099] The weights of each mode are globally averaged to obtain the mode-level importance coefficients qvis,qir,qel,q3D. After normalizing the feature vectors of each mode, the weighted sums are calculated according to the mode importance coefficients, i.e., Gattiped = (Gvis*qvis + Gir*qir + Gel*qel + G3D*q3D) / (qvis + qir + qel + q3D).

[0100] The fused features Gattended are mapped to the defect category dimension K through a fully connected layer to obtain logits. Softmax activation is applied to the logits to generate the probability distribution P of each defect type. Combined with a preset dynamic threshold Tk, k∈[1,K], defect categories with probabilities higher than the threshold are selected as detection results.

[0101] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0102] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for online detection of surface defects in photovoltaic modules based on multimodal image fusion, characterized in that, include: Step 1: Install a visible light array camera and pair it with a polarized light source and a ring LED, a short-wave infrared camera, an electroluminescent EL linear array camera, and a laser speckle imaging module. Use a precision clock to synchronize the trigger signals of each sensor, and use Zhang's calibration method combined with a dynamic distortion compensation algorithm to generate a spatiotemporal calibration matrix. Step 2: The acquired surface optical images are filtered using median filtering with dynamic window size; the infrared and EL images are segmented using time-moving average combined with dynamic thresholding; and the 3D point cloud data is filtered using image-guided outlier filtering. Step 3: Based on the gray-temperature correlation, determine whether a pixel in the optical or infrared image is a reflective pseudo-defect or a potential real defect, and calculate the pixel fusion weight w. Specifically, for each pixel (x,y) in the visible light image, if its gray value G(x,y) > Gmax and the corresponding infrared temperature T(x,y) ≤ Tvis+1℃, it is determined to be a reflective pseudo-defect, and the visible light feature weight of this pixel is reduced during fusion. If the pixel gray value G(x,y) < Gmin in the visible light image and the infrared temperature T(x,y) ≥ Tvis+2℃, it is determined to be a potential real defect, and the dual-modal feature weight of this pixel is enhanced during fusion. Here, Gmax and Gmin are pixel gray value thresholds, and Tvis is the global mean temperature of the infrared image. According to the formula... Calculate pixel fusion weights; Gmax is the top 5 percentile of the grayscale value of the visible light image, and Gmin is min{bottom 5 percentile, μg-30}, where μg is the global mean grayscale value and 30 is an empirical coefficient; Step 4: Optical images are input into GhostNet to extract features Fvis, infrared images are processed by lightweight ViT to generate features Fir, EL images are processed by CNN to extract features Fel, and 3D point clouds are processed by PointNet-Lite to extract features F3D. The four-modal features are globally averaged and then concatenated into Gconcat. The input channel attention module generates modal weights, which are weighted and fused to obtain Gattended. This is then mapped to the defect category dimension K through a fully connected layer. Softmax outputs the probability distribution P, which is combined with a dynamic threshold Tk to select defect categories that meet the criteria P.

2. The online detection method for surface defects of photovoltaic modules based on multimodal image fusion according to claim 1, characterized in that, The specific steps for generating the spatiotemporal calibration matrix using Zhang's calibration method combined with the dynamic distortion compensation algorithm are as follows: Using a checkerboard calibration board, 20 sets of images were acquired at different locations within the field of view, and the camera's intrinsic and extrinsic parameters were calculated using Zhang's calibration method. To address camera pose changes caused by production line vibrations, a compensation algorithm is automatically triggered every 100 frames of images acquired. The extrinsic parameter matrix is ​​updated in real time through feature point matching to ensure pixel-level alignment accuracy of less than 0.3 pixels. Based on visible light images, the transformation matrices of infrared, EL, and 3D point clouds are optimized using a mutual information maximization algorithm, so that the projection error of the same defect in each modal image is less than 1 pixel.

3. The online detection method for surface defects of photovoltaic modules based on multimodal image fusion according to claim 1, characterized in that, The specific steps for automatically adjusting the filter window based on the edge density of a local region are as follows: Calculate the gradient magnitude of each pixel's neighborhood, and count the number of pixels Ne whose gradient magnitude is greater than the threshold Tg within a 7×7 pre-window centered on the current pixel. Calculate the edge density using the formula ED=Ne / 49; The image is divided into three regions using edge density ED (Edge Filtering), and three different filter windows are matched accordingly. .

4. The online detection method for surface defects of photovoltaic modules based on multimodal image fusion according to claim 1, characterized in that, The average value of five consecutive infrared or EL images is calculated to suppress high-frequency noise and retain slowly changing thermal or electrical signal anomalies. Based on the grayscale distribution of the average image, the Otsu algorithm is used to automatically calculate the threshold and segment the abnormal regions.

5. The online detection method for surface defects of photovoltaic modules based on multimodal image fusion according to claim 1, characterized in that, The specific steps for image-guided outlier filtering of 3D point cloud data are as follows: Project the 3D point cloud onto the visible light image to obtain the 2D coordinates of each point; Canny edge detection is performed on visible light images to mark edge regions and retain outliers in the point cloud of edge regions; Statistical outlier filtering is applied to point clouds in non-edge regions to remove points that exceed twice the standard deviation.

6. The online detection method for surface defects of photovoltaic modules based on multimodal image fusion according to claim 1, characterized in that, For visible light feature Fvis and infrared feature Fir, extend the weight mask w to W1∈R in the channel dimension. H×W×C1 and W2∈R H×W×C2 We obtain the weighted features Fvis'=Fvis⊙W1, Fir'=Fir⊙W2, where ⊙ represents element-wise multiplication, H is the feature map height, W is the feature map width, and C1 and C2 are the number of feature channels.

7. The online detection method for surface defects of photovoltaic modules based on multimodal image fusion according to claim 1, characterized in that, The specific steps for generating Gatted modality weights based on the input channel attention module are as follows: The Gconcat input channel attention module is reduced to Ctotal / r by passing the first fully connected layer FC1, and nonlinearity is introduced by ReLU activation; The original dimension is restored by the second fully connected layer FC2, and channel attention weights A are generated by sigmoid activation. The attention weights of each modality Gvis, Gir, Gel, and G3D are extracted to obtain Avis, Air, Ael, and A3D. The weights of each modality are globally averaged to obtain modality-level importance coefficients qvis,qir,qel,q3D. After normalizing the feature vectors of each modality, the fused features are calculated according to the formula Gatted=(Gvis*qvis+Gir*qir+Gel*qel+G3D*q3D) / (qvis+qir+qel+q3D), where Ctotal=C1+C2+C3+D and r is the compression ratio.

Images