Photovoltaic module surface defect online detection method based on multi-modal image fusion

Abstract

The invention discloses a photovoltaic module surface defect online detection method based on multi-modal image fusion, and relates to the technical field of photovoltaic module detection.The method comprises the steps that a visible light line-scan digital camera, a short-wave infrared camera, an EL line-scan digital camera and a laser speckle imaging module are installed to collect real-time data, an optical image is subjected to dynamic window median filtering, and the surface defect of the photovoltaic module is obtained; the method comprises the following steps: carrying out time moving average combined dynamic threshold segmentation on an infrared image and an EL image, carrying out image-guided outlier filtering on 3D point cloud data, judging a pixel defect type based on gray-temperature correlation and calculating a fusion weight, respectively inputting the optical image, the infrared image, the EL image and 3D point cloud into GhostNet, lightweight ViT, CNN and PointNet-Lite to extract features, and carrying out image fusion on the 3D point cloud data. And after global average pooling and splicing, inputting into a channel attention module to generate a modal weight, and after weighted fusion, obtaining a defect type through a full connection layer. According to the invention, the accuracy of defect detection is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of photovoltaic module detection, in particular to a photovoltaic module surface defect online detection method based on multi-modal image fusion. BACKGROUND

[0002] At present, the photovoltaic industry is developing rapidly, and the quality of photovoltaic modules is directly related to the power generation efficiency. At the same time, during the production, transportation and use of photovoltaic modules, surface defects such as hot spots, cracks and broken grids are prone to occur, which will cause the performance of the modules to decline.

[0003] The traditional manual detection method is not only low in efficiency and poor in accuracy, but also easily affected by subjective factors. The detection method based on single modal image is difficult to detect the defects inside the module. For example, relying only on infrared image detection cannot obtain the accurate texture and shape information of the defects.

[0004] With the development of multi-modal image fusion technology, defect detection by integrating visible light, infrared, electroluminescence (EL) and other multi-source data has become a research hotspot. It can obtain module information from multiple dimensions such as optics, thermotics and electricity, and make up for the shortcomings of single modal. However, the existing technology has key defects in the preprocessing link of multi-modal information. Differentiated preprocessing methods are not designed according to the physical characteristics of different modal data, which leads to inaccurate feature extraction and ultimately affects the accuracy of defect detection. SUMMARY

[0005] In view of the shortcomings of the prior art, the present application provides a photovoltaic module surface defect online detection method based on multi-modal image fusion, which solves the problem of feature extraction redundancy caused by single modal detection difficulty and multi-modal preprocessing inaccuracy.

[0006] To achieve the above purpose, the present application realizes the following technical scheme: a photovoltaic module surface defect online detection method based on multi-modal image fusion, comprising: Step 1, install a visible light line array camera and match a polarized light source and a ring-shaped LED, a short-wave infrared camera, an electroluminescence (EL) line array camera and a laser speckle imaging module, synchronize the trigger signals of each sensor with a precise clock, and generate a space-time calibration matrix by using Zhang's calibration method combined with a dynamic distortion compensation algorithm; Step 2, adopt a dynamic window size median filter for the collected surface optical image, adopt a time sliding average combined with a dynamic threshold segmentation for the infrared image and the EL image, and adopt an image-guided outlier filtering for the 3D point cloud data; Step 3, based on the gray-scale-temperature correlation, determine the reflection false defects or potential real defects according to the pixels of the optical and infrared images, and calculate the pixel fusion weight w; Step 4, the optical image input GhostNet extracts features Fvis, the infrared image generates features Fir through the lightweight ViT, the EL image extracts features Fel through the CNN, and the 3D point cloud extracts features F3D through the PointNet-Lite. After global average pooling of the four modal features, they are spliced into Gconcat, the channel attention module is input to generate modal weights, and the weighted fusion obtains Gattended. After mapping to the defect category dimension K through the full connection layer, the Softmax outputs the probability distribution P, and the dynamic threshold Tk is combined to screen the defect categories that meet the standard.

[0007] As a further scheme of the present application, the specific steps for generating the space-time calibration matrix by Zhang's calibration method combined with a dynamic distortion compensation algorithm are as follows: A checkerboard calibration board is used to collect 20 groups of images at different positions in the field of view, and the camera intrinsic and extrinsic parameters are calculated by Zhang's calibration method; For the camera pose change caused by the production line vibration, the compensation algorithm is triggered automatically once every 100 frames of images, and the extrinsic matrix is updated in real time through feature point matching to ensure that the pixel-level alignment accuracy is less than 0.3 pixels; Taking the visible light image as the reference, the transformation matrix of the infrared, EL and 3D point cloud is optimized by the mutual information maximization algorithm, so that the projection error of the same defect in each modal image is less than 1 pixel.

[0008] As a further scheme of the present application, the specific steps for automatically adjusting the filter window according to the edge density of the local area are as follows: The gradient amplitude of each pixel neighborhood is calculated, and the number Ne of pixels with gradient amplitude greater than the threshold Tg is counted in the 7x7 pre-window centered on the current pixel; The edge density ED is calculated according to the formula ED=Ne / 49; The image is divided into 3 types of regions by the edge density ED, and 3 types of filter windows are matched, i.e. .

[0009] As a further scheme of the present application, the mean value of the continuous 5 frames of infrared or EL images is calculated to suppress high-frequency noise and retain slow-changing thermal anomalies or electrical signal anomalies. According to the gray scale distribution of the mean value image, the threshold is automatically calculated using the Otsu algorithm to segment the abnormal area.

[0010] As a further scheme of the present application, the specific steps for using image-guided outlier filtering on 3D point cloud data are as follows: The 3D point cloud is projected onto the visible light image to obtain the 2D coordinates of each point; The visible light image is subjected to Canny edge detection, the edge region is marked, and the outlier points of the edge region are retained; Statistical outlier filtering is used on the point cloud in non-edge areas to remove points with a value greater than 2 times the standard deviation.

[0011] As a further solution of the present invention, the specific steps of determining reflective false defects or potential real defects based on optical and infrared image pixels are as follows: For each pixel (x, y) in the visible light image, if its grayscale value G(x, y)>Gmax and the corresponding infrared temperature T(x, y)≤Tvis+1°C, it is determined to be a reflective false defect, and the visible light feature weight of the pixel is reduced during fusion; If the pixel gray value G(x,y) in visible light <Gmin,且红外温度T(x,y)≥Tvis+2℃,则判定为潜在真实缺陷,在融合时增强该像素的双模态特征权重,其中,Gmax、Gmin为像素灰度值阈值,Tvis为红外图像全局均值温度。

[0012] As a further embodiment of the present invention, according to the formula Calculate pixel fusion weights.

[0013] As a further solution of the present invention, Gmax takes the first 5% quantile of the grayscale value of the visible light image, and Gmin takes the min{last 5% quantile, μg-30}, where μg is the global mean grayscale and 30 is the empirical coefficient.

[0014] As a further solution of the present invention, for the visible light feature Fvis and the infrared feature Fir, the weight mask w is expanded to W1∈RH×W×C1 and W2∈RH×W×C2 in the channel dimension to obtain the weighted feature Fvis'=Fvis⊙W1, Fir'=Fir⊙W2, where ⊙ represents element-by-element multiplication, H is the feature map height, W is the feature map width, and C1 and C2 are the number of feature channels.

[0015] As a further solution of the present invention, the specific steps of generating modal weights according to the input channel attention module to obtain Gattended are: Gconcat is input into the channel attention module, and the dimension is reduced to Ctotal / r through the first fully connected layer FC1, and nonlinearity is introduced through ReLU activation; The original dimension is restored through the second fully connected layer FC2, and the channel attention weight A is generated through Sigmoid activation. The attention weights of each modality Gvis, Gir, Gel, and G3D are correspondingly extracted to obtain Avis, Air, Ael, and A3D. The weight of each modality is globally averaged to obtain the modal-level importance coefficients qvis, qir, qel, and q3D. After normalizing the feature vectors of each modality, the fusion features are calculated according to the formula Gattended = (Gvis*qvis+Gir*qir+Gel*qel+G3D*q3D) / (qvis+qir+qel+q3D), where Ctotal = C1+C2+C3+D and r is the compression ratio.

[0016] The present invention provides an online detection method for photovoltaic module surface defects based on multimodal image fusion, which has the following advantages compared with the existing technology: (1) The present invention integrates visible light, infrared, electroluminescence and 3D point cloud data, and can comprehensively obtain information on the surface and interior of photovoltaic modules from multiple angles. Compared with single-modality detection, it greatly improves the accuracy and completeness of defect detection; (2) The present invention adopts a dynamic preprocessing method to adaptively adjust processing parameters according to data characteristics, effectively remove noise of different modes, and retain defect characteristics, providing high-quality data for subsequent feature extraction and improving the robustness of the detection system; (3) Through cross-modal coarse matching and feature fusion strategies, the present invention can effectively reduce redundant information, highlight potential defect features, and enhance the detection model's ability to identify different defect types. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the steps of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] like Figure 1 The present invention provides a method for online detection of photovoltaic module surface defects based on multimodal image fusion, comprising: Step 1: Use a visible light array camera in conjunction with a polarized light source and ring LED, a short-wave infrared camera, an electroluminescent (EL) linear array camera, and a laser speckle imaging module to collect real-time surface optical images, infrared images, EL images, and 3D point cloud data of photovoltaic modules. A precision clock is used to synchronize the trigger signals of each sensor to ensure that the time difference in multimodal data acquisition at the same module location is within the specified time, ensuring real-time and consistency of the data. At the same time, Zhang's calibration method combined with a dynamic distortion compensation algorithm is used to generate a spatiotemporal calibration matrix, and the intrinsic and extrinsic parameters of each camera are calibrated to achieve pixel-level alignment. The specific reason for choosing a visible light array camera with a polarized light source and ring LED is that the tempered glass on the surface of photovoltaic modules is prone to specular reflection, resulting in strong light spots in the image, which obscures the true characteristics of defects. Using a polarized light source to adjust the polarization direction can effectively filter out the reflections and enhance the contrast of surface textures such as cracks and grid lines. For example, in the inspection of a certain crystalline silicon module, the use of a polarized light source significantly reduced the standard deviation of the grayscale value in the reflective area, and improved the gradient amplitude of the defect edge. Ring LED provides multi-angle coaxial lighting to eliminate shadow effects. Traditional single-sided light sources will cast shadows on defects such as cracks and depressions, leading to misjudgment. Ring lighting makes the edges of defects clearly visible. Line scan cameras achieve high-speed imaging through line-by-line scanning and dynamically adjust exposure time to match the production line speed to ensure that images remain unblurred even when in motion. For example, at a conveyor speed of 1.5 m / s, clear images with a pixel resolution of 0.1 mm can still be captured. The specific reason for choosing a short-wave infrared camera is that, based on Planck's law, objects with temperatures above absolute zero emit infrared radiation. Internal defects in photovoltaic modules can lead to increased local thermal resistance, such as hidden cracks and cell failure, resulting in temperature anomalies. Short-wave infrared can penetrate the glass surface and detect defects in the underlying cells, with a very high detection rate for obscured hot spots. At the same time, if a cooled InGaAs detector is used, the temperature resolution can reach 0.01°C, which can identify early hot spots and meet the real-time detection requirements of high-speed production lines. The specific reasons for choosing an electroluminescent EL linear array camera are as follows: when a forward bias voltage (usually 10-20V) is applied to the component, electron transitions inside the cell generate near-infrared light, and the luminous intensity of the defective area is weakened; It can detect defects such as micro cracks and broken grids in solar cells, which cannot be identified under visible light. It uses a dedicated near-infrared lens and a high-sensitivity InGaAs sensor, and the grayscale contrast between the defective area and the normal area reaches 8:1. The specific reasons for choosing the laser speckle imaging module are: by projecting structured light onto the component surface, the phase difference of the reflected light is calculated using the principle of binocular vision to reconstruct the three-dimensional topography; Its vertical resolution can reach 5μm, enabling it to detect surface defects such as bumps and depressions on components, such as solder bumps and cell cracks. Using near-infrared lasers, it is unaffected by ambient light and can maintain measurement accuracy even in brightly lit workshops. Furthermore, the single-frame point cloud generation time is less than 10ms, meeting online inspection requirements. For precise clock synchronization, the Microsemi SyncServer S650 can be selected. Its built-in rubidium atomic clock has a time synchronization error of less than 100ns, ensuring that the sampling time difference of multiple sensors at the same physical location is less than 1μs, avoiding fusion errors caused by time misalignment. For example, when detecting moving components, if the time synchronization error is greater than 10μs, it will cause the spatial registration deviation of visible light and infrared images to exceed 0.5mm. With dual redundant power supplies and anti-vibration design, the MTBF exceeds 200,000 hours in industrial environments, ensuring long-term stable operation. Furthermore, the built-in GNSS receiver automatically switches to GPS timing when the network is interrupted, ensuring uninterrupted system operation. The specific process of generating the spatiotemporal calibration matrix by combining Zhang's calibration method with the dynamic distortion compensation algorithm is as follows: Using a checkerboard calibration plate, 20 sets of images were collected at different positions within the field of view, and the camera intrinsic parameters (focal length, principal point, distortion coefficient) and extrinsic parameters (rotation matrix, translation vector) were calculated using Zhang's calibration method. To address camera pose changes caused by production line vibration, a compensation algorithm is automatically triggered every 100 frames of image acquisition. The extrinsic parameter matrix, such as ORB features, is updated in real time through feature point matching to ensure pixel-level alignment accuracy of less than 0.3 pixels. Taking the visible light image as the benchmark, the transformation matrices of infrared, EL and 3D point clouds are optimized by the mutual information maximization algorithm, such as Parzen window estimation, so that the projection error of the same defect in each modal image is less than 1 pixel.

[0020] Step 2: Different dynamic preprocessing methods are used for the collected surface optical images, infrared images, EL images, and 3D point cloud data; For optical images, a median filter with a dynamic window size is used. The dynamic window range is 3×3~7×7. The filter window is automatically adjusted according to the edge density of the local area. The specific steps are as follows: Calculate the gradient amplitude of each pixel neighborhood. Centered on the current pixel, count the number of pixels Ne whose gradient amplitude is greater than a threshold Tg within a 7×7 pre-window. Tg is automatically adjusted with the overall image contrast. For example, it can be increased to 2 times the mean value in strong light environments to avoid edge omission or misjudgment caused by a fixed threshold. The threshold is usually set to 1.5 times the mean gradient of the image. Calculate the edge density according to the formula ED=Ne / 49; The image is divided into three types of regions by edge density ED, and three types of filter windows are matched, namely ; PV panels have numerous regular edges, such as grid lines and borders, on their surfaces. Traditional fixed-window median filtering can easily blur edge details, such as the subtle fractures of broken grid defects. However, adaptive methods can maintain high resolution in strong edge areas and effectively remove noise in smooth areas. For example, in an optical image with many details, a 3×3 filter window can be used in areas with high edge density, such as component edges and near grid lines, to effectively remove noise while preserving details. A 7×7 filter window can be used in areas with low edge density, such as large blank areas, to improve filtering efficiency. For infrared images and EL images, time sliding average combined with dynamic threshold segmentation is used. The specific steps are as follows: Calculate the average of five consecutive infrared or EL images to suppress high-frequency noise, such as random thermal fluctuations caused by ambient light flicker and equipment vibration, while retaining slowly changing thermal or electrical signal anomalies, such as continuous temperature rise caused by hidden cracks and EL signal attenuation caused by broken gates. Based on the grayscale distribution of the mean image, the Otsu algorithm is used to automatically calculate the threshold. This replaces the traditional fixed threshold to segment abnormal areas. For example, the hot spot temperature must be higher than the mean plus 2σ, and the EL signal must be lower than the mean minus 1.5σ, where σ represents the standard deviation of the pixel values ​​in the local area of ​​the image. Changes in thermal and electrical signals are temporally continuous. For example, the temperature rise caused by seal aging is a gradual process, while noise is mostly high-frequency random signals. Time-dimensional filtering can effectively distinguish between real defects and noise. Dynamic thresholds adapt to the initial temperature and signal differences of different components, such as the different EL baselines of new and old components. For 3D point cloud data, image-guided outlier filtering is used. The specific operations are as follows: Project the 3D point cloud onto the visible light image to obtain the 2D coordinates of each point; Perform Canny edge detection on visible light images to mark edge areas, such as component borders, grid line edges, and defect outlines; For the point cloud in the edge area, outliers are retained, which may correspond to defective convexities or concavities. For the point cloud in the non-edge area, statistical outlier filtering is used to remove points with a value exceeding 2 times the standard deviation. 3D point clouds contain a large amount of non-defect noise, such as outliers caused by dust particles and equipment vibration. Especially in smooth areas of components, such as glass surfaces, more than 90% of the point clouds are background noise and have nothing to do with defects. Surface defects in photovoltaic modules are usually accompanied by edge features, which appear as grayscale mutations in visible light images. Noise points are mostly distributed in non-edge areas, such as dust. By guiding visible light edges, we can specifically retain defect-related point clouds and remove irrelevant noise.

[0021] Step 3: When performing cross-modal collaboration on the pre-processed optical image, infrared image, EL image, and 3D point cloud data, a coarse matching based on grayscale-temperature correlation is required. Directly fusing unverified multimodal data will introduce a large amount of redundant information, such as invalid features in reflective areas, which increases the computational load of the fusion model. At the same time, it can also solve the problem of single-mode pseudo-defects, because there are many single-mode anomalies on the surface of photovoltaic modules that are not real defects. For example, grid line reflection: It appears as high grayscale in visible light images, similar to cracks, but the temperature is normal in infrared images. The metal grid line conducts heat quickly and there is no local temperature rise. Surface stains: They appear as low-grayscale patches in visible light, but the electrical signal in EL images is normal, and they are not internal defects of the cell. The specific implementation steps are as follows: For each pixel (x, y) in the visible light image, if its grayscale value G(x, y) is greater than the threshold Gmax and the corresponding infrared temperature T(x, y) is ≤ Tvis + 1°C, it is determined to be a reflective pseudo-defect. During fusion, the visible light feature weight of this pixel is reduced. For example, the reflective area of ​​the grid line appears as a bright line in visible light, but the infrared temperature is consistent with the surrounding area. Correlation verification can avoid misjudging it as a hidden crack, which is often accompanied by temperature anomalies. If the pixel grayscale value G(x,y) in visible light is less than the threshold Gmin, and the infrared temperature T(x,y) is ≥Tvis+2°C, it is determined to be a potential real defect. During fusion, the bimodal feature weight of this pixel is enhanced. For example, a hidden crack appears as a subtle dark line in visible light, and the corresponding area temperature in infrared is 35°C, while the surrounding area is 30°C. By enhancing the weight, the subsequent detection model can more easily capture this feature. Through the spatiotemporal calibration matrix, it is ensured that the pixels (x, y) of the visible light and infrared images strictly correspond to the same physical position; Gmax takes the top 5% quantile of the grayscale value of the visible light image to adapt to changes in ambient light; Gmin takes the min (last 5% quantile, μg-30). The last 5% quantile covers the darkest 5% area in the image. Photovoltaic defects (such as hidden cracks and broken grids) usually occupy 1% to 3% of pixels. This ensures that Gmin can capture these extremely dark areas while excluding noise. A defect prior constraint is introduced. If Gmin is greater than the global mean grayscale μg-30, Gmin is forced to be set to the global mean grayscale μg-30 to avoid excessively high quantiles in extremely strong light environments, which may lead to missed detection of defective dark lines. μg is the global mean grayscale, and 30 is an empirical coefficient. Tvis takes the global mean temperature of the infrared image, reflecting the current working condition of the component; For each pixel (x, y), the weight w(x, y) is calculated, and the specific formula is .

[0022] Step 4, input the preprocessed optical image into GhostNet, extract shallow texture features Fvis∈R H ×W×C1 , including edges, grid lines and other optical details, input the preprocessed infrared image into lightweight ViT to generate global heat distribution features Fir∈R H×W×C2 , input the preprocessed EL image into CNN to extract electrical signal abnormality features Fel∈R H×W×C3 , and input the preprocessed 3D point cloud data into PointNet-Lite to extract topographic features F3D∈R N×D , where H is the feature map height, W is the feature map width, C1, C2, C3 are the number of feature channels, N is the number of points in the point cloud data, and D is the feature dimension of each point. The core value of the visible light image lies in the surface texture details, such as crack edges and grid line damage, which need to be preserved in shallow network with high resolution features while meeting the real-time requirements of online detection. GhostNet, as a lightweight CNN model, generates redundant features through Ghost module, effectively controls the calculation amount while maintaining the feature expression ability, adapts to the high resolution input of visible light image, and strengthens the extraction of low frequency texture features such as edges and corners through depth separable convolution and linear transformation. The defect representation of infrared image depends on global heat distribution pattern, such as temperature gradient of hot spot and continuous temperature rise of hidden crack, which needs to capture the temperature correlation between long distance pixels. The self-attention mechanism of Vision Transformer (ViT) is good at modeling global context, but traditional ViT has high computational complexity. Lightweight ViT reduces the embedding dimension and compresses the self-attention window, effectively reducing the calculation amount while maintaining the global feature extraction ability, adapting to the low resolution and single channel characteristics of infrared image. The defect of EL image is local electrical signal abnormality, such as luminous intensity attenuation at grid break and uneven luminous of micro crack, which belongs to local gray difference detection. CNN has natural advantages in local spatial features, such as connected regions and gray mutations, and has simple structure and fast training speed, which is suitable for high frequency dynamic acquisition of EL image. 3D point cloud data is unstructured and cannot be directly processed by traditional CNNs. PointNet-Lite, a lightweight point cloud processing network, maintains the permutation invariance of point clouds through symmetric functions while removing redundant computing nodes, such as reducing the number of fully connected layers, to adapt to the large-scale point cloud data generated by the laser speckle imaging module. For visible light features Fvis and infrared features Fir, expand the weight mask w to W1∈R in the channel dimension H×W×C1 and W2∈R H×W×C2 , we get Fvis'=Fvis⊙W1, Fir'=Fir⊙W2, ⊙ represents element-by-element multiplication, the eigenvalue of the reflective area is reduced, and the potential defect area is increased; The weight masks w for the visible light and infrared modes need to be expanded into three-dimensional tensors consistent with the number of feature channels of their respective modes, namely W1 and W2 above. The visible light feature channel C1 contains optical features such as edges and textures. Each channel corresponds to edge detection in different directions and scales. The infrared feature channel C2 contains thermal features such as temperature gradients and hot spots. Each channel corresponds to the temperature response of different regions, such as temperature rise in the central region and temperature uniformity in the edge region. Even if the 2D matrix of the weight mask w is the same, the physical meanings of the optical channel and the thermal channel are completely different and need to be weighted independently. For example, reflection only suppresses the visible light channel and does not affect the infrared temperature channel. Perform global average pooling GAP on the features of the four modalities, i.e. Gvis=GAP(Fvis'), Gir=GAP(Fir'), Gel=GAP(Fel), G3D=GAP(F3D), to obtain the one-dimensional feature vectors of the four modalities, and concatenate them into a unified feature vector Gconcat=[Gvis,Gir,Gel,G3D]; For image features H×W×C, we average the spatial dimensions H×W and compress them into a 1×1×C vector, or a one-dimensional vector C. This preserves channel-level global statistical information, such as the overall strength of each channel feature, while removing redundant spatial location details. The specific location of the defect may not affect the type judgment, but the global distribution is more important. For the point cloud features N×D, the number of points N is averaged and compressed to 1×D, converting the disordered point set into a global statistical vector to avoid feature fluctuations caused by the order of the point cloud arrangement; For example, the texture feature Fvis'∈RH×W×C1 extracted from the visible light image becomes Vvis∈RC1 after GAP, where each element is the average value of all pixels in the corresponding channel, reflecting the overall intensity of the texture of the channel, such as edges and grid lines; During defect detection, local reflections and noise points may interfere with judgment. GAP weakens the influence of local outliers through averaging operations and enhances global features, such as the overall temperature anomaly of the defect area in infrared images and the global distribution of electrical signal anomalies in EL images. Compared with the dimension reduction of the fully connected layer, GAP has no parameter learning, low computational complexity, and avoids overfitting caused by too many weights in the fully connected layer; Gconcat is input into the channel attention module, and the dimension is reduced to Ctotal / r through the first fully connected layer FC1, and nonlinearity is introduced through ReLU activation, where Ctotal=C1+C2+C3+D, and r is the compression ratio; The original dimension is restored through the second fully connected layer FC2, and the channel attention weight A is generated by Sigmoid activation. The attention weights of each modality Gvis, Gir, Gel, G3D, Avis, Air, Ael, A3D are extracted accordingly, where Avis is the first C1 dimension in A; The weight of each mode is globally averaged to obtain the modal-level importance coefficients qvis, qir, qel, and q3D. After normalizing the eigenvectors of each mode, the weighted sum is calculated according to the modal importance coefficients, i.e., Gattended = (Gvis*qvis+Gir*qir+Gel*qel+G3D*q3D) / (qvis+qir+qel+q3D). The fused feature Gattended is mapped to the defect category dimension K through the fully connected layer to obtain logits. Softmax activation is performed on the logits to generate the probability distribution P of each defect type. Combined with the preset dynamic threshold Tk, k∈[1,K], the defect categories with probabilities higher than the threshold are screened as detection results.

[0023] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0024] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An online detection method for photovoltaic module surface defects based on multimodal image fusion, characterized in that: include: Step 1: Install a visible light array camera and pair it with a polarized light source and ring LED, a short-wave infrared camera, an electroluminescent EL line array camera, and a laser speckle imaging module. Use a precision clock to synchronize the trigger signals of each sensor, and use Zhang's calibration method combined with a dynamic distortion compensation algorithm to generate a spatiotemporal calibration matrix. Step 2: Median filtering with dynamic window size is applied to the collected surface optical images, time sliding average combined with dynamic threshold segmentation is used for infrared and EL images, and image-guided outlier filtering is used for 3D point cloud data; Step 3: Based on the grayscale-temperature correlation, reflective false defects or potential real defects are determined based on the optical and infrared image pixels, and the pixel fusion weight w is calculated; Step 4: Input the optical image into GhostNet to extract the feature Fvis, the infrared image is lightweight ViT to generate the feature Fir, the EL image is used to extract the feature Fel through CNN, and the 3D point cloud is used to extract the feature F3D. The four modal features are globally averaged and pooled and spliced ​​into Gconcat. The input channel attention module generates modal weights, and the weighted fusion is obtained by Gattended. It is mapped to the defect category dimension K through the fully connected layer, and the Softmax output probability distribution P is combined with the dynamic threshold Tk to screen the defect categories that meet the P standard.

2. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 1, characterized in that: The specific steps of generating the spatiotemporal calibration matrix using Zhang's calibration method combined with the dynamic distortion compensation algorithm are as follows: Using a checkerboard calibration plate, 20 sets of images were collected at different positions within the field of view, and the camera intrinsic and extrinsic parameters were calculated using Zhang's calibration method. To address camera pose changes caused by production line vibration, a compensation algorithm is automatically triggered every 100 frames of image acquisition. The extrinsic parameter matrix is ​​updated in real time through feature point matching to ensure pixel-level alignment accuracy of less than 0.3 pixels. Taking the visible light image as the benchmark, the transformation matrices of infrared, EL and 3D point clouds are optimized by the mutual information maximization algorithm, so that the projection error of the same defect in each modal image is less than 1 pixel.

3. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 1, characterized in that: The specific steps for automatically adjusting the filter window according to the edge density of the local area are: Calculate the gradient magnitude of each pixel neighborhood, centering on the current pixel, and count the number of pixels Ne whose gradient magnitude is greater than the threshold Tg in the 7×7 pre-window; Calculate the edge density according to the formula ED = Ne / 49; The image is divided into three types of regions by edge density ED, and three types of filter windows are matched, namely 4. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 1, characterized in that: The average of five consecutive infrared or EL images is calculated to suppress high-frequency noise and retain slowly changing thermal anomalies or electrical signal anomalies. Based on the grayscale distribution of the average image, the Otsu algorithm is used to automatically calculate the threshold and segment the abnormal area.

5. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 1, characterized in that: The specific steps of using image-guided outlier filtering for 3D point cloud data are: Project the 3D point cloud onto the visible light image to obtain the 2D coordinates of each point; Perform Canny edge detection on visible light images, mark edge areas, and retain outliers in the point cloud of edge areas; Statistical outlier filtering is used on the point cloud in non-edge areas to remove points with a value greater than 2 times the standard deviation.

6. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 1, characterized in that: The specific steps for determining reflective false defects or potential real defects based on optical and infrared image pixels are as follows: For each pixel (x, y) in the visible light image, if its gray value G(x, y) > Gmax and the corresponding infrared temperature T(x, y) ≤ Tvis + 1°C, it is determined as a reflective pseudo-defect, and the weight of the visible light feature of this pixel is reduced during fusion; If the gray value of the pixel G(x, y) < Gmin in the visible light and the infrared temperature T(x, y) ≥ Tvis + 2°C, it is determined as a potential real defect, and the weight of the bimodal feature of this pixel is enhanced during fusion, where Gmax and Gmin are the thresholds of the pixel gray value, and Tvis is the global mean temperature of the infrared image.

7. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 1, characterized in that: According to the formula Calculate pixel fusion weights.

8. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 6, characterized in that: Gmax takes the 5% quantile of the gray value of the visible light image, and Gmin takes min{the 5% quantile, μg - 30}, where μg is the global mean gray value and 30 is an empirical coefficient.

9. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 1, characterized in that: For visible light features Fvis and infrared features Fir, expand the weight mask w to W1∈R in the channel dimension H×W×C1 and W2∈R H×W×C2 , and obtain the weighted feature Fvis'=Fvis⊙W1,Fir'=Fir⊙W2, where ⊙ represents element-by-element multiplication, H is the feature map height, W is the feature map width, and C1 and C2 are the number of feature channels.

10. The method for online detection of photovoltaic module surface defects based on multimodal image fusion according to claim 1, characterized in that: The specific steps to obtain Gattended by generating modal weights according to the input channel attention module are as follows: Input Gconcat into the channel attention module, reduce the dimension to Ctotal / r through the first fully connected layer FC1, and introduce non-linearity through ReLU activation; Restore to the original dimension through the second fully connected layer FC2, generate the channel attention weight A through Sigmoid activation, and extract the attention weights of each modality Gvis, Gir, Gel, G3D respectively to obtain Avis, Air, Ael, A3D; Perform global averaging on the weights of each modality to obtain the modality-level importance coefficients qvis, qir, qel, q3D, and after normalizing the feature vectors of each modality, calculate the fused feature according to the formula Gattended = (Gvis * qvis + Gir * qir + Gel * qel + G3D * q3D) / (qvis + qir + qel + q3D), where Ctotal = C1 + C2 + C3 + D and r is the compression ratio.

Images