A photovoltaic module multi-modal feature fusion defect detection method and system

Abstract

The application discloses a kind of photovoltaic module multimodal feature fusion defect detection method and system, it is related to photovoltaic module defect detection technical field, the method includes obtaining the image of photovoltaic module and pre-processing;Respective image is carried out multi-scale feature extraction, obtains infrared and visible light multi-scale feature map set;Cross-modal semantic alignment is carried out to each scale double mode feature map, and cross-layer interactive fusion is executed, obtains cross-layer enhanced feature map set;The confidence score of infrared and visible light mode is dynamically evaluated;Different defect types are assigned different modal contribution weights by defect type perception gate network, and the cross-layer enhanced feature map set is adaptively weighted and fused, and multi-modal fusion feature map is obtained;Input defect detection network outputs defect detection result.The application solves the problem of semantic scale mismatch between modes, static setting of fusion weight, and realizes accurate detection of multiple types of photovoltaic module defects.

Description

Technical Field

[0001] This invention relates to the field of photovoltaic module defect detection technology, and in particular to a method and system for detecting defects by fusing multimodal features of photovoltaic modules. Background Technology

[0002] As the core equipment of photovoltaic power plants, photovoltaic modules can develop various defects at every stage of production, transportation, installation, and operation, such as hot spots, cracks, microcracks, stains, and diode short circuits. These defects not only reduce the power generation efficiency of photovoltaic power plants but may also cause module damage or even fires, posing a serious threat to the economic benefits and safe operation of the power plant. Therefore, conducting efficient and accurate defect detection of photovoltaic modules at each stage is of significant practical importance.

[0003] Currently, photovoltaic module defect detection mainly falls into two technical categories: voltage / current data-based detection methods and image-based detection methods. Voltage / current data-based detection requires external data acquisition devices, but due to the massive amount of data, the detection efficiency is low, making it difficult to meet the needs of the large-scale development of the photovoltaic industry. On the other hand, image-based detection methods, with their advantages of fast data acquisition speed and ability to identify a wide range of defect types, have become the mainstream technical approach.

[0004] In image-based detection methods, multimodal fusion detection methods have become a research hotspot in recent years because single-modal images are insufficient to comprehensively characterize various defect features of photovoltaic modules. Existing technologies mainly suffer from the following shortcomings: First, most multimodal fusion methods employ fixed-level feature fusion strategies. Since infrared images encode coarse-grained global structural information such as temperature distribution, while visible light images encode fine-grained local details such as texture and edges, there is a natural difference in semantic granularity between the two. Single-scale fusion leads to the dilution of texture information of small defects in high-level semantic features by background information, and existing methods fail to effectively solve the problem of semantic scale mismatch between modes. Second, when fusing different modal features, existing methods typically use fixed fusion weights or manually designed static strategies, failing to fully consider the dynamic changes in the amount and reliability of information in each modality under different environmental conditions, resulting in insufficient robustness in complex environments. Third, considering the differences in the dependence of different defect types of photovoltaic modules on infrared and visible light modal information, existing methods lack differentiated modal contribution allocation mechanisms and fail to achieve adaptive fusion strategies tailored to specific defects.

[0005] In addition, while existing end-to-end multi-source image detection schemes can achieve high detection accuracy, their inference speed is limited when deployed on edge computing platforms, and they lack a model closed-loop optimization mechanism based on deployment feedback. Summary of the Invention

[0006] In view of the above-mentioned problems, this invention proposes a multi-modal feature fusion defect detection method and system for photovoltaic modules to solve the problems of semantic scale mismatch between modes, static setting of fusion weights and lack of differentiated response for defect types in the prior art. It realizes high-precision, adaptive and closed-loop optimizable intelligent detection of multiple types of defects in photovoltaic modules in complex environments.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide a method for detecting defects in photovoltaic modules by fusing multimodal features, comprising: acquiring infrared images and visible light images of the photovoltaic module to be detected, and performing preprocessing;

[0009] Multi-scale feature extraction was performed on the preprocessed infrared and visible light images to obtain sets of infrared multi-scale feature maps and sets of visible light multi-scale feature maps.

[0010] For each scale, cross-modal semantic alignment is performed on the infrared feature map and visible light feature map of the same scale to obtain the aligned fused feature map at each scale. Then, the aligned fused feature maps at each scale are cross-layer interactively fused to obtain a cross-layer enhanced feature map set.

[0011] Based on the modal quality metrics of infrared and visible light images, the confidence scores of infrared and visible light modes in the current scene are evaluated respectively.

[0012] Based on the confidence score, differentiated modal contribution weights are assigned to different defect types, and the modal contribution weights are used to adaptively weight and fuse the cross-layer enhanced feature map set to obtain a multimodal fusion feature map.

[0013] The multimodal fusion feature map is input into the defect detection network, and the defect detection result is output.

[0014] As a preferred embodiment of the multimodal feature fusion defect detection method for photovoltaic modules described in this invention, the method includes: acquiring infrared and visible light images of the photovoltaic module to be detected, and performing preprocessing, including:

[0015] Edge-preserving filtering is used to denoise infrared and visible light images respectively, and the filter kernel parameters are adaptively set according to the noise level of each modality image.

[0016] The contrast of the denoised infrared image is enhanced by contrast-limited adaptive histogram equalization, and the illumination equalization of the denoised visible light image is processed by multi-scale Retinex algorithm.

[0017] Using the visible light image as a reference, the phase correlation method is used to calculate the subpixel registration parameters and the infrared image is geometrically transformed to achieve subpixel-level spatial alignment between the two modal images, and then they are uniformly scaled to the same target size.

[0018] The pixel values ​​of the two modal images after registration and size unification are normalized to a preset range.

[0019] As a preferred embodiment of the multi-modal feature fusion defect detection method for photovoltaic modules described in this invention, the method involves: performing multi-scale feature extraction on the preprocessed infrared image and visible light image respectively to obtain an infrared multi-scale feature map set and a visible light multi-scale feature map set, including:

[0020] The preprocessed infrared and visible light images are respectively input into a feature extraction network with the same structure and shared weights.

[0021] The feature extraction network contains multiple cascaded feature extraction modules. The output of the current module serves as the input of the next module. Each feature extraction module consists of a convolutional layer, a batch normalization layer, and an activation function layer in sequence. At the end of each module, downsampling is performed through a convolution with a stride of 2.

[0022] After downsampling at each level, infrared and visible light feature maps of the corresponding scales are output respectively. The shallow feature map has a higher resolution than the deep feature map, and the receptive field of the deep feature map is larger than that of the shallow feature map.

[0023] As a preferred embodiment of the multimodal feature fusion defect detection method for photovoltaic modules described in this invention, the method involves: performing cross-modal semantic alignment on infrared and visible light feature maps of the same scale to obtain aligned fused feature maps at each scale; then performing cross-layer interactive fusion on the aligned fused feature maps at each scale to obtain a cross-layer enhanced feature map set, including:

[0024] Infrared and visible light feature maps of the same scale are unfolded into two-dimensional feature matrices along the channel dimension. The matrix product between the transpose of the infrared feature matrix and the visible light feature matrix is ​​calculated. After Softmax normalization, the cross-modal similarity weight matrix at this scale is obtained. The cross-modal similarity weight matrix is ​​multiplied again with the visible light feature matrix to perform weighted modulation of the visible light features. The modulated visible light features and infrared features are then concatenated along the channel dimension and fused through a convolutional layer to output the aligned and fused feature map at this scale.

[0025] The aligned and fused feature maps at each scale are input into the cross-layer interactive fusion module. The cross-layer interactive fusion module performs bidirectional cross-layer connections on the aligned and fused feature maps at each scale: the deep aligned and fused feature map is upsampled, and the upsampled deep features are concatenated with the adjacent shallow features along the channel dimension and then fused through a convolutional layer; the shallow aligned and fused feature map is downsampled by a convolution with a stride of 2, and the downsampled shallow features are concatenated with the adjacent deep features along the channel dimension and then fused through a convolutional layer; after bidirectional cross-layer connections, the cross-layer enhanced feature maps at each scale are output.

[0026] As a preferred embodiment of the multimodal feature fusion defect detection method for photovoltaic modules described in this invention, the method involves: evaluating the confidence scores of the infrared mode and the visible light mode in the current scene based on modal quality indices of infrared and visible light images, respectively, including:

[0027] Modal quality indices are calculated for the preprocessed infrared and visible light images, respectively. The modal quality indices include: the standard deviation of global pixel values ​​as a contrast index; the ratio of the mean local gradient magnitude to the mean global gradient magnitude as a signal-to-noise ratio index; the kurtosis coefficient of the histogram as a brightness distribution uniformity index; and the mean absolute value of the temperature difference between adjacent pixels in the infrared image as a temperature gradient magnitude index.

[0028] The contrast ratio, signal-to-noise ratio, brightness distribution uniformity, and temperature gradient amplitude of infrared images are combined into an infrared quality feature vector, and the contrast ratio, signal-to-noise ratio, and brightness distribution uniformity of visible light images are combined into a visible light quality feature vector.

[0029] The infrared quality feature vector and the visible light quality feature vector are respectively input into the confidence evaluation network, which includes a first fully connected layer and a second fully connected layer connected in sequence.

[0030] As a preferred embodiment of the multimodal feature fusion defect detection method for photovoltaic modules described in this invention, the method includes: assigning differentiated modal contribution weights to different defect types, and using the modal contribution weights to adaptively weightedly fuse the cross-layer enhanced feature map set to obtain a multimodal fused feature map, including: performing global average pooling on the cross-layer enhanced feature maps at each scale to obtain global feature description vectors at each scale, and concatenating the global feature description vectors at each scale along the channel dimension to form a multi-scale global feature vector;

[0031] The multi-scale global feature vector is input into a defect type-aware gating network, which includes a third fully connected layer, a fourth fully connected layer, and a gated output layer connected in sequence.

[0032] The infrared modal confidence scores and visible light modal confidence scores are multiplied by the infrared contribution weights and visible light contribution weights of each defect category, respectively, and then normalized to generate the modal fusion weight matrix for each defect category.

[0033] The cross-layer enhanced feature map set is weighted and multiplied with the modal fusion weight matrix under each defect category to obtain the category-specific fusion feature map of each defect category. The category-specific fusion feature maps of each defect category are concatenated along the channel dimension, and the number of channels is compressed by the convolutional layer to output the multimodal fusion feature map.

[0034] As a preferred embodiment of the multimodal feature fusion defect detection method for photovoltaic modules according to the present invention, the method involves inputting the multimodal fusion feature map into a defect detection network and outputting defect detection results, including:

[0035] The multimodal fused feature map is input into a defect detection network, which includes a region proposal network and a detection head network.

[0036] The region proposal network slides anchor boxes on the multimodal fused feature map with a preset stride. Each anchor box outputs the corresponding defect existence probability and preliminary position offset through convolutional layers and fully connected layers. Candidate regions are selected based on the defect existence probability, and the positions of the candidate regions are corrected based on the preliminary position offset.

[0037] The feature map of the candidate region is cropped and scaled to a fixed size and input into the detection head network. The detection head network contains a fifth fully connected layer and a sixth fully connected layer connected in sequence. The output of the fifth fully connected layer is activated by the ReLU activation function and then input into the sixth fully connected layer. The sixth fully connected layer outputs the defect category prediction score and the final position offset in parallel.

[0038] Non-maximum suppression is applied to the defect category prediction score and the final position offset, and the defect detection results are output.

[0039] Secondly, embodiments of the present invention provide a multimodal feature fusion defect detection system for photovoltaic modules, including: an image acquisition and preprocessing module, used to acquire infrared images and visible light images of the photovoltaic module to be detected, and perform preprocessing;

[0040] The multi-scale feature extraction module is used to extract multi-scale features from the preprocessed infrared image and the visible light image respectively, to obtain the infrared multi-scale feature map set and the visible light multi-scale feature map set.

[0041] The semantic alignment and cross-layer fusion module is used to perform cross-modal semantic alignment of infrared feature maps and visible light feature maps of the same scale for each scale, to obtain aligned fused feature maps at each scale, and then perform cross-layer interactive fusion of the aligned fused feature maps at each scale to obtain a set of cross-layer enhanced feature maps.

[0042] The confidence assessment module is used to evaluate the confidence scores of the infrared mode and the visible light mode in the current scene based on the modal quality index of infrared and visible light images, respectively.

[0043] The differentiated gated fusion module is used to assign differentiated modal contribution weights to different defect types based on the confidence score, and to perform adaptive weighted fusion of the cross-layer enhanced feature map set using the modal contribution weights to obtain a multimodal fusion feature map;

[0044] The defect detection and output module is used to input the multimodal fusion feature map into the defect detection network and output the defect detection result.

[0045] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, they implement the steps of a method for detecting defects in multimodal feature fusion of photovoltaic modules as described in the first aspect of the present invention.

[0046] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, they implement the steps of a method for detecting defects in a photovoltaic module by multimodal feature fusion as described in the first aspect of the present invention.

[0047] The beneficial effects of this invention are as follows: This invention aligns the infrared global temperature structure and visible light local texture details at multiple semantic levels through a multi-scale semantic alignment and cross-layer interactive fusion mechanism, and enhances the two through bidirectional cross-layer connections, effectively solving the problem of fine-grained defect features being diluted by higher-level semantics in existing single-scale fusion methods; and overcomes the shortcomings of traditional fixed-weight fusion in terms of insufficient robustness under complex environments such as lighting and weather by using a dynamic modality confidence evaluation module to calculate the image quality index of each modality in real time and adaptively generate confidence scores. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the method of the present invention.

[0050] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0052] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0053] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0054] Example 1

[0055] Reference Figure 1 This is the first embodiment of the present invention, which provides a method for detecting defects in photovoltaic modules through multimodal feature fusion, comprising:

[0056] S1: Acquire infrared and visible light images of the photovoltaic module to be tested and perform preprocessing.

[0057] Furthermore, edge-preserving filtering is used to denoise the infrared and visible light images respectively, and the filter kernel parameters are adaptively set according to the noise level of each modality image;

[0058] The contrast of the denoised infrared image is enhanced by contrast-limited adaptive histogram equalization, and the illumination equalization of the denoised visible light image is processed by multi-scale Retinex algorithm.

[0059] Using the visible light image as a reference, the phase correlation method is used to calculate the subpixel registration parameters and the infrared image is geometrically transformed to achieve subpixel-level spatial alignment between the two modal images, and then they are uniformly scaled to the same target size.

[0060] The pixel values ​​of the two modal images after registration and size unification are normalized to a preset range.

[0061] It should be noted that in this embodiment, a dual-light gimbal device equipped with an infrared thermal imager and a visible light camera is used to simultaneously acquire infrared and visible light images of the photovoltaic modules. The infrared thermal imager uses an uncooled focal plane detector with a response band of 8 to 14 micrometers, a temperature measurement range of -20 degrees Celsius to 150 degrees Celsius, a temperature resolution of 0.05 degrees Celsius (at 30 degrees Celsius), and an image resolution of 640×512 pixels. The visible light camera uses a CMOS image sensor with an image resolution of 1920×1080 pixels. The device is fixed on an inspection drone or a fixed bracket, with a acquisition distance of 5 to 15 meters to ensure that the photovoltaic modules are completely imaged within the field of view.

[0062] During actual data acquisition, due to the environmental conditions of the photovoltaic power station and the inherent factors of the equipment, the original infrared and visible light images suffer from varying degrees of noise, insufficient contrast, uneven illumination, and spatial position deviations. Preprocessing is required, as detailed below:

[0063] Bilateral filtering was used to denoise both infrared and visible light images. Bilateral filtering comprehensively considers the spatial proximity and pixel value similarity between pixels, effectively preserving image edge details while smoothing noise. For infrared images affected by detector thermal noise and readout noise, the spatial kernel coefficient was set to 3.0, and the value kernel coefficient to 0.15. For visible light images affected by CMOS sensor noise and illumination noise, the spatial kernel coefficient was set to 2.0, and the value kernel coefficient to 0.10. The filtering window size was 11×11 pixels to ensure sufficient noise smoothing while protecting structural information such as cell edges and crack edges from damage.

[0064] Contrast enhancement was achieved by employing contrast-limited adaptive histogram equalization on the denoised infrared image. The infrared image was divided into an 8x8 grid of local sub-blocks, each measuring 80x64 pixels. The grayscale histogram of each local sub-block was cropped, with a cropping limit set to 2% of the total pixels in the sub-block. The cropped excess pixels were then evenly redistributed across the grayscale levels of the histogram to prevent excessive local contrast amplification that could lead to noise amplification. Independent histogram equalization was performed on each sub-block, and then bilinear interpolation was used to eliminate the boundary stitching effect between sub-blocks, resulting in a contrast-enhanced infrared image that significantly improved the distinction between hot spots and temperature anomaly areas and the background.

[0065] The denoised visible light image was processed using a multi-scale Retinex algorithm for illumination equalization. The visible light image was converted from the RGB color space to the HSV color space, and illumination correction was applied only to the V channel (luminance channel), while the H and S channels remained unchanged. Gaussian wrap functions at three scales (large, medium, and small) were selected, with scales of 120, 60, and 15 pixels respectively. Illuminance components were extracted at each scale, and the reflectance components were obtained by removing the illuminance components from the original V channel. The reflectance components from the three scales were then weighted and synthesized to obtain the illumination-equalized V channel. The corrected V channel was then merged back with the original H and S channels and converted back to the RGB color space. This processing effectively suppressed uneven illumination caused by variations in solar altitude angle, cloud cover, and different module tilt angles in photovoltaic power plant scenarios.

[0066] Using the visible light image as a reference image, a phase-correlation-based subpixel registration algorithm is employed to perform geometric transformation on the infrared image. First, two-dimensional fast Fourier transforms are performed on both the infrared and visible light images to calculate their cross-power spectra. After performing an inverse Fourier transform on the cross-power spectra, the impulse response matrix is ​​obtained. The integer pixel position where the impulse peak is located represents the integer pixel shift of the infrared image relative to the visible light image.

[0067] Based on the integer pixel translation, a 3×3 neighborhood window is constructed around the peak position. A quadratic surface is fitted using the phase correlation response values ​​within the neighborhood window. After solving for the fitting parameters of the quadratic surface using the least squares method, the partial derivative of the fitting function is calculated and set to zero to obtain the sub-pixel level peak offset. The sum of the integer pixel translation and the sub-pixel offset is the sub-pixel translation. Simultaneously, the rotation between the two images is determined based on the skewness of the cross-power spectrum peak position. The calculated sub-pixel translation and rotation are used to construct an affine transformation matrix. Bicubic interpolation is then used to perform an affine transformation on the infrared image, achieving sub-pixel alignment between the infrared and visible light images in spatial position, with a registration accuracy better than 0.1 pixels.

[0068] After registration, the visible light image and the transformed infrared image are uniformly scaled to a preset target size of 512×512 pixels using bicubic interpolation, so that the two modal images have the same spatial resolution, which is convenient for subsequent batch input into the feature extraction network.

[0069] S2: Perform multi-scale feature extraction on the preprocessed infrared and visible light images respectively to obtain infrared multi-scale feature map sets and visible light multi-scale feature map sets.

[0070] Furthermore, the preprocessed infrared and visible light images are input into feature extraction networks with identical structures and shared weights.

[0071] The feature extraction network contains multiple cascaded feature extraction modules. The output of the current module serves as the input of the next module. Each feature extraction module consists of a convolutional layer, a batch normalization layer, and an activation function layer in sequence. The end of each module is downsampled by a convolution with a stride of 2.

[0072] After downsampling at each level, infrared and visible light feature maps of the corresponding scales are output respectively. The shallow feature map has a higher resolution than the deep feature map, and the receptive field of the deep feature map is larger than that of the shallow feature map.

[0073] It should be noted that in this embodiment, the preprocessed infrared and visible light images are input into a feature extraction network with identical structure and shared weight parameters. This feature extraction network employs a lightweight convolutional neural network architecture, built upon the inverse residual structure of MobileNetV3. This architecture controls the number of parameters while ensuring feature representation capabilities, facilitating subsequent deployment at the edge. The infrared and visible light images share the same set of network weight parameters, ensuring that the features of both modalities are mapped to the same feature space, laying the foundation for subsequent cross-modal semantic alignment.

[0074] The feature extraction network comprises five cascaded feature extraction stages. The output of the current stage serves as the input to the next stage. Each stage contains several feature extraction modules. At the end of each stage, downsampling is performed through convolutions with a stride of 2, which halves the spatial resolution of the feature map at each stage and doubles the number of channels at each stage. After downsampling at each stage, feature maps of the corresponding scale are output, resulting in five scales of infrared feature maps and five scales of visible light feature maps.

[0075] Given an input image size of 512×512×3 (height×width×number of channels), after five stages of progressive downsampling, the specific parameters of the feature maps at each scale are as follows:

[0076] First scale (shallowest layer): After the first stage of downsampling, the feature map size is 256×256×16, with the highest spatial resolution and the smallest receptive field of 3×3 pixels. The feature map at this scale retains the richest spatial location information and edge texture details.

[0077] Second scale: After the second stage of downsampling, the feature map size is 128×128×24, the receptive field is expanded to 7×7 pixels, and the encoding of local texture combination information begins;

[0078] Third scale: After the third stage of downsampling, the feature map size is 64×64×40, the receptive field is expanded to 15×15 pixels, and intermediate semantic features are encoded.

[0079] Fourth scale: After the fourth stage of downsampling, the feature map size is 32×32×80, the receptive field is expanded to 31×31 pixels, and global structural information is encoded;

[0080] Fifth scale (deepest layer): After the fifth stage of downsampling, the feature map size is 16×16×112, with the lowest spatial resolution and the largest receptive field of 63×63 pixels, encoding the global temperature distribution and overall component structure information.

[0081] Shallow feature maps (first and second scales) have high resolution and small receptive field, effectively preserving local details such as cell edges, electrode grid lines, and crack textures of photovoltaic modules; deep feature maps (fourth and fifth scales) have low resolution and large receptive field, effectively encoding global structural information such as the overall temperature field distribution of the module and the location and extent of hot spot areas; the third scale is between the two, playing a transitional role.

[0082] Each feature extraction module consists of a convolutional layer, a batch normalization layer, and an activation function layer. The convolutional layer uses a 3×3 kernel with a stride of 1 and uses "same padding" (the output feature map size is the same as the input). The convolution operation first extracts local features by sliding along the spatial dimension, and then stacks the extraction results of each channel along the channel dimension for output. The batch normalization layer follows the convolutional layer and performs batch normalization on the convolutional output: during training, the mean and variance of the current batch of data are calculated, and the feature map is standardized before scaling and translation; during inference, the global mean and variance statistically derived from the training set are used for standardization. The batch normalization layer accelerates network convergence and acts as a regularization mechanism.

[0083] The activation function layer uses the ReLU activation function, which performs a non-linear mapping on each feature element of the batch normalization layer output: when the input value is greater than 0, the output is equal to the input value; when the input value is less than or equal to 0, the output is forced to be 0. The ReLU activation function introduces non-linear transformation capability while avoiding the gradient vanishing problem.

[0084] The convolutional layer with a stride of 2 at the end of each stage: The last feature extraction module in each stage changes the stride of the convolutional layer from 1 to 2, and with the padding method, reduces the height and width of the feature map to half of their original values, while doubling the number of channels, thereby achieving the simultaneous completion of spatial downsampling and channel expansion.

[0085] The preprocessed infrared image (512×512×3) is first input into a shared-weight feature extraction network, and then undergoes five stages of progressive downsampling and feature transformation. The first stage outputs an infrared feature map F_I^1 (256×256×16), which encodes the edges of the battery cells and the local temperature gradient distribution in the infrared image; the second stage outputs F_I^2 (128×128×24), which encodes the contours of local temperature anomaly regions; the third stage outputs F_I^3 (64×64×40), which encodes the shape features of the hot spot region; the fourth stage outputs F_I^4 (32×32×80), which encodes the overall temperature field distribution pattern of the component; and the fifth stage outputs F_I^5 (16×16×112), which encodes the location of the global highest temperature region and the overall temperature difference distribution trend.

[0086] The preprocessed visible light image (512×512×3) is input into the same feature extraction network. The first stage outputs a visible light feature map F_V^1 (256×256×16), which encodes the fine edges, grid texture, and crack outlines of the battery cells in the visible light image; the second stage outputs F_V^2 (128×128×24), which encodes local texture combinations and surface stain areas; the third stage outputs F_V^3 (64×64×40), which encodes the crack direction and the distribution of large-area stains; the fourth stage outputs F_V^4 (32×32×80), which encodes the overall surface structure layout of the component; and the fifth stage outputs F_V^5 (16×16×112), which encodes the global background illumination distribution and the general outline of the component.

[0087] S3: For each scale, perform cross-modal semantic alignment on the infrared feature map and visible light feature map of the same scale to obtain the aligned fused feature map at each scale. Then, perform cross-layer interactive fusion on the aligned fused feature maps at each scale to obtain a cross-layer enhanced feature map set.

[0088] Furthermore, the infrared and visible light feature maps at the same scale are unfolded into two-dimensional feature matrices along the channel dimension. The matrix product between the transposes of the infrared and visible light feature matrices is calculated. After Softmax normalization, the cross-modal similarity weight matrix at this scale is obtained. The cross-modal similarity weight matrix is ​​multiplied again with the visible light feature matrix to perform weighted modulation of the visible light features. The modulated visible light features and infrared features are then concatenated along the channel dimension and fused through a convolutional layer to output the aligned and fused feature map at this scale.

[0089] The aligned and fused feature maps at each scale are input into the cross-layer interactive fusion module. The cross-layer interactive fusion module performs bidirectional cross-layer connections on the aligned and fused feature maps at each scale: the deep aligned and fused feature map is upsampled, and the upsampled deep features are concatenated with the adjacent shallow features along the channel dimension and then fused through a convolutional layer; the shallow aligned and fused feature map is downsampled by a convolution with a stride of 2, and the downsampled shallow features are concatenated with the adjacent deep features along the channel dimension and then fused through a convolutional layer; after bidirectional cross-layer connections, the cross-layer enhanced feature maps at each scale are output.

[0090] It should be noted that in this embodiment, for each of the five scales, the infrared feature map and the visible light feature map of the same scale are performed cross-modal semantic alignment to obtain the aligned and fused feature map at each scale. Taking the third scale as an example, the feature map size at this scale is 64×64×40, and the processing flow for the other scales is completely the same.

[0091] First, the infrared feature map F_I^3 (64×64×40) at the third scale is expanded along the channel dimension into a two-dimensional feature matrix M_I^3, with a size of 4096×40 (4096 represents the total number of spatial locations in the 64×64 matrix, and 40 represents the number of channels). Each row in the matrix corresponds to a feature vector of a spatial location across all channels. Simultaneously, the visible light feature map F_V^3 (64×64×40) is expanded along the channel dimension into a two-dimensional feature matrix M_V^3, also with a size of 4096×40.

[0092] The matrix product between the infrared feature matrix M_I^3 and the transpose (M_V^3)^T of the visible light feature matrix is ​​calculated to obtain the original similarity matrix of size 4096×4096. The element in the i-th row and j-th column of this matrix represents the feature similarity between the i-th spatial location of the infrared feature map and the j-th spatial location of the visible light feature map. The original similarity matrix is ​​then subjected to Softmax normalization row by row, so that the sum of all elements in each row is 1, resulting in the cross-modal similarity weight matrix A^3, also of size 4096×4096.

[0093] The cross-modal similarity weight matrix A^3 is multiplied again by the visible light feature matrix M_V^3, resulting in a 4096×40 matrix, to obtain the weighted modulated visible light feature matrix. This operation essentially utilizes the similarity relationship between each position in the infrared feature map and all positions in the visible light feature map to perform a global context-weighted reorganization of the visible light features: for a given spatial position in the infrared feature map, the corresponding row in the similarity weight matrix describes the degree of similarity between that position and all spatial positions in the visible light feature map. Using these weights, the features at all visible light positions are weighted and summed, enhancing the semantically relevant parts of the visible light features and suppressing the irrelevant parts.

[0094] The weighted modulated visible light feature matrix is ​​restored to a feature map form (64×64×40), and then concatenated with the original infrared feature map F_I^3 (64×64×40) along the channel dimension to obtain a 64×64×80 concatenated feature map. This concatenated feature map is then input into a 3×3 convolutional layer for channel fusion, with 40 kernels, a stride of 1, and same padding. The output feature map size is 64×64×40, which is the aligned and fused feature map F_A^3 at the third scale.

[0095] Repeat the above process, processing each of the five scales one by one, to obtain the set of aligned and fused feature maps {F_A^1, F_A^2, F_A^3, F_A^4, F_A^5} at each scale. The spatial resolution and number of channels at each scale are as follows: F_A^1 is 256×256×16, F_A^2 is 128×128×24, F_A^3 is 64×64×40, F_A^4 is 32×32×80, and F_A^5 is 16×16×112.

[0096] The aligned and fused feature maps of the five scales are input into the cross-layer interactive fusion module, which performs bidirectional cross-layer connections between adjacent scales, including a top-down global structure guidance path and a bottom-up local detail supplementation path. The interaction between the third and fourth scales is used as an example for illustration.

[0097] Top-down approach (global structural information guiding local details): The fourth-scale (deep) aligned and fused feature map F_A^4 (32×32×80) is upsampled using bilinear interpolation, doubling its spatial resolution to 64×64 while maintaining the number of channels at 80. The upsampled deep features are then concatenated with the third-scale (shallow) aligned and fused feature map F_A^3 (64×64×40) along the channel dimension, resulting in a 64×64×120 concatenated feature map. This concatenated feature map is then fused into a 3×3 convolutional layer with 40 kernels and a stride of 1, outputting a feature map of 64×64×40. This approach allows the global temperature distribution structure and hotspot region localization information in the deep feature map to be transmitted from top to bottom to the shallow layer, guiding the shallow features to perceive global contextual information while focusing on local texture details, thus avoiding misjudgment of local anomalies.

[0098] Bottom-up approach (local details supplement global structure): The third-scale (shallow) aligned and fused feature map F_A^3 (64×64×40) is downsampled using a 3×3 convolution with a stride of 2, reducing its spatial resolution by half to 32×32 and doubling the number of channels to 80. The downsampled shallow features are then concatenated with the fourth-scale (deep) aligned and fused feature map F_A^4 (32×32×80) along the channel dimension, resulting in a 32×32×160 concatenated feature map. This concatenated feature map is then fused into a 3×3 convolutional layer with 80 kernels and a stride of 1, outputting a feature map of 32×32×80. This approach allows crack texture details and surface stain edge information in the shallow feature map to be transmitted from bottom to top to the deeper layers, supplementing the missing fine-grained information in the deep features and incorporating local detail discrimination criteria into the global structural features.

[0099] The above-described bidirectional cross-layer connection operation is performed on each adjacent scale pair (first and second, second and third, third and fourth, and fourth and fifth) across the five scales. Specifically, the first scale (shallowest layer) only participates in the bottom-up path (interacting with the second scale), the fifth scale (deepest layer) only participates in the top-down path (interacting with the fourth scale), and the three intermediate scales participate in both paths simultaneously.

[0100] After bidirectional cross-layer connections, five scales are output as cross-layer enhanced feature maps F_C^1 to F_C^5, respectively. The spatial resolution and number of channels at each scale are consistent with the input aligned and fused feature map, which are 256×256×16, 128×128×24, 64×64×40, 32×32×80, and 16×16×112, respectively, forming the cross-layer enhanced feature map set {F_C^1, F_C^2, F_C^3, F_C^4, F_C^5}. In this cross-layer enhanced feature map, the shallow features retain the original high-resolution texture details and incorporate global structural guidance information from the deeper layers; the deep features retain the global temperature distribution structure and supplement local detail information from the shallow layers, achieving full fusion and enhancement of infrared temperature information and visible light texture information at each scale.

[0101] S4: Based on the modal quality metrics of infrared and visible light images, evaluate the confidence scores of the infrared and visible light modes in the current scene, respectively.

[0102] Further, modal quality indices are calculated for the preprocessed infrared and visible light images, respectively. The modal quality indices include: calculating the standard deviation of global pixel values ​​as a contrast index, calculating the ratio of the mean local gradient magnitude to the mean global gradient magnitude as a signal-to-noise ratio index, calculating the histogram kurtosis coefficient as a brightness distribution uniformity index, and calculating the mean absolute value of the temperature difference between adjacent pixels in the infrared image as a temperature gradient magnitude index.

[0103] The contrast ratio, signal-to-noise ratio, brightness distribution uniformity, and temperature gradient amplitude of infrared images are combined into an infrared quality feature vector, and the contrast ratio, signal-to-noise ratio, and brightness distribution uniformity of visible light images are combined into a visible light quality feature vector.

[0104] The infrared quality feature vector and the visible light quality feature vector are respectively input into the confidence evaluation network, which includes a first fully connected layer and a second fully connected layer connected in sequence.

[0105] It should be noted that in this embodiment, the standard deviation of the global pixel values ​​is used as a contrast indicator. For infrared images, the standard deviation of the grayscale values ​​of all pixels is calculated by traversing 512×512 pixels. The larger the standard deviation, the more dispersed the pixel values ​​are in the image, the more obvious the temperature contrast, and the higher the distinction between abnormal temperature areas such as hot spots and normal areas. For visible light images, the standard deviation of the grayscale values ​​of the entire image is also calculated. The larger the standard deviation, the stronger the contrast between light and dark areas, and the clearer the texture and edges. In the specific calculation, the mean of all pixels in the image is calculated first, then the average of the squares of the differences between each pixel and the mean is calculated, and finally the square root is taken to obtain the standard deviation.

[0106] The ratio of the mean local gradient magnitude to the mean global gradient magnitude is used as the signal-to-noise ratio (SNR) metric. First, the Sobel operator is used to calculate the gradient magnitudes in the horizontal and vertical directions of the image. The square root of the sum of the squares of the gradient magnitudes in both directions is taken to obtain the gradient magnitude of each pixel. The mean global gradient magnitude is the average of the gradient magnitudes of all pixels in the entire image. The mean local gradient magnitude is calculated as follows: the image is divided into 8×8 local sub-blocks, each 64×64 pixels in size. The mean gradient magnitude of pixels within each sub-block is calculated, and then the average of the mean gradient magnitudes of all sub-blocks is taken to obtain the mean local gradient magnitude. The SNR metric is the ratio of the mean local gradient magnitude to the mean global gradient magnitude. A larger ratio indicates significant gradient differences between local regions in the image, rich effective texture information, and low noise interference. A ratio close to 1 or less than 1 indicates that the overall gradient distribution of the image is uniform, lacking obvious texture features or with high noise levels.

[0107] The kurtosis coefficient of the histogram is used as an indicator of brightness distribution uniformity. The image pixel gray values ​​([0,1] interval) are divided into 256 gray levels, and the pixel frequency of each gray level is counted to construct a gray-level histogram. The formula for calculating the kurtosis coefficient is: first calculate the ratio of the fourth central moment of the histogram to the square of the second central moment, and then subtract 3 (to make the kurtosis coefficient of the normal distribution 0). When the kurtosis coefficient is positive, it indicates that the histogram distribution is relatively steep, the pixel gray values ​​are concentrated in a few gray levels, the image is generally too bright or too dark, and the uniformity is poor; when the kurtosis coefficient is negative, it indicates that the histogram distribution is relatively flat, the pixel gray values ​​are evenly distributed across multiple gray levels, and the image brightness uniformity is good.

[0108] The average absolute value of the temperature difference between adjacent pixels in an infrared image is calculated as the temperature gradient magnitude index. For each pixel position (i,j) in the infrared image, the absolute value of the gray value difference between it and its right-hand neighbor (i,j+1), and the absolute value of the gray value difference between it and its lower-hand neighbor (i+1,j), are calculated. These two differences are added together and divided by 2 to obtain the average temperature difference value for that pixel position. This process is repeated for all pixel positions (excluding boundary pixels), and the sum of all average temperature difference values ​​is divided by the total number of pixels to obtain the temperature gradient magnitude index of the infrared image. A larger temperature gradient magnitude indicates more drastic temperature changes in the infrared image, clearer edges of temperature anomalies, and richer effective information in the infrared modes.

[0109] The contrast ratio, signal-to-noise ratio, brightness distribution uniformity, and temperature gradient magnitude of infrared images are combined into an infrared quality feature vector with a dimension of 4. Similarly, the contrast ratio, signal-to-noise ratio, and brightness distribution uniformity of visible light images are combined into a visible light quality feature vector with a dimension of 3.

[0110] The infrared quality feature vector Q_I and the visible light quality feature vector Q_V are input into the confidence evaluation network. This network uses the same structure for both infrared and visible light modalities, consisting of a first fully connected layer and a second fully connected layer connected sequentially. The first fully connected layer has an input dimension of 4 (infrared) or 3 (visible light) and 16 output neurons, mapping the quality feature vectors to a higher-dimensional latent space and extracting the cross-features between quality indicators. The output of the first fully connected layer is activated by the ReLU activation function and then input into the second fully connected layer. The second fully connected layer has an input dimension of 16 and 1 output neuron, mapping the latent space features to a single scalar value. The output of the second fully connected layer is mapped to the (0,1) interval by the Sigmoid activation function, outputting the confidence score of this modality in the current scene.

[0111] For the infrared mode, the confidence score C_I is output by the confidence evaluation network; for the visible light mode, the confidence score C_V is output by the same network. To ensure the comparability and mutual exclusivity of the two mode confidence scores, C_I and C_V are normalized so that their sum is 1. The final output confidence score for the infrared mode is C_I / (C_I + C_V), and the confidence score for the visible light mode is C_V / (C_I + C_V).

[0112] The confidence score output by the confidence assessment network reflects the reliability of each modal image under the current acquisition conditions. When the infrared image has high contrast and a large temperature gradient amplitude, the infrared confidence score C_I increases, indicating that the infrared modal information is rich and reliable under the current scene. When the visible light image has high contrast, rich texture, and uniform illumination, the visible light confidence score C_V increases, indicating that the visible light modal information is rich and reliable under the current scene. The two are complementary and their sum after normalization is 1, ensuring dynamic adjustment of the modal contribution ratio under different environmental conditions.

[0113] S5: Based on the confidence score, assign differentiated modal contribution weights to different defect types, and use the modal contribution weights to adaptively weight and fuse the cross-layer enhanced feature map set to obtain a multimodal fusion feature map.

[0114] Furthermore, the cross-layer enhanced feature maps at each scale are subjected to global average pooling to obtain global feature description vectors at each scale. The global feature description vectors at each scale are then concatenated along the channel dimension to form a multi-scale global feature vector.

[0115] The multi-scale global feature vector is input into a defect type-aware gating network, which includes a third fully connected layer, a fourth fully connected layer, and a gated output layer connected in sequence.

[0116] The infrared modal confidence scores and visible light modal confidence scores are multiplied by the infrared contribution weights and visible light contribution weights of each defect category, respectively, and then normalized to generate the modal fusion weight matrix for each defect category.

[0117] The cross-layer enhanced feature map set is weighted and multiplied with the modal fusion weight matrix under each defect category to obtain the category-specific fusion feature map of each defect category. The category-specific fusion feature maps of each defect category are concatenated along the channel dimension, and the number of channels is compressed by the convolutional layer to output the multimodal fusion feature map.

[0118] It should be noted that in this embodiment, the cross-layer enhanced feature map sets {F_C^1, F_C^2, F_C^3, F_C^4, F_C^5} at five scales are subjected to global average pooling respectively. Taking the third-scale feature map F_C^3 (size 64×64×40) as an example, the average pixel value of each channel is calculated in the 64×64 spatial dimension, that is, the arithmetic mean of the 4096 pixel values ​​of each channel is calculated to obtain a global feature description vector of length 40. Similarly, F_C^1 (256×256×16) is obtained as a vector of length 16 after global average pooling, F_C^2 (128×128×24) is obtained as a vector of length 24, F_C^4 (32×32×80) is obtained as a vector of length 80, and F_C^5 (16×16×112) is obtained as a vector of length 112.

[0119] The global feature description vectors at five scales are concatenated sequentially along the channel dimension to form a multi-scale global feature vector with a total dimension of 16+24+40+80+112=272. This vector comprehensively represents the global distribution information of the fused features at each scale. The feature descriptions at shallow scales reflect the overall distribution of texture details, while the feature descriptions at deep scales reflect the overall distribution of temperature structure, providing comprehensive global feature information for subsequent defect type identification.

[0120] A multi-scale global feature vector (272 dimensions) is input into a defect type-aware gating network. This gating network consists of a third fully connected layer, a fourth fully connected layer, and a gated output layer connected in sequence.

[0121] The third fully connected layer has an input dimension of 272 and an output neuron count of 128. It maps multi-scale global feature vectors to a 128-dimensional latent space, extracting correlation and discriminative information between features at different scales. The output of the third fully connected layer is activated by a ReLU activation function and then input into the fourth fully connected layer. The fourth fully connected layer has an input dimension of 128 and an output neuron count of 64, further refining the compact feature representation required for defect type discrimination. The output of the fourth fully connected layer is activated by a ReLU activation function and then input into the gated output layer.

[0122] The gated output layer has an input dimension of 64 and an output dimension of 2K (K is the preset total number of defect categories). In this embodiment, five defect categories are preset: hot spots, diode short circuits, cracks, microcracks, and stains, i.e., K=5, and the output dimension of the gated output layer is 10. The output of the gated output layer is normalized by category using the Softmax activation function to obtain the modal selection gating parameters for each defect category.

[0123] Specifically, the 10 output neurons in the gated output layer are divided into 5 groups, with 2 neurons in each group, corresponding to the infrared contribution weight and visible light contribution weight of a defect category, respectively. After Softmax normalization, the sum of the two weights in each group is 1. For example, after calculation by the gated network, the gate parameters corresponding to hot spot defects are [g_hot spot_I, g_hot spot_V]=[0.78,0.22], indicating that the infrared mode contributes 78% and the visible light mode contributes 22% during hot spot detection; the gate parameters corresponding to crack defects are [g_crack_I, g_crack_V]=[0.19, 0.81], indicating that the infrared mode contributes 19% and the visible light mode contributes 81% during crack detection; the gate parameters corresponding to hidden crack defects are [g_hidden crack_I, g_hidden crack_V]=[0.25, 0.75]; the gate parameters corresponding to stain defects are [g_stain_I, g_stain_V]=[0.20, 0.80]; and the gate parameters corresponding to diode short-circuit defects are [g_diode_I, g_diode_V]=[0.85, 0.15]. The gating parameters for each defect category constitute the modal selection gating parameter set G={g_1, g_2, ..., g_5}.

[0124] The obtained infrared modal confidence scores C_I and visible light modal confidence scores C_V are fused with the modal selection gating parameters of each defect category to generate modal fusion weight matrices for each defect category.

[0125] Taking hot spot defects as an example, their gating parameters are g_hot spot_I=0.78 and g_hot spot_V=0.22. Combined with the confidence scores in the current scene (assuming C_I=0.65 and C_V=0.35), the infrared fusion weight of hot spot defects is calculated to be 0.78×0.65=0.507, and the visible light fusion weight is 0.22×0.35=0.077. After normalization, the infrared fusion weight is 0.507 / (0.507+0.077)≈0.868, and the visible light fusion weight is 0.077 / (0.507+0.077)≈0.132. The final modal fusion weight vector of hot spot defects is [0.868, 0.132].

[0126] Similarly, the modal fusion weight vectors for the other four types of defects are calculated. The fusion weight for crack defects is the normalized result of [C_I × g_crack_I, C_V × g_crack_V]; the same applies to hidden cracks, stains, and diode short circuits. The fusion weight vectors of all five types of defects constitute the modal fusion weight matrix W, with a size of 5×2, where each row corresponds to the infrared weight and visible light weight of one type of defect.

[0127] The set of cross-layer enhanced feature maps at five scales {F_C^1, F_C^2, F_C^3, F_C^4, F_C^5} is weighted and multiplied by the modal fusion weight matrix of the five types of defects to obtain the category-specific fusion feature maps corresponding to each type of defect.

[0128] The specific operation is as follows: For hotspot-type defects, the cross-layer enhanced infrared feature map at each scale is multiplied by the infrared fusion weight of the hotspot type (0.868), and the cross-layer enhanced visible light feature map at each scale is multiplied by the visible light fusion weight of the hotspot type (0.132). Then, the two are added element-wise to obtain the category-specific fusion feature map of the hotspot-type defect at that scale. For the fifth scale (deepest layer), the size of the hotspot-type specific fusion feature map is 16×16×112, obtained by multiplying F_C^5 (infrared) by 0.868 and F_C^5 (visible light) by 0.132, and then adding them together. The other four scales are operated on similarly to obtain the five category-specific fusion feature maps of hotspot-type defects at the five scales.

[0129] The same operation was performed on crack-type, hidden crack-type, stain-type, and diode short-circuit-type defects to obtain category-specific fusion feature maps of each of the five scales.

[0130] The category-specific fusion feature maps corresponding to the five defect types are concatenated along the channel dimension. Taking the fifth scale as an example, each of the five defect types outputs a 16×16×112 feature map, which is concatenated along the channel to obtain a 16×16×560 concatenated feature map. This concatenated feature map is then input into a 1×1 convolutional layer for channel compression, with 112 convolutional kernels and a stride of 1, outputting a multimodal fusion feature map (16×16×112) for the fifth scale.

[0131] The above channel concatenation and 1×1 convolution compression operations were performed on each of the five scales to obtain multimodal fusion feature maps at five scales. These five multimodal fusion feature maps were then upsampled to a uniform size (256×256×112) using bilinear interpolation, concatenated along the channel dimension, and compressed to 112 channels using a 3×3 convolutional layer. The final output is a single multimodal fusion feature map F_fusion with a size of 256×256×112. In this fusion feature map, the feature channels corresponding to different defect categories carry their respective optimal modal contribution allocations, achieving adaptive fusion based on defect type differences.

[0132] S6: Input the multimodal fusion feature map into the defect detection network and output the defect detection result.

[0133] Furthermore, the multimodal fused feature map is input into a defect detection network, which includes a region proposal network and a detection head network;

[0134] The region proposal network slides anchor boxes on the multimodal fused feature map with a preset stride. Each anchor box outputs the corresponding defect existence probability and preliminary position offset through convolutional layers and fully connected layers. Candidate regions are selected based on the defect existence probability, and the positions of the candidate regions are corrected based on the preliminary position offset.

[0135] The feature map of the candidate region is cropped and scaled to a fixed size and input into the detection head network. The detection head network contains a fifth fully connected layer and a sixth fully connected layer connected in sequence. The output of the fifth fully connected layer is activated by the ReLU activation function and then input into the sixth fully connected layer. The sixth fully connected layer outputs the defect category prediction score and the final position offset in parallel.

[0136] Non-maximum suppression is applied to the defect category prediction score and the final position offset, and the defect detection results are output.

[0137] It should be noted that, in this embodiment, the actual defect detection process of a certain type of monocrystalline silicon photovoltaic module is used as an example to fully illustrate the method of the present invention. The detection scenario is a photovoltaic power station in operation under clear weather. A dual-light gimbal device (640×512 pixels infrared thermal imager and 1920×1080 pixels visible light camera) is used to simultaneously acquire module images at a distance of 12 meters. The images include a module with a hot spot defect and a module with a crack defect. The acquired infrared and visible light images are first preprocessed: bilateral filtering for noise reduction, infrared CLAHE enhancement, visible light multi-scale Retinex illumination equalization, phase-correlated subpixel registration, and uniform scaling to 512×512 and normalization. After preprocessing, the hot spot area appears as a bright high-temperature anomaly in the infrared image, and the crack appears as a clear dark linear texture in the visible light image. The preprocessed dual-modal image is input into a lightweight feature extraction network with shared weights. Through five cascaded stages, it downsamples step by step and outputs infrared and visible light feature maps at five scales (resolution from 256×256 to 16×16, number of channels from 16 to 112). The shallow features preserve the details of hot spot edges and crack textures, while the deep features encode the overall temperature field and structural layout of the components.

[0138] The cross-modal semantic alignment stage processes each scale individually. Taking the third scale as an example, the infrared and visible light feature maps are expanded into 4096×40 feature matrices. A similarity weight matrix is ​​calculated to perform global context-weighted modulation on the visible light features, enhancing the visible light features related to the hot spot location. These are then concatenated with the infrared features and fused using a 3×3 convolution to output an aligned fused feature map. Subsequently, bidirectional cross-layer interactive fusion is performed: from top to bottom, deep features are upsampled and fused with shallow features, allowing global hot spot localization information to guide shallow features; from bottom to top, shallow features are downsampled and fused with deep features, allowing crack texture details to supplement deep features. Finally, cross-layer enhanced feature maps for five scales are output. The dynamic modality confidence assessment module calculates the quality indices of the infrared and visible light images respectively. The confidence assessment network outputs confidence scores, which are then normalized. In this embodiment, C_I=0.58 and C_V=0.42, indicating that the infrared modality has a slight advantage in the current scene. In the differentiated gating fusion stage, the cross-layer enhanced feature maps of five scales are concatenated into a 272-dimensional global feature vector after global average pooling. This vector is then input into a defect type-aware gating network (fully connected layers 272→128→64→10, Softmax normalization by category), outputting gating parameters for five defect types: hot spots [0.78, 0.22], cracks [0.19, 0.81], hidden cracks [0.25, 0.75], stains [0.20, 0.80], and diode short circuits [0.85, 0.15]. The confidence scores are fused with the gating parameters to generate modal fusion weights for each type of defect (hot spots [0.84, 0.16], cracks [0.24, 0.76]). The cross-layer enhanced feature maps of each scale are then weighted by category, concatenated, and compressed to output a multimodal fusion feature map (256×256×112). Finally, the input is fed into the defect detection network (region proposal network + detection head network). After anchor box sliding, candidate region filtering, RoI Align scaling, classification and location refinement, and non-maximum suppression, the final detection results are output: 1 hot spot defect (boundary box [210,175,95,88], confidence level 0.94) and 1 crack defect (boundary box [380,220,120,15], confidence level 0.89). These are marked on the visible light image with red and yellow boxes, and a detection report is generated. This embodiment fully verifies the feasibility and effectiveness of the present invention in the detection of multiple types of defects in photovoltaic modules.

[0139] Reference Figure 2 This is a second embodiment of the present invention, which provides a photovoltaic module multimodal feature fusion defect detection system, comprising:

[0140] The image acquisition and preprocessing module is used to acquire infrared and visible light images of the photovoltaic module to be inspected and to perform preprocessing.

[0141] The multi-scale feature extraction module is used to extract multi-scale features from the preprocessed infrared image and the visible light image respectively, to obtain the infrared multi-scale feature map set and the visible light multi-scale feature map set.

[0142] The semantic alignment and cross-layer fusion module is used to perform cross-modal semantic alignment of infrared feature maps and visible light feature maps of the same scale for each scale, to obtain aligned fused feature maps at each scale, and then perform cross-layer interactive fusion of the aligned fused feature maps at each scale to obtain a set of cross-layer enhanced feature maps.

[0143] The confidence assessment module is used to evaluate the confidence scores of the infrared mode and the visible light mode in the current scene based on the modal quality index of infrared and visible light images, respectively.

[0144] The differentiated gated fusion module is used to assign differentiated modal contribution weights to different defect types based on the confidence score, and to perform adaptive weighted fusion of the cross-layer enhanced feature map set using the modal contribution weights to obtain a multimodal fusion feature map;

[0145] The defect detection and output module is used to input the multimodal fusion feature map into the defect detection network and output the defect detection result.

[0146] This embodiment also provides a computer device applicable to a method for detecting defects in the fusion of multimodal features of photovoltaic modules, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method for detecting defects in the fusion of multimodal features of photovoltaic modules as proposed in the above embodiment.

[0147] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0148] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements a method for detecting defects in the fusion of multimodal features of photovoltaic modules as proposed in the above embodiments.

[0149] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for detecting defects in photovoltaic modules by fusing multimodal features, characterized in that, include: Acquire infrared and visible light images of the photovoltaic module to be tested, and perform preprocessing; Multi-scale feature extraction was performed on the preprocessed infrared and visible light images to obtain sets of infrared multi-scale feature maps and sets of visible light multi-scale feature maps. For each scale, cross-modal semantic alignment is performed on the infrared feature map and visible light feature map of the same scale to obtain the aligned fused feature map at each scale. Then, the aligned fused feature maps at each scale are cross-layer interactively fused to obtain a cross-layer enhanced feature map set. Based on the modal quality metrics of infrared and visible light images, the confidence scores of infrared and visible light modes in the current scene are evaluated respectively. Based on the confidence score, differentiated modal contribution weights are assigned to different defect types, and the modal contribution weights are used to adaptively weight and fuse the cross-layer enhanced feature map set to obtain a multimodal fusion feature map. The multimodal fusion feature map is input into the defect detection network, and the defect detection result is output.

2. The method for detecting defects in photovoltaic modules by fusing multimodal features as described in claim 1, characterized in that, The acquisition and preprocessing of infrared and visible light images of the photovoltaic module to be tested includes: Edge-preserving filtering is used to denoise infrared and visible light images respectively, and the filter kernel parameters are adaptively set according to the noise level of each modality image. The contrast of the denoised infrared image is enhanced by contrast-limited adaptive histogram equalization, and the illumination equalization of the denoised visible light image is processed by multi-scale Retinex algorithm. Using the visible light image as a reference, the phase correlation method is used to calculate the subpixel registration parameters and the infrared image is geometrically transformed to achieve subpixel-level spatial alignment between the two modal images, and then they are uniformly scaled to the same target size. The pixel values ​​of the two modal images after registration and size unification are normalized to a preset range.

3. The method for detecting defects in multimodal feature fusion of photovoltaic modules as described in claim 1, characterized in that, The step involves performing multi-scale feature extraction on the preprocessed infrared and visible light images respectively to obtain infrared multi-scale feature map sets and visible light multi-scale feature map sets, including: The preprocessed infrared and visible light images are respectively input into a feature extraction network with the same structure and shared weights. The feature extraction network contains multiple cascaded feature extraction modules. The output of the current module serves as the input of the next module. Each feature extraction module consists of a convolutional layer, a batch normalization layer, and an activation function layer in sequence. The end of each module is downsampled by a convolution with a stride of 2. After downsampling at each level, infrared and visible light feature maps of the corresponding scales are output respectively. The shallow feature map has a higher resolution than the deep feature map, and the receptive field of the deep feature map is larger than that of the shallow feature map.

4. The method for detecting defects in multimodal feature fusion of photovoltaic modules as described in claim 1, characterized in that, The process involves performing cross-modal semantic alignment on infrared and visible light feature maps of the same scale to obtain aligned and fused feature maps at various scales. Then, the aligned and fused feature maps at various scales are subjected to cross-layer interactive fusion to obtain a set of cross-layer enhanced feature maps, including: Infrared and visible light feature maps of the same scale are unfolded into two-dimensional feature matrices along the channel dimension. The matrix product between the transpose of the infrared feature matrix and the visible light feature matrix is ​​calculated. After Softmax normalization, the cross-modal similarity weight matrix at this scale is obtained. The cross-modal similarity weight matrix is ​​multiplied again with the visible light feature matrix to perform weighted modulation of the visible light features. The modulated visible light features and infrared features are then concatenated along the channel dimension and fused through a convolutional layer to output the aligned and fused feature map at this scale. The aligned and fused feature maps at each scale are input into the cross-layer interactive fusion module. The cross-layer interactive fusion module performs bidirectional cross-layer connections on the aligned and fused feature maps at each scale: the deep aligned and fused feature map is upsampled, and the upsampled deep features are concatenated with the adjacent shallow features along the channel dimension and then fused through a convolutional layer; the shallow aligned and fused feature map is downsampled by a convolution with a stride of 2, and the downsampled shallow features are concatenated with the adjacent deep features along the channel dimension and then fused through a convolutional layer; after bidirectional cross-layer connections, the cross-layer enhanced feature maps at each scale are output.

5. The method for detecting defects in multimodal feature fusion of photovoltaic modules as described in claim 1, characterized in that, The modal quality metrics based on infrared and visible light images evaluate the confidence scores of the infrared and visible light modes in the current scene, respectively, including: Modal quality indices are calculated for the preprocessed infrared and visible light images, respectively. The modal quality indices include: the standard deviation of global pixel values ​​as a contrast index; the ratio of the mean local gradient magnitude to the mean global gradient magnitude as a signal-to-noise ratio index; the kurtosis coefficient of the histogram as a brightness distribution uniformity index; and the mean absolute value of the temperature difference between adjacent pixels in the infrared image as a temperature gradient magnitude index. The contrast ratio, signal-to-noise ratio, brightness distribution uniformity, and temperature gradient amplitude of infrared images are combined into an infrared quality feature vector, and the contrast ratio, signal-to-noise ratio, and brightness distribution uniformity of visible light images are combined into a visible light quality feature vector. The infrared quality feature vector and the visible light quality feature vector are respectively input into the confidence evaluation network, which includes a first fully connected layer and a second fully connected layer connected in sequence.

6. The method for detecting defects in multimodal feature fusion of photovoltaic modules as described in claim 1, characterized in that, The method of assigning differentiated modal contribution weights to different defect types and using the modal contribution weights to adaptively weighted fuse the cross-layer enhanced feature map set to obtain a multimodal fused feature map includes: performing global average pooling on the cross-layer enhanced feature maps at each scale to obtain global feature description vectors at each scale, and concatenating the global feature description vectors at each scale along the channel dimension to form a multi-scale global feature vector. The multi-scale global feature vector is input into a defect type-aware gating network, which includes a third fully connected layer, a fourth fully connected layer, and a gated output layer connected in sequence. The infrared modal confidence scores and visible light modal confidence scores are multiplied by the infrared contribution weights and visible light contribution weights of each defect category, respectively, and then normalized to generate the modal fusion weight matrix for each defect category. The cross-layer enhanced feature map set is weighted and multiplied with the modal fusion weight matrix under each defect category to obtain the category-specific fusion feature map of each defect category. The category-specific fusion feature maps of each defect category are concatenated along the channel dimension, and the number of channels is compressed by the convolutional layer to output the multimodal fusion feature map.

7. The method for detecting defects in multimodal feature fusion of photovoltaic modules as described in claim 1, characterized in that, The step of inputting the multimodal fused feature map into the defect detection network and outputting the defect detection result includes: The multimodal fused feature map is input into a defect detection network, which includes a region proposal network and a detection head network. The region proposal network slides anchor boxes on the multimodal fused feature map with a preset stride. Each anchor box outputs the corresponding defect existence probability and preliminary position offset through convolutional layers and fully connected layers. Candidate regions are selected based on the defect existence probability, and the positions of the candidate regions are corrected based on the preliminary position offset. The feature map of the candidate region is cropped and scaled to a fixed size and input into the detection head network. The detection head network contains a fifth fully connected layer and a sixth fully connected layer connected in sequence. The output of the fifth fully connected layer is activated by the ReLU activation function and then input into the sixth fully connected layer. The sixth fully connected layer outputs the defect category prediction score and the final position offset in parallel. Non-maximum suppression is applied to the defect category prediction score and the final position offset, and the defect detection results are output.

8. A multi-modal feature fusion defect detection system for photovoltaic modules, characterized in that, include: The image acquisition and preprocessing module is used to acquire infrared and visible light images of the photovoltaic module to be inspected and to perform preprocessing. The multi-scale feature extraction module is used to extract multi-scale features from the preprocessed infrared image and the visible light image respectively, to obtain the infrared multi-scale feature map set and the visible light multi-scale feature map set. The semantic alignment and cross-layer fusion module is used to perform cross-modal semantic alignment of infrared feature maps and visible light feature maps of the same scale for each scale, to obtain aligned fused feature maps at each scale, and then perform cross-layer interactive fusion of the aligned fused feature maps at each scale to obtain a set of cross-layer enhanced feature maps. The confidence assessment module is used to evaluate the confidence scores of the infrared mode and the visible light mode in the current scene based on the modal quality index of infrared and visible light images, respectively. The differentiated gated fusion module is used to assign differentiated modal contribution weights to different defect types based on the confidence score, and to perform adaptive weighted fusion of the cross-layer enhanced feature map set using the modal contribution weights to obtain a multimodal fusion feature map; The defect detection and output module is used to input the multimodal fusion feature map into the defect detection network and output the defect detection result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for detecting defects in multimodal feature fusion of photovoltaic modules as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting defects in multimodal feature fusion of photovoltaic modules as described in any one of claims 1 to 7.

Images