A method, apparatus, and equipment for photovoltaic panel defect identification based on multispectral fusion.
The photovoltaic panel defect identification method based on multispectral fusion, which utilizes a defect identification model based on a multispectral attention module and a cross-modal global attention fusion module, solves the problems of low efficiency and insufficient accuracy in photovoltaic panel defect identification and achieves high-precision defect identification.
Patent Information
- Application Number
- CN202611020446.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-09
- Publication Date
- 2026-11-17
AI Technical Summary
In existing photovoltaic power plants, the efficiency of photovoltaic panel defect identification is low. Single visible light detection is severely affected by glass reflection, and single infrared detection has low resolution. Multimodal fusion schemes have failed to effectively solve the semantic alignment problem between visible light texture and infrared temperature field, resulting in a high false negative rate for target detection models.
A photovoltaic panel defect identification method based on multispectral fusion is adopted. By acquiring visible light images and infrared thermal imaging images, an input vector is constructed. A defect identification model based on multispectral attention module and cross-modal global attention fusion module is used to extract and fuse features. Combined with temperature anomaly scoring branch and joint loss function, feature alignment and high-precision defect identification are achieved.
It significantly improves the accuracy of defect identification in scenarios with strong reflection and no appearance changes, enhances the accuracy of identifying microcracks and micro-hot spots with extremely small pixel ratios, reduces the false detection rate, and achieves high-precision identification of defects such as hot spots, microcracks, and stains on photovoltaic panels.
Smart Images

Figure CN123118445A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and new energy operation and maintenance technology, and in particular to a method, device and equipment for identifying defects in photovoltaic panels based on multispectral fusion. Background Technology
[0002] After the large-scale deployment of photovoltaic power plants, the identification of photovoltaic panel defects relies on manual inspection, which is inefficient and struggles to detect minute microcracks or early hot spots. Existing single visible light detection is severely affected by glass reflections, while single infrared detection lacks texture details and has low resolution. Although multimodal fusion schemes exist, current technologies often simply stitch together features during the feature fusion stage, failing to effectively address the semantic alignment problem between visible light textures and infrared temperature fields. Furthermore, existing target detection models have a high false negative rate when dealing with minute defects. In addition, existing target detection models lack targeted optimization of their loss functions and attention mechanisms when handling dual-modal inputs and specific photovoltaic defect distributions. Summary of the Invention
[0003] In view of this, the present invention provides a method, device and equipment for identifying defects in photovoltaic panels based on multispectral fusion, which realizes high-precision identification of defects such as hot spots, microcracks and stains in photovoltaic panels.
[0004] According to one aspect of the present invention, a method for identifying defects in photovoltaic panels based on multispectral fusion is provided, the method comprising: Acquire visible light and infrared thermal images of the photovoltaic panel to be identified, and construct an input vector based on the visible light and infrared thermal images; The input vector is input into the trained defect recognition model. Based on the backbone network of the defect recognition model, features are extracted from the input vector to obtain features at different scales. Based on the neck network of the defect recognition model, features at different scales are fused to obtain multiple fused features. Based on the decoupled detection head of the defect recognition model, the fused features are predicted to obtain the target prediction result. The backbone network is constructed based on a multispectral attention module, and the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module. The decoupled detection head includes a temperature anomaly scoring branch. The defect recognition model is obtained by training an initial defect recognition model based on a joint loss function. The target prediction results are post-processed to determine the photovoltaic panel defect identification results.
[0005] According to another aspect of the present invention, a photovoltaic panel defect identification device based on multispectral fusion is provided, the device comprising: The image acquisition and processing module is used to acquire visible light images and infrared thermal imaging images of the photovoltaic panel to be identified, and to construct an input vector based on the visible light images and the infrared thermal imaging images; The prediction module is used to input the input vector into the trained defect recognition model, extract features from the input vector according to the backbone network of the defect recognition model to obtain features at different scales, fuse the features at different scales according to the neck network of the defect recognition model to obtain multiple fused features, and predict the fused features according to the decoupled detection head of the defect recognition model to obtain the target prediction result. The backbone network is constructed based on a multispectral attention module, the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module, the decoupled detection head includes a temperature anomaly scoring branch, and the defect recognition model is obtained by training an initial defect recognition model based on a joint loss function. The determination module is used to post-process the target prediction results to determine the photovoltaic panel defect identification results.
[0006] According to another aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method for identifying photovoltaic panel defects based on multispectral fusion.
[0007] According to another aspect of the present invention, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described method for identifying photovoltaic panel defects based on multispectral fusion.
[0008] By employing the above technical solution, this invention provides a method, apparatus, and device for photovoltaic panel defect identification based on multispectral fusion. The technical solution of this invention first acquires visible light and infrared thermal images of the photovoltaic panel to be identified. Based on the visible light and infrared thermal images, an input vector is constructed. Then, the input vector is input into a trained defect identification model. Based on the backbone network of the defect identification model, features are extracted from the input vector to obtain features at different scales. Based on the neck network of the defect identification model, features at different scales are fused to obtain multiple fused features. Based on the decoupled detection head of the defect identification model, the fused features are predicted to obtain the target prediction result. The backbone network is constructed based on a multispectral attention module, and the neck network is based on a cross-modal... The decoupled detection head, constructed using a global attention fusion module and a multispectral attention module, includes a temperature anomaly scoring branch. The defect recognition model is trained based on a joint loss function. Finally, the target prediction result is post-processed to determine the photovoltaic panel defect recognition result. Through the multispectral attention module and the cross-modal global attention fusion module, deep semantic alignment of visible light features and infrared features is achieved, significantly improving the defect recognition accuracy in scenarios with strong reflection and no appearance changes. The joint loss function improves the accuracy of the defect recognition model in recognizing microcracks and micro-hot spots with extremely small pixel proportions. The temperature anomaly scoring branch effectively distinguishes between real hot spots and pseudo-high temperature areas caused by environmental reflection, reducing the false detection rate and achieving high-precision recognition of defects such as hot spots, microcracks, and stains on photovoltaic panels.
[0009] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0010] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The diagram illustrates a flowchart of a photovoltaic panel defect identification method based on multispectral fusion provided by an embodiment of the present invention. Figure 2 A flowchart illustrating another photovoltaic panel defect identification method based on multispectral fusion provided by an embodiment of the present invention is shown. Figure 3A schematic diagram of a photovoltaic panel defect identification device based on multispectral fusion provided in an embodiment of the present invention is shown. Figure 4 A schematic diagram of another photovoltaic panel defect identification device based on multispectral fusion provided in an embodiment of the present invention is shown; Figure 5 This diagram illustrates the structure of an initial defect identification model provided in an embodiment of the present invention. Figure 6 This diagram illustrates the structure of a multispectral attention module according to an embodiment of the present invention. Figure 7 The diagram shows a structural schematic of a cross-modal global attention fusion module provided in an embodiment of the present invention. Detailed Implementation
[0011] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0012] This embodiment provides a method for identifying defects in photovoltaic panels based on multispectral fusion, such as... Figure 1 As shown, the method includes: 101. Obtain the visible light image and infrared thermal image of the photovoltaic panel to be identified, and construct an input vector based on the visible light image and the infrared thermal image.
[0013] In this embodiment, the visible light image and the infrared thermal image are acquired simultaneously. However, due to the different optical center positions of the two cameras and their different imaging principles, denoising preprocessing is required for each camera. The denoised visible light image is then processed separately. Compared with the denoised preprocessed infrared thermal image Cross-modal pixel-level registration is performed. Specifically, as one implementation method, the denoised preprocessed infrared thermal imaging image is... Transform to the visible light image coordinate system to obtain the registered infrared thermal image. In the channel dimension, the denoised preprocessed visible light image Registered infrared thermal imaging images Concatenate the vectors to obtain the input vector. .
[0014]
[0015] 102. Input the input vector into the trained defect recognition model. Based on the backbone network of the defect recognition model, extract features from the input vector to obtain features at different scales. Based on the neck network of the defect recognition model, fuse the features at different scales to obtain multiple fused features. Based on the decoupled detection head of the defect recognition model, predict the fused features to obtain the target prediction result. The backbone network is constructed based on a multispectral attention module, the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module, the decoupled detection head includes a temperature anomaly scoring branch, and the defect recognition model is obtained by training an initial defect recognition model based on a joint loss function.
[0016] In this embodiment, the deep semantic alignment of visible light features and infrared features is achieved through the multispectral attention module and the cross-modal global attention fusion module, which significantly improves the accuracy of defect identification in strong reflective and unchanging scenes. Through the joint loss function, the defect identification model improves the accuracy of identifying hidden cracks and micro hot spots with extremely small pixel proportions. Through the temperature anomaly scoring branch, the model effectively distinguishes between real hot spots and pseudo-high temperature areas caused by environmental reflection, thus reducing the false detection rate.
[0017] 103. Post-process the target prediction results to determine the photovoltaic panel defect identification results.
[0018] In this embodiment, the coordinates of all screened target predicted defect boxes, the photovoltaic panel defect categories of all screened target predicted defect boxes, and the temperature anomaly confidence scores of all screened target predicted defect boxes are used as the photovoltaic panel defect identification results.
[0019] This invention provides a method, apparatus, and device for photovoltaic panel defect identification based on multispectral fusion. The technical solution of this invention first acquires visible light and infrared thermal images of the photovoltaic panel to be identified. An input vector is constructed based on the visible light and infrared thermal images. Then, the input vector is input into a trained defect identification model. Features are extracted from the input vector using the backbone network of the defect identification model to obtain features at different scales. These features are then fused using the neck network of the defect identification model to obtain multiple fused features. Finally, the fused features are predicted using the decoupled detection head of the defect identification model to obtain a target prediction result. The backbone network is constructed based on a multispectral attention module, and the neck network is based on cross-modal global attention. The decoupled detection head, constructed using a force fusion module and a multispectral attention module, includes a temperature anomaly scoring branch. The defect recognition model is trained based on a joint loss function. Finally, the target prediction result is post-processed to determine the photovoltaic panel defect recognition result. Through the multispectral attention module and the cross-modal global attention fusion module, deep semantic alignment of visible light features and infrared features is achieved, significantly improving the defect recognition accuracy in scenarios with strong reflection and no appearance changes. The joint loss function improves the recognition accuracy of the defect recognition model for microcracks and micro-hot spots with extremely small pixel proportions. The temperature anomaly scoring branch effectively distinguishes between real hot spots and pseudo-high temperature areas caused by environmental reflection, reducing the false detection rate and achieving high-precision recognition of defects such as hot spots, microcracks, and stains on photovoltaic panels.
[0020] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and to fully illustrate the specific implementation process in this embodiment, another photovoltaic panel defect identification method based on multispectral fusion is provided, such as... Figure 2 As shown, the method includes: 201. Obtain the visible light image and infrared thermal image of the photovoltaic panel to be identified, and construct an input vector based on the visible light image and the infrared thermal image.
[0021] In this embodiment, constructing the input vector based on the visible light image and the infrared thermal imaging image includes: performing corresponding denoising preprocessing on the visible light image and the infrared thermal imaging image respectively; performing cross-modal pixel-level registration on the denoised visible light image and the denoised infrared thermal imaging image; and stitching them together in the channel dimension after registration to obtain the input vector.
[0022] Specifically, the visible light image undergoes corresponding denoising preprocessing, namely, using a polarization-based de-reflection algorithm to eliminate highlights on the glass surface, resulting in a denoised preprocessed visible light image. For infrared thermal imaging images, corresponding denoising preprocessing is performed, namely, non-uniformity correction is applied to the infrared thermal imaging images to eliminate fixed pattern noise, and grayscale values are mapped to absolute temperature values to obtain denoised preprocessed infrared thermal imaging images. .
[0023] The Scale-Invariant Feature Transform (SIFT) algorithm is used to extract the denoised preprocessed visible light image. Visible light feature points are extracted from the denoised preprocessed infrared thermal image. For infrared feature points, a Fast Library for Approximate Nearest Neighbors (FLANN) is used to perform preliminary matching between visible and infrared feature points to obtain preliminary matching results. Then, the Random Sample Consensus (RANSAC) algorithm is used to remove mismatches from the preliminary matching results to obtain correct matching results. The correct matching results include multiple pairs of visible and infrared feature points, and each pair of visible and infrared feature points is correctly matched. Based on the correct matching results, the homography matrix H is calculated.
[0024] Infrared thermal imaging image after noise reduction preprocessing The coordinates are Visible light image after denoising preprocessing The coordinates are The transformation relationship is as follows:
[0025] in, The equality relationship in homogeneous coordinates is represented by the homography matrix. The infrared thermal imaging image after noise reduction preprocessing Transform to the visible light image coordinate system to obtain the registered infrared thermal image. In the channel dimension, the denoised preprocessed visible light image Registered infrared thermal imaging images Concatenate the vectors to obtain the input vector. .
[0026]
[0027] And the visible light image after noise reduction preprocessing It is a 3-channel, registered infrared thermal imaging image. It is a 1-channel input vector obtained after concatenation. It is 4-channel. For the height of the image, The width of the image.
[0028] 202. Construct an initial defect identification model.
[0029] The initial defect identification model includes a backbone network, a neck network, and a decoupled detection head. The backbone network is constructed based on a multispectral attention module. The decoupled detection head includes a temperature anomaly scoring branch. The neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module.
[0030] For this embodiment, as Figure 5 The diagram shown is a schematic of an initial defect identification model. It should be noted that the backbone network in this embodiment is built based on the multispectral attention module C3k2-PV, unlike the backbone network of YOLOv11. Specifically, the C3k2 module in YOLOv11 is replaced with the multispectral attention module C3k2-PV, as shown below. Figure 6 The diagram shown is a schematic of a multispectral attention module. The neck network in this embodiment is constructed based on the cross-modal global attention fusion module CGAF and the multispectral attention module C3k2-PV, unlike the neck network of YOLOv11. Figure 7 The diagram shown is a structural schematic of a cross-modal global attention fusion module. The decoupled detection head in this embodiment includes three detection layers, each containing three branches. One of these branches is a temperature anomaly scoring branch, unlike the detection head of YOLOv11.
[0031] 203. Obtain input vector samples, input the input vector samples into the initial defect identification model, output the initial prediction result, determine the prediction result based on the initial prediction result, calculate the value of the joint loss function based on the prediction result and the true result, adjust the model parameters of the initial defect identification model based on the value of the joint loss function until the preset stopping condition is reached, and obtain the trained defect identification model.
[0032] It should be noted that, Figure 5 This is also a structural diagram of a defect identification model, that is, the structure of the initial defect identification model is the same as the structure of the defect identification model.
[0033] The process of obtaining the input vector sample is the same as that of obtaining the input vector in step 201 of the embodiment. The input vector sample is obtained by stitching together the denoised and preprocessed visible light image sample and the registered infrared thermal imaging image sample, which will not be described in detail here.
[0034] The process of inputting the input vector sample into the initial defect recognition model and outputting the initial prediction result is the same as the process of inputting the input vector into the trained defect recognition model in step 204 of the embodiment to obtain the target prediction result, and will not be repeated here. The initial prediction result includes: the category prediction probability of each first-scale predicted defect box output by the first classification branch. The category prediction probability refers to... The probability of each category, for example The categories are: hot spots, microcracks, cracks, stains, obstructions, solder joint detachment, and background. The first regression branch outputs the coordinate displacement of each predicted defect box at the first scale. The first temperature anomaly scoring branch outputs the infrared predicted temperature and temperature anomaly confidence score of each predicted defect box at the first scale. The second classification branch outputs the category prediction probability of each predicted defect box at the second scale. The category prediction probability refers to... The probability of each category, for example The categories are: hot spots, microcracks, cracks, stains, obstructions, solder joint detachment, and background. The second regression branch outputs the coordinate displacement of the predicted defect box at each second scale. The second temperature anomaly scoring branch outputs the infrared predicted temperature and temperature anomaly confidence score of the predicted defect box at each second scale. The third classification branch outputs the category prediction probability of the predicted defect box at each third scale. The category prediction probability refers to... The probability of each category, for example The categories are: hot spots, microcracks, cracks, stains, obstructions, solder joint detachment, and background. The coordinate displacement of each predicted defect box at the third scale output by the third regression branch, and the infrared predicted temperature and temperature anomaly confidence score of each predicted defect box at the third scale output by the third temperature anomaly scoring branch, are used to summarize all predicted defect boxes at the first, second, and third scales. The predicted defect box set includes all predicted defect boxes. It should be noted that the temperature anomaly confidence score refers to the degree of certainty that the predicted defect box is thermally abnormal. The closer the temperature anomaly confidence score is to 1, the more certain the predicted defect box is thermally abnormal; the closer the temperature anomaly confidence score is to 0, the more certain the predicted defect box is not thermally abnormal.
[0035] Based on the initial prediction results, the prediction results are determined, including: calculating the coordinates of the predicted defect boxes based on the coordinate displacement of the predicted defect boxes. The prediction results include: the coordinates of the predicted defect boxes, the category prediction probability of each predicted defect box, and the infrared predicted temperature of each predicted defect box.
[0036] In this embodiment, the prediction result includes: the coordinates of the predicted defect box, the predicted category probability, and the predicted infrared temperature; the true result includes: the coordinates of the true defect box, the true category, and the true infrared temperature. Calculating the value of the joint loss function based on the prediction result and the true result includes: calculating the bounding box loss based on the coordinates of the predicted defect box and the coordinates of the true defect box; calculating the classification loss based on the predicted category probability and the true category; calculating the temperature anomaly monitoring loss based on the predicted infrared temperature and the true infrared temperature; and calculating the value of the joint loss function based on the bounding box loss, the classification loss, and the temperature anomaly monitoring loss.
[0037] The actual result corresponds to the input vector sample. The bounding box loss is calculated based on the coordinates of the predicted defect box and the coordinates of the actual defect box. Specifically: Replace the traditional CIoU Loss with MPDIoU Loss:
[0038] in, For bounding box loss, To predict the coordinates of the defect box, The coordinates of the actual defect bounding box. To predict the intersection-union ratio (IoU) of the coordinates of the defect bounding box and the coordinates of the actual defect bounding box, To predict the Euclidean distance between the top-left corner coordinates of the defect bounding box and the top-left corner coordinates of the actual defect bounding box, To predict the Euclidean distance between the coordinates of the bottom right corner of the defect bounding box and the coordinates of the bottom right corner of the actual defect bounding box, For the height of the image, The width of the image.
[0039] Focal Loss is used to address the imbalance between positive and negative samples.
[0040] in, For classifying losses, For each predicted probability, calculate the probability corresponding to the true class. This is the category balance coefficient. For focusing parameters.
[0041] It should be noted that the category prediction probability is a set of probabilities, which includes multiple probabilities, each corresponding to a category. The probability corresponding to the true category is selected from this set of probabilities and used as the prediction probability. .
[0042] The mean square error is used to calculate the temperature anomaly monitoring loss based on the infrared predicted temperature and the infrared true temperature.
[0043] in, The temperature anomaly monitoring loss is represented by , where i is the ordinal number of the coordinates of the actual defect bounding box. This represents the total number of coordinates of the actual defect bounding box. Let the infrared predicted temperature be the coordinate of the i-th true defect box. The true infrared temperature is the coordinate of the i-th real defect box.
[0044] To adapt to the imbalanced nature of photovoltaic defect samples and the large number of small targets, a joint loss function is adopted:
[0045] in, For the joint loss function, , , These correspond to the first, second, and third equilibrium hyperparameters, respectively, and can be determined through grid search, which will not be elaborated here.
[0046] As one implementation method, the preset stopping condition is that the value of the joint loss function is less than or equal to a preset loss threshold.
[0047] 204. Input the input vector into the trained defect recognition model, extract features from the input vector according to the backbone network of the defect recognition model to obtain features at different scales, fuse the features at different scales according to the neck network of the defect recognition model to obtain multiple fused features, and predict the fused features according to the decoupled detection head of the defect recognition model to obtain the target prediction result.
[0048] The backbone network is constructed based on a multispectral attention module, the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module, and the defect recognition model is obtained by training an initial defect recognition model based on a joint loss function.
[0049] like Figure 5In this embodiment, the step of extracting features from the input vector based on the backbone network of the defect recognition model to obtain features at different scales includes: extracting features from the input vector based on the first convolutional module, second convolutional module, first multispectral attention module, third convolutional module, second multispectral attention module, fourth convolutional module, third multispectral attention module, fifth convolutional module, fourth multispectral attention module, fast spatial pyramid pooling module, and cross-stage local spatial attention module connected sequentially in the backbone network of the defect recognition model; the second multispectral attention module outputs a first-scale feature, the third multispectral attention module outputs a second-scale feature, and the cross-stage local spatial attention module outputs a third-scale feature; and the first-scale feature, the second-scale feature, and the third-scale feature are used as features at different scales. The process of outputting the first-scale feature by the second multispectral attention module is as follows: determine the features output by the third convolution module, input them into the second multispectral attention module, extract the visible light features and infrared features therein, generate a spatial attention map based on the multispectral interaction branch of the second multispectral attention module according to the visible light features and the infrared features, calculate the temperature change edge features based on the temperature gradient enhancement branch of the second multispectral attention module according to the infrared features, and calculate the first-scale feature according to the spatial attention map, the features output by the third convolution module and the temperature change edge features.
[0050] It should be noted that the process of the third multispectral attention module outputting the second-scale feature and the cross-stage local spatial attention module outputting the third-scale feature is the same as the process of the second multispectral attention module outputting the first-scale feature, and will not be repeated here. Figure 6 Extract visible light features from the features output by the third convolutional module. With infrared features For visible light characteristics The process of generating a spatial attention map based on the visible light features and the infrared features is as follows:
[0051] in, This is a spatial attention map. Corresponding to average pooling layer, Corresponding to the max pooling layer, Corresponding to the first connection layer, Corresponding to the first convolutional layer, This corresponds to the first activation function layer.
[0052] By enhancing the temperature gradient branch, the edge response of defects such as hot spots and microcracks is improved. The process of calculating the edge features of temperature abrupt changes based on the infrared characteristics is as follows: The infrared features are passed through horizontal and vertical Sobel convolutional layers respectively. Horizontal and vertical Sobel convolutions are performed accordingly to obtain the left-right and top-bottom temperature differences. The left-right temperature difference is passed through the first absolute value layer, i.e., the absolute value is taken to obtain the horizontal response map. The top-bottom temperature difference is passed through the second absolute value layer, i.e., the absolute value is taken to obtain the vertical response map. The horizontal response map and the vertical response map are added element by element to obtain the temperature change edge features.
[0053] The process of calculating the first-scale features based on the spatial attention map, the features output by the third convolutional module, and the temperature abrupt change edge features is as follows:
[0054] in, For element-wise multiplication, there is a corresponding element-wise multiplication layer. The features output by the third convolutional module, Corresponding to the second connection layer, Corresponding to the second convolutional layer, This is the first-scale feature.
[0055] like Figure 5 The neck network of the defect identification model includes: a first cross-modal global attention fusion module, a second cross-modal global attention fusion module, a third cross-modal global attention fusion module, a fourth cross-modal global attention fusion module, a first upsampling module, a second upsampling module, a fifth multispectral attention module, a sixth multispectral attention module, a seventh multispectral attention module, an eighth multispectral attention module, a sixth convolution module, and a seventh convolution module; The method involves fusing features at different scales based on the neck network of the defect identification model to obtain multiple fused features. This includes: inputting the first-scale feature into the first cross-modal global attention fusion module; inputting the second-scale feature into the second cross-modal global attention fusion module; inputting the third-scale feature into the second upsampling module to output a first upsampled feature; inputting the first upsampled feature into the second cross-modal global attention fusion module; fusing the second-scale feature with the first upsampled feature to output a first fused feature; inputting the first fused feature into the fifth multispectral attention module to obtain a fourth-scale feature; inputting the fourth-scale feature into the first upsampling module to output a second upsampled feature; inputting the second upsampled feature into the first cross-modal global attention fusion module; fusing the second upsampled feature with the first-scale feature to output a second fused feature; and inputting the third-scale feature into the second upsampled feature into the second upsampled feature. The second fused feature is input to the sixth multispectral attention module, which outputs the first fused feature. The first fused feature is input to the sixth convolution module, which outputs the first extracted feature. The first extracted feature is input to the third cross-modal global attention fusion module, which performs feature fusion based on the first extracted feature and the fourth scale feature, and outputs the third fused feature. The third fused feature is input to the seventh multispectral attention module, which outputs the second fused feature. The second fused feature is input to the seventh convolution module, which outputs the second extracted feature. The second extracted feature is input to the fourth cross-modal global attention fusion module, which performs feature fusion based on the second extracted feature and the third scale feature, and outputs the fourth fused feature. The fourth fused feature is input to the eighth multispectral attention module, which outputs the third fused feature. The first fused feature, the second fused feature, and the third fused feature are used as multiple fused features. like Figure 7 The process of inputting the first upsampled feature into the second cross-modal global attention fusion module, and performing feature fusion based on the second scale feature and the first upsampled feature to output the first fused feature is as follows: inputting the first upsampled feature into the third convolutional layer to output the first projected feature; inputting the second scale feature into the fourth convolutional layer to output the second projected feature; concatenating the first projected feature and the second projected feature to obtain the concatenated feature; based on the sequentially connected global average pooling layer, the first fully connected layer, the second activation function layer, the second fully connected layer, and the third activation function layer, generating the first channel weight corresponding to the first projected feature and the second channel weight corresponding to the second projected feature according to the concatenated feature; inputting the first projected feature, the second projected feature, the first channel weight, and the second channel weight into the weighted summation layer to output the first fused feature.
[0056] It should be noted that the processes for outputting the second, third, and fourth fusion features are the same as those for outputting the first fusion feature, and will not be repeated here.
[0057] like Figure 7 Specifically, the first upsampled feature is input into the third convolutional layer to output the first projected feature, and the second scaled feature is input into the fourth convolutional layer to output the second projected feature. Input to the third convolutional layer, and input the second-scale features The input is a fourth convolutional layer, so that both are projected to the same dimension through a 1×1 convolution. For the globally average pooling layer, the first fully connected layer, the second activation function layer, the second fully connected layer, and the third activation function layer connected in sequence, the first channel weights corresponding to the first projected feature and the second channel weights corresponding to the second projected feature are generated according to the concatenated features. Specifically, the concatenated features are input into the global average pooling layer to obtain the global descriptor corresponding to the first projected feature. And the global descriptor corresponding to the features after the second projection. The global descriptor corresponding to the first projected feature Global descriptor corresponding to the second projected features The concatenated global descriptor is obtained and input into the first fully connected layer. Based on the sequentially connected first fully connected layer, second activation function layer, and third activation function layer, the first channel weights corresponding to the first projected features and the second channel weights corresponding to the second projected features are generated. Specifically:
[0058] in, This is the concatenated global descriptor. Corresponding to the first fully connected layer, Corresponding to the second activation function layer, Corresponding to the second fully connected layer, Corresponding to the third activation function layer, The first channel weights are the features corresponding to the first projected features. The weights of the second channel are the features after the second projection.
[0059] The first projected feature, the second projected feature, the first channel weight, and the second channel weight are input into a weighted summation layer, and the first fused feature is output. Specifically:
[0060] in, This is the first upsampling feature. This is a second-scale feature. As the weight of the first channel, This is the weight for the second channel.
[0061] By using a cross-modal global attention fusion module, optimal multispectral characterization is ensured for both small defects (shallow features) and large hot spots (deep features) in the top-down and bottom-up paths of the neck network.
[0062] like Figure 5 The decoupled detection head of the defect recognition model includes a first detection layer, a second detection layer, and a third detection layer. The first detection layer includes a first classification branch, a first regression branch, and a first temperature anomaly scoring branch. The second detection layer includes a second classification branch, a second regression branch, and a second temperature anomaly scoring branch. The third detection layer includes a third classification branch, a third regression branch, and a third temperature anomaly scoring branch. The step of predicting the fused features based on the decoupled detection head of the defect recognition model to obtain target prediction results includes: predicting the first fused features based on the first detection layer; obtaining the category prediction probability of each first-scale target prediction defect box output by the first classification branch; obtaining the coordinate displacement of each first-scale target prediction defect box output by the first regression branch; and obtaining the temperature anomaly confidence level of each first-scale target prediction defect box output by the first temperature anomaly scoring branch. Based on the prediction of the second fused features by the second detection layer, the category prediction probability of each second-scale target prediction defect box output by the second classification branch is obtained, the coordinate displacement of each second-scale target prediction defect box output by the second regression branch is obtained, and the temperature anomaly confidence of each second-scale target prediction defect box output by the second temperature anomaly scoring branch is obtained. Based on the prediction of the third fused features by the third detection layer, the category prediction probability of each third-scale target prediction defect box output by the third classification branch is obtained, the coordinate displacement of each third-scale target prediction defect box output by the third regression branch is obtained, and the temperature anomaly confidence of each third-scale target prediction defect detection box output by the third temperature anomaly scoring branch is obtained. The target prediction defect boxes of the first scale, the second scale, and the third scale are summarized into a target prediction defect box set.
[0063] Category prediction probability refers to The probability of each category, for example The categories are: hot spots, microcracks, cracks, stains, obstructions, solder joint detachment, and background.
[0064] 205. Post-process the target prediction results to determine the photovoltaic panel defect identification results.
[0065] In this embodiment, the post-processing of the target prediction results to determine the photovoltaic panel defect identification results includes: for any target prediction defect box in the target prediction defect box set, if the maximum value of its category prediction probability is less than a first preset threshold, it is considered a mispredicted target prediction defect box and deleted; if the maximum value of its category prediction probability is greater than or equal to the first preset threshold, the category corresponding to the maximum value of its category prediction probability is taken as its photovoltaic panel defect category; if its photovoltaic panel defect category is a hot spot, it is determined whether its temperature anomaly confidence level is less than a second preset threshold. If so, it is considered a mispredicted target prediction defect box and deleted, until all the target prediction defect boxes are filtered to obtain a filtered target prediction defect box set; for any filtered target prediction defect box in the filtered target prediction defect box set, its coordinates are calculated based on its coordinate displacement; the coordinates of all filtered target prediction defect boxes, the photovoltaic panel defect categories of all filtered target prediction defect boxes, and the temperature anomaly confidence levels of all filtered target prediction defect boxes are taken as the photovoltaic panel defect identification results.
[0066] This invention provides a method, apparatus, and device for photovoltaic panel defect identification based on multispectral fusion. The technical solution of this invention first acquires visible light and infrared thermal images of the photovoltaic panel to be identified. An input vector is constructed based on the visible light and infrared thermal images. Then, the input vector is input into a trained defect identification model. Features are extracted from the input vector using the backbone network of the defect identification model to obtain features at different scales. These features are then fused using the neck network of the defect identification model to obtain multiple fused features. Finally, the fused features are predicted using the decoupled detection head of the defect identification model to obtain a target prediction result. The backbone network is constructed based on a multispectral attention module, and the neck network is based on cross-modal global attention. The decoupled detection head, constructed using a force fusion module and a multispectral attention module, includes a temperature anomaly scoring branch. The defect recognition model is trained based on a joint loss function. Finally, the target prediction result is post-processed to determine the photovoltaic panel defect recognition result. Through the multispectral attention module and the cross-modal global attention fusion module, deep semantic alignment of visible light features and infrared features is achieved, significantly improving the defect recognition accuracy in scenarios with strong reflection and no appearance changes. The joint loss function improves the recognition accuracy of the defect recognition model for microcracks and micro-hot spots with extremely small pixel proportions. The temperature anomaly scoring branch effectively distinguishes between real hot spots and pseudo-high temperature areas caused by environmental reflection, reducing the false detection rate and achieving high-precision recognition of defects such as hot spots, microcracks, and stains on photovoltaic panels.
[0067] Furthermore, as Figure 1 and Figure 2 The specific implementation of the method shown in this invention provides a photovoltaic panel defect identification device based on multispectral fusion, such as... Figure 3 As shown, the device includes: an image acquisition and processing module 31, a prediction module 32, and a determination module 33; The image acquisition and processing module 31 is used to acquire visible light images and infrared thermal imaging images of the photovoltaic panel to be identified, and to construct an input vector based on the visible light images and the infrared thermal imaging images; The prediction module 32 is used to input the input vector into the trained defect recognition model, extract features from the input vector according to the backbone network of the defect recognition model to obtain features at different scales, fuse the features at different scales according to the neck network of the defect recognition model to obtain multiple fused features, and predict the fused features according to the decoupled detection head of the defect recognition model to obtain the target prediction result. The backbone network is constructed based on a multispectral attention module, the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module, the decoupled detection head includes a temperature anomaly scoring branch, and the defect recognition model is obtained by training an initial defect recognition model based on a joint loss function. The determination module 33 is used to post-process the target prediction results to determine the photovoltaic panel defect identification results.
[0068] Accordingly, in order to construct an input vector based on the visible light image and the infrared thermal imaging image, the image acquisition and processing module 31 is used to perform corresponding denoising preprocessing on the visible light image and the infrared thermal imaging image respectively; to perform cross-modal pixel-level registration on the denoised visible light image and the denoised infrared thermal imaging image, and after registration is completed, to stitch them together in the channel dimension to obtain the input vector.
[0069] In specific application scenarios, a photovoltaic panel defect identification device based on multispectral fusion, such as... Figure 4 As shown, the device further includes: a training module 34, used to construct an initial defect recognition model, wherein the initial defect recognition model includes: a backbone network, a neck network, and a decoupled detection head, wherein the backbone network is constructed based on a multispectral attention module, and the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module; acquiring input vector samples, inputting the input vector samples into the initial defect recognition model, outputting initial prediction results, determining prediction results based on the initial prediction results, calculating the value of a joint loss function based on the prediction results and the true results, adjusting the model parameters of the initial defect recognition model based on the value of the joint loss function, until a preset stopping condition is reached, thus obtaining a trained defect recognition model.
[0070] Accordingly, the prediction results include: the coordinates of the predicted defect box, the predicted category probability, and the predicted infrared temperature; the true results include: the coordinates of the true defect box, the true category, and the true infrared temperature. To calculate the value of the joint loss function based on the prediction results and the true results, the training module 34 is used to calculate the bounding box loss based on the coordinates of the predicted defect box and the coordinates of the true defect box; calculate the classification loss based on the predicted category probability and the true category; calculate the temperature anomaly supervision loss based on the predicted infrared temperature and the true infrared temperature; and calculate the value of the joint loss function based on the bounding box loss, the classification loss, and the temperature anomaly supervision loss.
[0071] Accordingly, in order to extract features from the input vector based on the backbone network of the defect recognition model and obtain features at different scales, the prediction module 32 is used to extract features from the input vector based on the first convolutional module, second convolutional module, first multispectral attention module, third convolutional module, second multispectral attention module, fourth convolutional module, third multispectral attention module, fifth convolutional module, fourth multispectral attention module, fast spatial pyramid pooling module, and cross-stage local spatial attention module connected sequentially in the backbone network of the defect recognition model. The second multispectral attention module outputs the first-scale features, the third multispectral attention module outputs the second-scale features, and the cross-stage local spatial attention module outputs the features at different scales. The process of outputting the first-scale feature from the second multispectral attention module is as follows: the features output by the third convolution module are determined and input into the second multispectral attention module to extract the visible light and infrared features. Based on the multispectral interaction branch of the second multispectral attention module, a spatial attention map is generated according to the visible light and infrared features. Based on the temperature gradient enhancement branch of the second multispectral attention module, the temperature change edge features are calculated according to the infrared features. The first-scale feature is calculated according to the spatial attention map, the features output by the third convolution module, and the temperature change edge features.
[0072] Accordingly, the neck network of the defect recognition model includes: a first cross-modal global attention fusion module, a second cross-modal global attention fusion module, a third cross-modal global attention fusion module, a fourth cross-modal global attention fusion module, a first upsampling module, a second upsampling module, a fifth multispectral attention module, a sixth multispectral attention module, a seventh multispectral attention module, an eighth multispectral attention module, a sixth convolution module, and a seventh convolution module. To perform feature fusion on features of different scales based on the neck network of the defect recognition model to obtain multiple fused features, the prediction module 32 is used to input the first-scale feature into the first cross-modal global attention fusion module, input the second-scale feature into the second cross-modal global attention fusion module, input the third-scale feature into the second upsampling module, output a first upsampling feature, input the first upsampling feature into the second cross-modal global attention fusion module, perform feature fusion based on the second-scale feature and the first upsampling feature, output a first fused feature, input the first fused feature into the fifth multispectral attention module to obtain a fourth-scale feature, and input the fourth-scale feature into the first upsampling module. The module outputs a second upsampled feature, inputs the second upsampled feature into the first cross-modal global attention fusion module, performs feature fusion based on the second upsampled feature and the first scale feature, outputs a second fused feature, inputs the second fused feature into the sixth multispectral attention module, outputs a first fused feature, inputs the first fused feature into the sixth convolution module, outputs a first extracted feature, inputs the first extracted feature into the third cross-modal global attention fusion module, performs feature fusion based on the first extracted feature and the fourth scale feature, outputs a third fused feature, inputs the third fused feature into the seventh multispectral attention module, outputs a second fused feature, inputs the second fused feature into the seventh convolution module, outputs a second extracted feature, inputs the second extracted feature into the fourth cross-modal global attention fusion module, performs feature fusion based on the second extracted feature and the third scale feature, outputs a fourth fused feature, inputs the fourth fused feature into the eighth multispectral attention module, outputs a third fused feature, and uses the first fused feature, the second fused feature, and the third fused feature as multiple fused features;The process of inputting the first upsampled feature into the second cross-modal global attention fusion module, and performing feature fusion based on the second scale feature and the first upsampled feature to output the first fused feature is as follows: The first upsampled feature is input into the third convolutional layer to output the first projected feature; the second scale feature is input into the fourth convolutional layer to output the second projected feature; the first projected feature and the second projected feature are concatenated to obtain the concatenated feature; based on the sequentially connected global average pooling layer, first fully connected layer, second activation function layer, second fully connected layer, and third activation function layer, the first channel weight corresponding to the first projected feature and the second channel weight corresponding to the second projected feature are generated according to the concatenated feature; the first projected feature, the second projected feature, the first channel weight, and the second channel weight are input into a weighted summation layer to output the first fused feature.
[0073] Accordingly, the decoupled detection head of the defect recognition model includes: a first detection layer, a second detection layer, and a third detection layer. The first detection layer includes: a first classification branch, a first regression branch, and a first temperature anomaly scoring branch. The second detection layer includes: a second classification branch, a second regression branch, and a second temperature anomaly scoring branch. The third detection layer includes: a third classification branch, a third regression branch, and a third temperature anomaly scoring branch. To predict the fused features based on the decoupled detection head of the defect recognition model and obtain the target prediction result, the prediction module 32 is used to predict the first fused features based on the first detection layer, obtain the category prediction probability of each first-scale target prediction defect box output by the first classification branch, obtain the coordinate displacement of each first-scale target prediction defect box output by the first regression branch, and obtain the temperature anomaly of each first-scale target prediction defect box output by the first temperature anomaly scoring branch. The system calculates the confidence level of the target prediction defect box at each second scale output by the second classification branch based on the prediction of the second fused features by the second detection layer, the coordinate displacement of the target prediction defect box at each second scale output by the second regression branch, and the temperature anomaly confidence level of the target prediction defect box at each second scale output by the second temperature anomaly scoring branch. The system then calculates the confidence level of the target prediction defect box at each third scale output by the third classification branch based on the prediction of the third fused features by the third detection layer, the coordinate displacement of the target prediction defect box at each third scale output by the third regression branch, and the temperature anomaly confidence level of the target prediction defect detection box at each third scale output by the third temperature anomaly scoring branch. Finally, the target prediction defect boxes at the first, second, and third scales are aggregated into a target prediction defect box set.
[0074] Accordingly, in order to post-process the target prediction results and determine the photovoltaic panel defect identification results, the determining module 33 is used to: for any target prediction defect box in the target prediction defect box set, if the maximum value of its category prediction probability is less than a first preset threshold, then it is regarded as a mispredicted target prediction defect box and deleted; if the maximum value of its category prediction probability is greater than or equal to the first preset threshold, then the category corresponding to the maximum value of its category prediction probability is regarded as its photovoltaic panel defect category; if its photovoltaic panel defect category is a hot spot, then it is determined whether its temperature anomaly confidence level is less than a second preset threshold. If so, then it is regarded as a mispredicted target prediction defect box and deleted, until all the target prediction defect boxes are filtered to obtain a filtered target prediction defect box set; for any filtered target prediction defect box in the filtered target prediction defect box set, its coordinates are calculated based on its coordinate displacement; the coordinates of all filtered target prediction defect boxes, the photovoltaic panel defect categories of all filtered target prediction defect boxes, and the temperature anomaly confidence levels of all filtered target prediction defect boxes are used as the photovoltaic panel defect identification results.
[0075] It should be noted that other corresponding descriptions of the functional units involved in the photovoltaic panel defect identification device based on multispectral fusion provided in this embodiment can be found in [reference]. Figures 1 to 2 The corresponding description will not be repeated here.
[0076] Based on the above, Figures 1 to 2 Accordingly, this embodiment also provides a storage medium, which may be volatile or non-volatile, storing a computer program that, when executed by a processor, implements the above-described method. Figures 1 to 2 The method for identifying defects in photovoltaic panels based on multispectral fusion is shown.
[0077] Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present invention.
[0078] Based on the above, Figures 1 to 2 The method shown and Figure 3 , Figure 4 To achieve the above objectives, the present application also provides a computer device, specifically a personal computer, server, network device, etc., as shown in the illustrated embodiment. This computer device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figure 1 and Figure 2The method for identifying defects in photovoltaic panels based on multispectral fusion is shown.
[0079] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.
[0080] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.
[0081] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the non-volatile storage medium, as well as communication with other hardware and software in the information processing entity device.
[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware.
[0083] This invention provides a method, apparatus, and device for photovoltaic panel defect identification based on multispectral fusion. The technical solution of this invention first acquires visible light and infrared thermal images of the photovoltaic panel to be identified. An input vector is constructed based on the visible light and infrared thermal images. Then, the input vector is input into a trained defect identification model. Features are extracted from the input vector using the backbone network of the defect identification model to obtain features at different scales. These features are then fused using the neck network of the defect identification model to obtain multiple fused features. Finally, the fused features are predicted using the decoupled detection head of the defect identification model to obtain a target prediction result. The backbone network is constructed based on a multispectral attention module, and the neck network is based on cross-modal global attention. The decoupled detection head, constructed using a force fusion module and a multispectral attention module, includes a temperature anomaly scoring branch. The defect recognition model is trained based on a joint loss function. Finally, the target prediction result is post-processed to determine the photovoltaic panel defect recognition result. Through the multispectral attention module and the cross-modal global attention fusion module, deep semantic alignment of visible light features and infrared features is achieved, significantly improving the defect recognition accuracy in scenarios with strong reflection and no appearance changes. The joint loss function improves the recognition accuracy of the defect recognition model for microcracks and micro-hot spots with extremely small pixel proportions. The temperature anomaly scoring branch effectively distinguishes between real hot spots and pseudo-high temperature areas caused by environmental reflection, reducing the false detection rate and achieving high-precision recognition of defects such as hot spots, microcracks, and stains on photovoltaic panels.
[0084] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or they can be located in one or more apparatuses different from this embodiment, with corresponding changes. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.
[0085] The serial numbers used above are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenarios. The above disclosures are merely a few specific implementation scenarios of the present invention; however, the present invention is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. A method for identifying defects in photovoltaic panels based on multispectral fusion, characterized in that, The method includes: Acquire visible light and infrared thermal images of the photovoltaic panel to be identified, and construct an input vector based on the visible light and infrared thermal images; The input vector is input into the trained defect recognition model. Based on the backbone network of the defect recognition model, features are extracted from the input vector to obtain features at different scales. Based on the neck network of the defect recognition model, features at different scales are fused to obtain multiple fused features. Based on the decoupled detection head of the defect recognition model, the fused features are predicted to obtain the target prediction result. The backbone network is constructed based on a multispectral attention module, and the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module. The decoupled detection head includes a temperature anomaly scoring branch. The defect recognition model is obtained by training an initial defect recognition model based on a joint loss function. The target prediction results are post-processed to determine the photovoltaic panel defect identification results.
2. The method according to claim 1, characterized in that, The step of constructing an input vector based on the visible light image and the infrared thermal image includes: The visible light image and the infrared thermal imaging image are respectively subjected to corresponding noise reduction preprocessing; Cross-modal pixel-level registration is performed on the denoised visible light image and the denoised infrared thermal image. After registration, the images are stitched together in the channel dimension to obtain the input vector.
3. The method according to claim 1, characterized in that, Before inputting the input vector into the trained defect recognition model, the method further includes: An initial defect identification model is constructed, comprising: a backbone network, a neck network, and a decoupled detection head. The backbone network is constructed based on a multispectral attention module, and the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module. Obtain input vector samples, input the input vector samples into the initial defect identification model, output initial prediction results, determine prediction results based on the initial prediction results, calculate the value of the joint loss function based on the prediction results and the true results, adjust the model parameters of the initial defect identification model based on the value of the joint loss function, until a preset stopping condition is reached, and obtain the trained defect identification model.
4. The method according to claim 3, characterized in that, The prediction results include: the coordinates of the predicted defect box, the predicted probability of the category, and the predicted infrared temperature; the true results include: the coordinates of the true defect box, the true category, and the true infrared temperature. The step of calculating the value of the joint loss function based on the predicted result and the actual result includes: Calculate the bounding box loss based on the coordinates of the predicted defect box and the coordinates of the actual defect box; calculate the classification loss based on the predicted category probability and the actual category; and calculate the temperature anomaly supervision loss based on the predicted infrared temperature and the true infrared temperature. The value of the joint loss function is calculated based on the bounding box loss, the classification loss, and the temperature anomaly supervision loss.
5. The method according to claim 1, characterized in that, The step involves extracting features from the input vector based on the backbone network of the defect identification model to obtain features at different scales, including: Based on the first convolutional module, second convolutional module, first multispectral attention module, third convolutional module, second multispectral attention module, fourth convolutional module, third multispectral attention module, fifth convolutional module, fourth multispectral attention module, fast spatial pyramid pooling module, and cross-stage local spatial attention module sequentially connected in the backbone network of the defect recognition model, feature extraction is performed on the input vector. The second multispectral attention module outputs a first-scale feature, the third multispectral attention module outputs a second-scale feature, and the cross-stage local spatial attention module outputs a third-scale feature. The first-scale feature, the second-scale feature, and the third-scale feature are used as features of different scales. The process of outputting the first-scale feature by the second multispectral attention module is as follows: determine the features output by the third convolution module, input them into the second multispectral attention module, extract the visible light features and infrared features therein, generate a spatial attention map based on the multispectral interaction branch of the second multispectral attention module according to the visible light features and the infrared features, calculate the temperature change edge features based on the temperature gradient enhancement branch of the second multispectral attention module according to the infrared features, and calculate the first-scale feature according to the spatial attention map, the features output by the third convolution module and the temperature change edge features.
6. The method according to claim 5, characterized in that, The neck network of the defect identification model includes: a first cross-modal global attention fusion module, a second cross-modal global attention fusion module, a third cross-modal global attention fusion module, a fourth cross-modal global attention fusion module, a first upsampling module, a second upsampling module, a fifth multispectral attention module, a sixth multispectral attention module, a seventh multispectral attention module, an eighth multispectral attention module, a sixth convolutional module, and a seventh convolutional module; The neck network of the defect identification model is used to fuse features at different scales to obtain multiple fused features, including: The first scale feature is input into the first cross-modal global attention fusion module, the second scale feature is input into the second cross-modal global attention fusion module, the third scale feature is input into the second upsampling module, and a first upsampled feature is output. The first upsampled feature is input into the second cross-modal global attention fusion module, and feature fusion is performed based on the second scale feature and the first upsampled feature, outputting a first fused feature. The first fused feature is input into the fifth multispectral attention module to obtain a fourth scale feature. The fourth scale feature is input into the first upsampling module, and a second upsampled feature is output. The second upsampled feature is input into the first cross-modal global attention fusion module, and feature fusion is performed based on the second upsampled feature and the first scale feature, outputting a second fused feature. The second fused feature is input into the sixth multispectral attention module, and outputs... The first fused feature is input into the sixth convolution module, which outputs the first extracted feature. The first extracted feature is then input into the third cross-modal global attention fusion module. Based on the first extracted feature and the fourth scale feature, feature fusion is performed, and a third fused feature is output. The third fused feature is input into the seventh multispectral attention module, which outputs the second fused feature. The second fused feature is then input into the seventh convolution module, which outputs the second extracted feature. The second extracted feature is then input into the fourth cross-modal global attention fusion module. Based on the second extracted feature and the third scale feature, feature fusion is performed, and a fourth fused feature is output. The fourth fused feature is then input into the eighth multispectral attention module, which outputs the third fused feature. The first fused feature, the second fused feature, and the third fused feature are used as multiple fused features. The process of inputting the first upsampled feature into the second cross-modal global attention fusion module, performing feature fusion based on the second scale feature and the first upsampled feature, and outputting the first fused feature is as follows: The first upsampled feature is input into the third convolutional layer, which outputs the first projected feature. The second scale feature is input into the fourth convolutional layer, which outputs the second projected feature. The first projected feature and the second projected feature are concatenated to obtain the concatenated feature. Based on the sequentially connected global average pooling layer, the first fully connected layer, the second activation function layer, the second fully connected layer, and the third activation function layer, the first channel weight corresponding to the first projected feature and the second channel weight corresponding to the second projected feature are generated according to the concatenated feature. The first projected feature, the second projected feature, the first channel weight, and the second channel weight are input into the weighted summation layer to output the first fused feature.
7. The method according to claim 6, characterized in that, The decoupled detection head of the defect identification model includes: a first detection layer, a second detection layer and a third detection layer. The first detection layer includes: a first classification branch, a first regression branch and a first temperature anomaly scoring branch. The second detection layer includes: a second classification branch, a second regression branch and a second temperature anomaly scoring branch. The third detection layer includes: a third classification branch, a third regression branch and a third temperature anomaly scoring branch. The step of predicting the fused features based on the decoupled detection head of the defect recognition model to obtain the target prediction result includes: predicting the first fused features based on the first detection layer to obtain the category prediction probability of the target prediction defect box at each first scale output by the first classification branch, obtaining the coordinate displacement of the target prediction defect box at each first scale output by the first regression branch, obtaining the temperature anomaly confidence level of the target prediction defect box at each first scale output by the first temperature anomaly scoring branch; predicting the second fused features based on the second detection layer to obtain the category prediction probability of the target prediction defect box at each second scale output by the second classification branch, and obtaining the second regression branch output... The coordinate displacement of each second-scale target predicted defect box is obtained, and the temperature anomaly confidence of each second-scale target predicted defect box output by the second temperature anomaly scoring branch is obtained. The third fused features are predicted according to the third detection layer to obtain the category prediction probability of each third-scale target predicted defect box output by the third classification branch. The coordinate displacement of each third-scale target predicted defect box output by the third regression branch is obtained, and the temperature anomaly confidence of each third-scale target predicted defect detection box output by the third temperature anomaly scoring branch is obtained. The target predicted defect boxes of the first scale, the second scale, and the third scale are summarized into a target predicted defect box set. The post-processing of the target prediction results to determine the photovoltaic panel defect identification results includes: For any target predicted defect box in the target predicted defect box set, if the maximum value of its category prediction probability is less than the first preset threshold, it is regarded as a mispredicted target predicted defect box and deleted. If the maximum value of its category prediction probability is greater than or equal to the first preset threshold, the category corresponding to the maximum value of its category prediction probability is regarded as its photovoltaic panel defect category. If its photovoltaic panel defect category is hot spot, it is determined whether its temperature anomaly confidence is less than the second preset threshold. If so, it is regarded as a mispredicted target predicted defect box and deleted. This process continues until all target predicted defect boxes are filtered to obtain the filtered target predicted defect box set. For any predicted defect box in the set of predicted defect boxes, calculate its coordinates based on its coordinate displacement. The coordinates of all filtered target predicted defect boxes, the photovoltaic panel defect categories of all filtered target predicted defect boxes, and the temperature anomaly confidence scores of all filtered target predicted defect boxes are used as the photovoltaic panel defect identification results.
8. A photovoltaic panel defect identification device based on multispectral fusion, characterized in that, The device includes: The image acquisition and processing module is used to acquire visible light images and infrared thermal imaging images of the photovoltaic panel to be identified, and to construct an input vector based on the visible light images and the infrared thermal imaging images; The prediction module is used to input the input vector into the trained defect recognition model, extract features from the input vector according to the backbone network of the defect recognition model to obtain features at different scales, fuse the features at different scales according to the neck network of the defect recognition model to obtain multiple fused features, and predict the fused features according to the decoupled detection head of the defect recognition model to obtain the target prediction result. The backbone network is constructed based on a multispectral attention module, the neck network is constructed based on a cross-modal global attention fusion module and a multispectral attention module, the decoupled detection head includes a temperature anomaly scoring branch, and the defect recognition model is obtained by training an initial defect recognition model based on a joint loss function. The determination module is used to post-process the target prediction results to determine the photovoltaic panel defect identification results.
9. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the photovoltaic panel defect identification method based on multispectral fusion as described in any one of claims 1 to 7.
10. A computer device comprising a memory, a processor, and a computer program stored on a storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the photovoltaic panel defect identification method based on multispectral fusion as described in any one of claims 1 to 7.
