Photovoltaic hot spot detection method and system based on visible light and thermal infrared image fusion

Abstract

The invention discloses a photovoltaic hot spot detection method and system based on visible light and thermal infrared image fusion, and the method comprises the steps: carrying out the preset multi-scale feature extraction of a visible light image and a thermal infrared image, and obtaining the preset multi-scale visible light image features and thermal infrared image features; performing first processing on the visible light image features and the thermal infrared image features of the same scale to obtain preset multi-scale first features; performing multi-scale feature complementary fusion on the preset multi-scale first features according to the improved pyramid network, and performing second processing on each fusion result to obtain preset multi-scale second features; and obtaining the position and category of the photovoltaic hot spot according to the preset multi-scale second feature in combination with the detection head.

Description

Technical Field

[0001] The present invention relates to the field of photovoltaic inspection technology, and in particular to a photovoltaic hot spot detection method and system based on the fusion of visible light and thermal infrared images. Background Art

[0002] With the rapid growth of energy demand, global photovoltaic power generation capacity has exploded. PV modules, exposed to complex natural environments for extended periods, are susceptible to external and internal factors, leading to various failures and defects. Hot spot defects refer to performance degradation caused by localized overheating in PV modules. Hot spots not only significantly reduce the power generation efficiency of PV modules, but in severe cases, they can spread, causing overall module failure and even fire hazards. Therefore, timely and accurate detection of hot spot defects in PV modules is crucial.

[0003] Traditional manual inspection methods are not universal in terms of efficiency, cost and adaptability. In recent years, with the coordinated development of drone platforms and remote sensing technology, photovoltaic module detection methods based on drone multimodal remote sensing have become a research hotspot in the field of intelligent operation and maintenance of photovoltaic power stations due to their advantages such as high efficiency, flexibility and strong adaptability. By using drone platforms equipped with high-precision thermal infrared imagers, the surface temperature field distribution data of photovoltaic arrays can be efficiently obtained, and combined with advanced computer vision algorithms, the automatic identification and positioning of hot spot defects can be achieved. However, the existing drone infrared photovoltaic hot spot detection technology still has significant limitations. Due to the physical characteristics of infrared sensors, the images they obtain generally have problems such as insufficient spatial resolution and missing texture features, which leads to a decrease in the contrast between the target and the background in the infrared image. In complex power station environments, two typical types of misjudgments are easily caused: (1) background interference is mistakenly identified as hot spots; (2) similar thermal radiation characteristics cause incorrect recognition of hot spot categories. These problems seriously affect the practical application potential and reliability of the detection algorithm.

[0004] Therefore, a new technical solution is urgently needed to solve the technical problem of how to detect photovoltaic hot spots efficiently and accurately. Summary of the Invention

[0005] The present invention provides a photovoltaic hot spot detection method and system based on the fusion of visible light and thermal infrared images, which are used to solve the technical problem of how to detect photovoltaic hot spots efficiently and accurately.

[0006] To achieve the above objectives, the present invention provides a photovoltaic hot spot detection method based on visible light and thermal infrared image fusion, comprising: Preset multi-scale feature extraction is performed on visible light images and thermal infrared images respectively to obtain preset multi-scale visible light image features and thermal infrared image features; a first processing is performed on the visible light image features and thermal infrared image features of the same scale to obtain a preset multi-scale first feature; multi-scale feature complementary fusion is performed on the preset multi-scale first feature based on an improved pyramid network, and a second processing is performed on each fusion result to obtain a preset multi-scale second feature; the location and category of the photovoltaic hot spot are obtained based on the preset multi-scale second feature in combination with the detection head.

[0007] The first processing includes: extracting the difference features between the visible light image features and the thermal infrared image features to obtain the visible light image-specific features and the thermal infrared image-specific features; fusing the visible light image-specific features with the thermal infrared image to obtain the third feature; fusing the thermal infrared image-specific features with the visible light image features to obtain the fourth feature; and adaptively fusing the third feature and the fourth feature to obtain the first feature.

[0008] The second processing includes: extracting local features of the input features to obtain a fifth feature; capturing global context information of the input features to obtain a sixth feature; and adding the fifth feature and the sixth feature to obtain a second feature.

[0009] Preferably, extracting local features of the input features to obtain the fifth feature; capturing global context information of the input features to obtain the sixth feature includes: The input features are sequentially subjected to convolution operation, bottleneck operation, channel addition and 1×1 convolution operation on the convolution operation results of the input features to obtain the fifth feature. , expressed as: ; in, Represents a 1×1 convolution operation; Indicates channel addition; Indicates the bottleneck operation; Represents the convolution operation; Represents the input features.

[0010] The input features are sequentially subjected to layer normalization, 2D selective scanning, connected to the input features through residual connection, and 1×1 convolution operation to obtain the sixth feature. , expressed as: ; in, Represents a 2D selective scanning operation; Representation layer normalization operation.

[0011] Preferably, extracting the difference features between the visible light image features and the thermal infrared image features to obtain the visible light image-specific features and the thermal infrared image-specific features includes: Assume that the visible light image feature is , the thermal infrared image features are ;Will and Generate corresponding queries through 1×1 convolution operation and dimension change q ,key k Sum v feature: ; in, 、 and Represents the query generated based on the thermal infrared image features q ,key k Sum v feature; 、 and Query generation based on visible image features q ,key k Sum v feature; Represents a dimension change operation.

[0012] The unique features of thermal infrared images include: right and Perform matrix multiplication and Normalization operation, we get and The first similarity matrix between ; The first similarity matrix and Perform feature weighted fusion to obtain the first similar feature vector between the visible light image and the thermal infrared image; subtract the first similar feature vector from the thermal infrared image feature to obtain the thermal infrared image-specific feature , expressed as: ; ; Obtaining unique features of visible light images includes: right and Perform matrix multiplication and Normalization operation, we get and The second similarity matrix between ; The second similarity matrix and Perform feature weighted fusion to obtain the second similar feature vector between the visible light image and the thermal infrared image; subtract the second similar feature vector from the visible light image feature to obtain the visible light image-specific feature , expressed as: ; ; in, Represents matrix multiplication; is the scale parameter; Represents matrix operations; express Normalization operation.

[0013] Preferably, the third feature is obtained by fusing the specific features of the visible light image with the thermal infrared image; and the fourth feature is obtained by fusing the specific features of the thermal infrared image with the features of the visible light image, including: The third characteristic include: ; The fourth characteristic include: ; in, and represents the learnable parameters used to adaptively remove noise feature interference.

[0014] Preferably, the third feature and the fourth feature are adaptively fused to obtain the first feature, which includes: Based on the dynamic spatial channel attention mechanism, from the third feature and the fourth characteristic Adaptively select important features for fusion, including: The third feature and the fourth characteristic Splicing along the channel dimension to obtain preliminary fusion features ; Obtain channel weight information based on dynamic convolution channel attention mechanism , according to the channel weight information Calibrate the importance of each channel to obtain the calibrated features : ; ; ; Introducing spatial information After calibration, the features Perform spatial level selection to obtain the first feature : ; ; in, represents the activation function; represents dynamic convolution; represents the average pooling operation; Indicates multiplication; Indicates addition.

[0015] Preferably, performing multi-scale feature complementary fusion on the preset multi-scale first features according to the improved pyramid network, and performing a second processing on each fusion result to obtain the preset multi-scale second features includes: Assume that there are n scales of first features, where n is an integer greater than 2; and the sizes of the first scale to the nth scale decrease.

[0016] The processing path at the nth scale, or the smallest scale, includes: The first feature of the minimum scale is upsampled to obtain the first transmission feature of the minimum scale processing path; the first transmission feature of the minimum scale processing path is sent to the first channel addition node of the adjacent larger scale processing path; the second channel addition node of the minimum scale processing path receives the second transmission feature sent by the adjacent larger scale processing path, and the first feature of the minimum scale and the second transmission feature sent by the adjacent larger scale processing path are channel-added and then subjected to second processing to obtain the second feature of the minimum scale.

[0017] In the processing path from the n-1th to the 2nd scale, it includes: The first channel addition node of the processing path of this scale receives the first transmission feature sent by the adjacent smaller scale processing path, performs channel addition on the first transmission feature sent by the adjacent smaller scale processing path and the first feature of this scale to obtain the first fused feature of this scale; performs second processing on the first fused feature of this scale to obtain the first intermediate feature of this scale; upsamples the first intermediate feature of this scale to obtain the first transmission feature of this scale; and sends the first transmission feature of this scale to the first channel addition node of the processing path of the adjacent larger scale.

[0018] The first intermediate feature of the current scale is sent to the second channel addition node of the processing path of the current scale; the second channel addition node of the processing path of the current scale receives the second transmitted feature sent by the adjacent larger scale processing path, and the first intermediate feature of the current scale is channel-added with the second transmitted feature sent by the adjacent larger scale processing path to obtain the second intermediate feature of the current scale; the second intermediate feature of the current scale is convolved to obtain the second transmitted feature of the current scale; the second transmitted feature of the current scale is sent to the second channel addition node of the processing path of the adjacent smaller scale; the second intermediate feature of the current scale is subjected to the second processing to obtain the second feature of the current scale.

[0019] The first scale, or largest scale, processing path includes: The first channel addition node of the maximum scale processing path receives the first transmission feature sent by the adjacent smaller scale processing path, performs channel addition on the first transmission feature sent by the adjacent smaller scale processing path and the first feature of the maximum scale to obtain a first fusion feature of the maximum scale; performs a second processing on the first fusion feature of the maximum scale to obtain a second feature of the maximum scale; performs a convolution operation on the second feature of the maximum scale to obtain a second transmission feature of the maximum scale, and sends the second transmission feature of the maximum scale to the second channel addition node of the adjacent smaller scale processing path.

[0020] Preferably, the location and category of the photovoltaic hot spot obtained based on the preset multi-scale second feature combined with the detection head include: The preset multi-scale second features are input into the detection head, and the input features are encoded through the convolution layer. The detection head outputs the bounding box coordinate offset, target confidence and category probability distribution in the second features of each scale.

[0021] A coordinate decoding strategy based on sigmoid activation is adopted to convert the bounding box coordinate offsets into prediction boxes.

[0022] The target confidence and category probability distribution are coupled and calculated to obtain the normalized detection confidence score.

[0023] Redundancy optimization of prediction frames is achieved through the non-maximum suppression algorithm: the prediction frame with the highest confidence among the second features of the predicted target at each scale is selected according to the detection confidence score, and redundant frames that highly overlap with the prediction frame with the highest confidence are eliminated to obtain the preferred prediction frame of all predicted targets; the location and category of the photovoltaic hot spot are obtained based on the preferred prediction frame.

[0024] The present invention also provides a photovoltaic hot spot detection system based on the fusion of visible light and thermal infrared images, which is used in the method of the present invention. The system includes a first module, a second module, a third module and a fourth module.

[0025] The first module is used to perform preset multi-scale feature extraction on the visible light image and the thermal infrared image respectively, to obtain preset multi-scale visible light image features and thermal infrared image features.

[0026] The second module is used to perform a first processing on visible light image features and thermal infrared image features of the same scale to obtain preset multi-scale first features.

[0027] The first processing includes: extracting the difference features between the visible light image features and the thermal infrared image features to obtain the visible light image-specific features and the thermal infrared image-specific features; fusing the visible light image-specific features with the thermal infrared image to obtain the third feature; fusing the thermal infrared image-specific features with the visible light image features to obtain the fourth feature; and adaptively fusing the third feature and the fourth feature to obtain the first feature.

[0028] The third module is used to perform multi-scale feature complementary fusion on the preset multi-scale first features according to the improved pyramid network, and perform a second processing on each fusion result to obtain the preset multi-scale second features.

[0029] The second processing includes: extracting local features of the input features to obtain a fifth feature; capturing global context information of the input features to obtain a sixth feature; and adding the fifth feature and the sixth feature to obtain a second feature.

[0030] The fourth module is used to obtain the location and category of the photovoltaic hot spot based on the preset multi-scale second feature combined with the detection head.

[0031] The present invention has the following beneficial effects: The photovoltaic hot spot detection method based on the fusion of visible light and thermal infrared images of the present invention uses drone remote sensing images as input, and can realize hot spot detection of photovoltaic panels under large-scale complex terrain. The method of the present invention combines the imaging advantages of visible light and thermal infrared images, and solves the false detection problem caused by the lack of details in the single-modal image detection method. In order to avoid the loss of important features in the fusion process and fully integrate the complementary features between infrared and visible light, a proprietary feature complementary enhancement strategy is designed, which can enhance the expression of complementary features and effectively solve the problem of the loss of small hot spot features in the subsequent fusion process; and an adaptive feature fusion strategy is designed to realize the selection and fusion of important features through a dynamic attention mechanism, and reduce the interference of noise features. The method of the present invention can overcome the limitations of the single infrared image detection method, quickly adapt to different complex environments, and realize the accurate detection and classification of photovoltaic hot spots. The method of the present invention solves the problems of difficult manual inspections of photovoltaics in complex scenes and insufficient accuracy of single-modal detection.

[0032] The photovoltaic hot spot detection system based on the fusion of visible light and thermal infrared images of the present invention is used in the method of the present invention and has the same beneficial effects as the method of the present invention.

[0033] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings: Figure 1 It is a schematic diagram of a method flow of a preferred embodiment of the present invention.

[0035] Figure 2 It is a schematic diagram of the network framework of the method of the preferred embodiment of the present invention.

[0036] Figure 3 It is a schematic diagram of the specific implementation process of the CFEM module of the preferred embodiment of the present invention.

[0037] Figure 4 It is a schematic diagram of the specific implementation process of the AFFM module in the preferred embodiment of the present invention.

[0038] Figure 5 It is a schematic diagram of the specific implementation process of the DCM module in the preferred embodiment of the present invention.

[0039] Figure 6 3 is a comparison chart of the detection visualization results of the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0040] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, the present invention can be implemented in many different ways as defined and covered by the claims.

[0041] See also Figures 1 to 2 In a preferred embodiment of the present invention, a photovoltaic hot spot detection method based on visible light and thermal infrared image fusion is provided, comprising: S1. Perform preset multi-scale feature extraction on the visible light image and the thermal infrared image respectively to obtain preset multi-scale visible light image features and thermal infrared image features.

[0042] In a preferred embodiment of the present invention, a dual-branch backbone network is used to extract features at preset low, medium, and high scales for both visible light and thermal infrared images. This means that the preset multi-scale is set to the preset low, medium, and high scales. Parameters are not shared between the parallel dual-branch backbone networks, maintaining independent parameter and gradient update processes.

[0043] S2. Perform a first processing on the visible light image features and the thermal infrared image features of the same scale to obtain a preset multi-scale first feature.

[0044] The first processing includes: extracting the difference features between the visible light image features and the thermal infrared image features to obtain the visible light image-specific features and the thermal infrared image-specific features; fusing the visible light image-specific features with the thermal infrared image to obtain the third feature; fusing the thermal infrared image-specific features with the visible light image features to obtain the fourth feature; and adaptively fusing the third feature and the fourth feature to obtain the first feature.

[0045] In a preferred embodiment of the present invention, by Figure 2 The CFEM module in the first process realizes the extraction of the difference features between the visible light image features and the thermal infrared image features, and obtains the unique features of the visible light image and the unique features of the thermal infrared image; fuses the unique features of the visible light image with the thermal infrared image to obtain the third feature; and fuses the unique features of the thermal infrared image with the features of the visible light image to obtain the fourth feature. Figure 2 The AFFM module in the first processing implements the adaptive fusion of the third feature and the fourth feature to obtain the first feature.

[0046] The specific implementation process of the CFEM module can be found in Figure 3 The specific implementation process of the AFFM module can be found in Figure 4 .

[0047] In a preferred embodiment of the present invention, extracting the difference features between the visible light image features and the thermal infrared image features to obtain the visible light image-specific features and the thermal infrared image-specific features includes: In a preferred embodiment of the present invention, proprietary features are acquired based on a cross-attention mechanism.

[0048] Assume that the visible light image feature is , the thermal infrared image features are ;Will and Generate corresponding queries through 1×1 convolution operation and dimension change q ,key k Sum v feature: ; in, 、 and Represents the query generated based on the thermal infrared image features q ,key k Sum v feature; 、 and Represents the query generated based on the visible light image features q ,key k Sumv feature; Represents a dimension change operation.

[0049] The unique features of thermal infrared images include: right and Perform matrix multiplication and Normalization operation, we get and The first similarity matrix between ; The first similarity matrix and Perform feature weighted fusion to obtain the first similar feature vector between the visible light image and the thermal infrared image; subtract the first similar feature vector from the thermal infrared image feature to obtain the thermal infrared image-specific feature , expressed as: ; ; Obtaining unique features of visible light images includes: right and Perform matrix multiplication and Normalization operation, we get and The second similarity matrix between ; The second similarity matrix and Perform feature weighted fusion to obtain the second similar feature vector between the visible light image and the thermal infrared image; subtract the second similar feature vector from the visible light image feature to obtain the visible light image-specific feature , expressed as: ; ; in, Represents matrix multiplication; is the scale parameter; Represents matrix operations; express Normalization operation.

[0050] In a preferred embodiment of the present invention, the third feature is obtained by fusing the specific features of the visible light image with the thermal infrared image; and the fourth feature is obtained by fusing the specific features of the thermal infrared image with the features of the visible light image, including: The third characteristic include: ; The fourth characteristic include: ; in, and represents the learnable parameters used to adaptively remove noise feature interference.

[0051] In a preferred embodiment of the present invention, the third feature and the fourth feature are adaptively fused to obtain the first feature including: Based on the dynamic spatial channel attention mechanism, from the third feature and the fourth characteristic Adaptively select important features for fusion, including: The third feature and the fourth characteristic Splicing along the channel dimension to obtain preliminary fusion features ; Obtain channel weight information based on dynamic convolution channel attention mechanism , according to the channel weight information Calibrate the importance of each channel to obtain the calibrated features : ; ; ; In order to model the spatial correspondence between the two features of infrared and visible light, spatial information is introduced After calibration, the features Perform spatial level selection to obtain the first feature : ; ; in, represents the activation function; represents dynamic convolution; represents the average pooling operation; Indicates multiplication; Indicates addition.

[0052] S3. Perform multi-scale feature complementary fusion on the preset multi-scale first features according to the improved pyramid network, and perform a second processing on each fusion result to obtain the preset multi-scale second features.

[0053] The second processing includes: extracting local features of the input features to obtain a fifth feature; capturing global context information of the input features to obtain a sixth feature; and adding the fifth feature and the sixth feature to obtain a second feature.

[0054] In a preferred embodiment of the present invention, by Figure 2The DCM module in implements the second processing. For the specific implementation process of the DCM module, see Figure 5 .

[0055] In a preferred embodiment of the present invention, extracting local features of the input features to obtain the fifth feature; capturing global context information of the input features to obtain the sixth feature includes: The input features are sequentially subjected to convolution operation, bottleneck operation, channel addition and 1×1 convolution operation, which reduces the computational complexity and the number of parameters while ensuring the effective transmission of key information, and obtain the fifth feature. , expressed as: ; in, Represents a 1×1 convolution operation; Indicates channel addition; Indicates the bottleneck operation; Represents the convolution operation; Represents the input features.

[0056] The input features are sequentially subjected to layer normalization, 2D selective scanning, connected to the input features through residual connection, and 1×1 convolution operation to obtain the sixth feature. , expressed as: ; in, Represents a 2D selective scanning operation; Representation layer normalization operation.

[0057] The 2D selective scanning operation performs multi-angle scanning from multiple directions of image features, allowing each feature element to integrate information from multiple positions, forming a global perception field of view and enhancing the ability to extract global context information.

[0058] In a preferred embodiment of the present invention, the fifth feature and the sixth feature are added together to obtain the second feature f include: ; In a preferred embodiment of the present invention, S3 specifically includes: Assume that there are n scales of first features, where n is an integer greater than 2; the sizes of the first scale to the nth scale decrease; The processing path at the nth scale, or the smallest scale, includes: The first feature of the minimum scale is upsampled to obtain the first transmission feature of the minimum scale processing path; the first transmission feature of the minimum scale processing path is sent to the first channel addition node of the adjacent larger scale processing path; the second channel addition node of the minimum scale processing path receives the second transmission feature sent by the adjacent larger scale processing path, and the first feature of the minimum scale and the second transmission feature sent by the adjacent larger scale processing path are channel-added and then subjected to second processing to obtain the second feature of the minimum scale.

[0059] In the processing path from the n-1th to the 2nd scale, it includes: The first channel addition node of the processing path of this scale receives the first transmission feature sent by the adjacent smaller scale processing path, performs channel addition on the first transmission feature sent by the adjacent smaller scale processing path and the first feature of this scale to obtain the first fused feature of this scale; performs second processing on the first fused feature of this scale to obtain the first intermediate feature of this scale; upsamples the first intermediate feature of this scale to obtain the first transmission feature of this scale; and sends the first transmission feature of this scale to the first channel addition node of the processing path of the adjacent larger scale.

[0060] The first intermediate feature of the current scale is sent to the second channel addition node of the processing path of the current scale; the second channel addition node of the processing path of the current scale receives the second transmitted feature sent by the adjacent larger scale processing path, and the first intermediate feature of the current scale is channel-added with the second transmitted feature sent by the adjacent larger scale processing path to obtain the second intermediate feature of the current scale; the second intermediate feature of the current scale is convolved to obtain the second transmitted feature of the current scale; the second transmitted feature of the current scale is sent to the second channel addition node of the processing path of the adjacent smaller scale; the second intermediate feature of the current scale is subjected to the second processing to obtain the second feature of the current scale.

[0061] The first scale, or largest scale, processing path includes: The first channel addition node of the maximum scale processing path receives the first transmission feature sent by the adjacent smaller scale processing path, performs channel addition on the first transmission feature sent by the adjacent smaller scale processing path and the first feature of the maximum scale to obtain a first fusion feature of the maximum scale; performs a second processing on the first fusion feature of the maximum scale to obtain a second feature of the maximum scale; performs a convolution operation on the second feature of the maximum scale to obtain a second transmission feature of the maximum scale, and sends the second transmission feature of the maximum scale to the second channel addition node of the adjacent smaller scale processing path.

[0062] In a preferred embodiment of the present invention, after complementary fusion of pre-set multi-scale first features using an improved pyramid network, each fusion result undergoes a second processing step, rather than processing it through the C3 module in a traditional pyramid network. This second processing step, by introducing a state-space model, generates multi-level features that combine global semantic consistency with local detail accuracy. This enhances the global perceptual field of view and the ability to capture long-range dependencies, thereby increasing the robustness of the present method across diverse scenarios.

[0063] S4. Determine the location and category of the photovoltaic hot spot based on the preset multi-scale second feature combined with the detection head. S4 specifically includes: The preset multi-scale second features are input into the detection head, and the input features are encoded through the convolution layer. The detection head outputs the bounding box coordinate offset, target confidence and category probability distribution in the second features of each scale.

[0064] A coordinate decoding strategy based on sigmoid activation is adopted to convert the bounding box coordinate offsets into prediction boxes.

[0065] The target confidence and category probability distribution are coupled and calculated to obtain the normalized detection confidence score.

[0066] Redundancy optimization of prediction frames is achieved through the non-maximum suppression algorithm: the prediction frame with the highest confidence among the second features of the predicted target at each scale is selected according to the detection confidence score, and redundant frames that highly overlap with the prediction frame with the highest confidence are eliminated to obtain the preferred prediction frame of all predicted targets; the location and category of the photovoltaic hot spot are obtained based on the preferred prediction frame.

[0067] The photovoltaic hot spot detection method based on the fusion of visible light and thermal infrared images of the present invention uses drone remote sensing images as input, and can realize hot spot detection of photovoltaic panels under large-scale complex terrain. The method of the present invention combines the imaging advantages of visible light and thermal infrared images, and solves the false detection problem caused by the lack of details in the single-modal image detection method. In order to avoid the loss of important features in the fusion process and fully integrate the complementary features between infrared and visible light, a proprietary feature complementary enhancement strategy is designed, which can enhance the expression of complementary features and effectively solve the problem of the loss of small hot spot features in the subsequent fusion process; and an adaptive feature fusion strategy is designed to realize the selection and fusion of important features through a dynamic attention mechanism, and reduce the interference of noise features. The method of the present invention can overcome the limitations of the single infrared image detection method, quickly adapt to different complex environments, and realize the accurate detection and classification of photovoltaic hot spots. The method of the present invention solves the problems of difficult manual inspections of photovoltaics in complex scenes and insufficient accuracy of single-modal detection.

[0068] In a preferred embodiment of the present invention, a photovoltaic hot spot detection system based on visible light and thermal infrared image fusion is also provided, which is used in the method of the present invention. The system includes a first module, a second module, a third module and a fourth module; The first module is used to perform preset multi-scale feature extraction on the visible light image and the thermal infrared image respectively, to obtain preset multi-scale visible light image features and thermal infrared image features.

[0069] The second module is used to perform a first processing on visible light image features and thermal infrared image features of the same scale to obtain preset multi-scale first features.

[0070] The first processing includes: extracting the difference features between the visible light image features and the thermal infrared image features to obtain the visible light image-specific features and the thermal infrared image-specific features; fusing the visible light image-specific features with the thermal infrared image to obtain the third feature; fusing the thermal infrared image-specific features with the visible light image features to obtain the fourth feature; and adaptively fusing the third feature and the fourth feature to obtain the first feature.

[0071] The third module is used to perform multi-scale feature complementary fusion on the preset multi-scale first features according to the improved pyramid network, and perform a second processing on each fusion result to obtain the preset multi-scale second features.

[0072] The second processing includes: extracting local features of the input features to obtain a fifth feature; capturing global context information of the input features to obtain a sixth feature; and adding the fifth feature and the sixth feature to obtain a second feature.

[0073] The fourth module is used to obtain the location and category of the photovoltaic hot spot based on the preset multi-scale second feature combined with the detection head.

[0074] The photovoltaic hot spot detection system based on the fusion of visible light and thermal infrared images of the present invention is used in the method of the present invention and has the same beneficial effects as the method of the present invention.

[0075] Verification part: A model is constructed according to the method of the present invention; the parameters of the dual-branch backbone network are initialized by pre-trained YOLOv5 weights; the visible light branch and the thermal infrared branch use the same network structure but each maintains an independent parameter space and gradient flow. A preset training set is obtained to train the model; in the early stage of training, the strategy of freezing the shallow network parameters is used to retain the model's advantages in basic feature extraction, while at the same time, the feature adaptation capability for photovoltaic scenes is achieved through a trainable high-level network. This partially frozen training strategy can not only ensure that the network has robust feature expression capabilities, but also enable the model to gradually learn the specific representation of photovoltaic hot spots in dual-modal images. And by jointly optimizing the multi-task loss function constructed by classification loss, confidence loss and bounding box loss, efficient end-to-end training of the model is achieved, thereby achieving accurate detection of photovoltaic hot spot targets in complex scenes by the model. Set training rounds and train the model based on the joint loss function. The joint loss function includes: ; in represents the total training loss; represents the bounding box loss; represents the confidence loss; represents the classification loss; and Represents the weight coefficient of each loss.

[0076] After completing the model training, the model of the present invention is used to detect photovoltaic hot spots based on visible light and thermal infrared drone images; and YOLOv5 is used to detect photovoltaic hot spots on single-modal drone images; the comparison chart of the detection visualization results can be found in Figure 6 .

[0077] Data analysis was performed based on a comparison chart of the output and detection visualization results of the model of the present invention. Compared with the single-modal YOLOv5, the method of the present invention achieved performance improvements of 11.2% and 3.9% in the mAP@0.5 and mAP@0.5-0.95 indicators, respectively. At the same time, the classification accuracy P of the three categories of internal defects, shadow occlusion, and vegetation occlusion was improved by 1.1%, 4.7%, and 10.4%, respectively. The results of YOLOv5 had more detection omissions and errors than the results of the method of the present invention. The method of the present invention identified multiple internal defects, shadow occlusions, and vegetation occlusions, and was more accurate than the results of YOLOv5. In other words, the method of the present invention significantly improved the detection performance of photovoltaic hot spots in complex scenes.

[0078] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A photovoltaic hot spot detection method based on visible light and thermal infrared image fusion, characterized in that: include: Performing preset multi-scale feature extraction on the visible light image and the thermal infrared image respectively to obtain preset multi-scale visible light image features and thermal infrared image features; Performing a first processing on visible light image features and thermal infrared image features of the same scale to obtain a preset multi-scale first feature; Performing multi-scale feature complementary fusion on the preset multi-scale first features according to the improved pyramid network, and performing a second processing on each fusion result to obtain the preset multi-scale second features; Obtaining the location and category of the photovoltaic hot spot based on the preset multi-scale second feature combined with the detection head; The first processing includes: extracting difference features between the visible light image feature and the thermal infrared image feature to obtain a visible light image-specific feature and a thermal infrared image-specific feature; fusing the visible light image-specific feature with the thermal infrared image to obtain a third feature; fusing the thermal infrared image-specific feature with the visible light image feature to obtain a fourth feature; and adaptively fusing the third feature with the fourth feature to obtain a first feature; The second processing includes: extracting local features of the input features to obtain a fifth feature; capturing global context information of the input features to obtain a sixth feature; and adding the fifth feature and the sixth feature to obtain a second feature.

2. The photovoltaic hot spot detection method based on visible light and thermal infrared image fusion according to claim 1 is characterized in that: Extracting local features of the input features to obtain the fifth feature; capturing global context information of the input features to obtain the sixth feature include: The input features are sequentially subjected to convolution operation, bottleneck operation, channel addition and 1×1 convolution operation on the convolution operation results of the input features to obtain the fifth feature. , expressed as: ； in, Represents a 1×1 convolution operation; Indicates channel addition; Indicates the bottleneck operation; Represents the convolution operation; Represents input features; The input features are sequentially subjected to layer normalization, 2D selective scanning, connected with the input features through residual connection, and 1×1 convolution operation to obtain the sixth feature. , expressed as: ； in, Represents a 2D selective scanning operation; Representation layer normalization operation.

3. The photovoltaic hot spot detection method based on visible light and thermal infrared image fusion according to claim 2 is characterized in that: Extracting the difference features between the visible light image features and the thermal infrared image features to obtain the visible light image-specific features and the thermal infrared image-specific features includes: Assume that the visible light image feature is , the thermal infrared image feature is ;Will and Generate corresponding queries through 1×1 convolution operation and dimension change q ,key k Sum v feature: ； in, 、 and Represents the query generated based on the thermal infrared image features q ,key k Sum v feature; 、 and Represents the query generated based on the visible light image features q ,key k Sum v feature; Indicates dimension change operation; The unique features of thermal infrared images include: right and Perform matrix multiplication and Normalization operation, we get and The first similarity matrix between ; The first similarity matrix and Perform feature weighted fusion to obtain the first similar feature vector between the visible light image and the thermal infrared image; subtract the first similar feature vector from the thermal infrared image feature to obtain the thermal infrared image-specific feature , expressed as: ； ； Obtaining unique features of visible light images includes: right and Perform matrix multiplication and Normalization operation, we get and The second similarity matrix between ; The second similarity matrix and Perform feature weighted fusion to obtain the second similar feature vector between the visible light image and the thermal infrared image; subtract the second similar feature vector from the visible light image feature to obtain the visible light image-specific feature , expressed as: ； ； in, Represents matrix multiplication; is the scale parameter; Represents matrix operations; express Normalization operation.

4. The photovoltaic hot spot detection method based on visible light and thermal infrared image fusion according to claim 3 is characterized in that: fusing the specific feature of the visible light image with the thermal infrared image to obtain a third feature; The fourth feature obtained by fusing the thermal infrared image-specific feature with the visible light image feature includes: The third feature include: ； The fourth feature include: ； in, and represents the learnable parameters used to adaptively remove noise feature interference.

5. The photovoltaic hot spot detection method based on visible light and thermal infrared image fusion according to claim 4 is characterized in that: Adaptively fusing the third feature and the fourth feature to obtain a first feature includes: Based on the dynamic spatial channel attention mechanism, from the third feature And the fourth feature Adaptively select important features for fusion, including: The third feature And the fourth feature Splicing along the channel dimension to obtain preliminary fusion features ; Obtain channel weight information based on dynamic convolution channel attention mechanism , according to the channel weight information Calibrate the importance of each channel to obtain the calibrated features : ； ； ； Introducing spatial information After calibration, the features Perform spatial level selection to obtain the first feature : ； ； in, represents the activation function; represents dynamic convolution; represents the average pooling operation; Indicates multiplication; Indicates addition.

6. The photovoltaic hot spot detection method based on visible light and thermal infrared image fusion according to claim 5 is characterized in that: The step of performing multi-scale feature complementary fusion on the preset multi-scale first features according to the improved pyramid network and performing a second processing on each fusion result to obtain the preset multi-scale second features includes: Assume that there are n scales of first features, where n is an integer greater than 2; the sizes of the first scale to the nth scale decrease; The processing path at the nth scale, or the smallest scale, includes: Upsampling the first feature of the minimum scale to obtain a first transmitted feature of the minimum scale processing path; sending the first transmitted feature of the minimum scale processing path to a first channel addition node of an adjacent larger scale processing path; a second channel addition node of the minimum scale processing path receives a second transmitted feature sent by the adjacent larger scale processing path, performs channel addition on the first feature of the minimum scale and the second transmitted feature sent by the adjacent larger scale processing path, and then performs a second processing to obtain a second feature of the minimum scale; In the processing path from the n-1th to the 2nd scale, it includes: The first channel addition node of the processing path at the current scale receives the first transmitted feature sent by the adjacent smaller-scale processing path, performs channel addition on the first transmitted feature sent by the adjacent smaller-scale processing path and the first feature of the current scale to obtain a first fused feature of the current scale; performs a second processing on the first fused feature of the current scale to obtain a first intermediate feature of the current scale; performs upsampling on the first intermediate feature of the current scale to obtain a first transmitted feature of the current scale; and sends the first transmitted feature of the current scale to the first channel addition node of the processing path at the adjacent larger scale; Sending the first intermediate feature of the current scale to the second channel addition node of the current scale processing path; the second channel addition node of the current scale processing path receives the second transmitted feature sent by the adjacent larger scale processing path, performing channel addition on the first intermediate feature of the current scale and the second transmitted feature sent by the adjacent larger scale processing path, obtaining the second intermediate feature of the current scale; performing a convolution operation on the second intermediate feature of the current scale to obtain the second transmitted feature of the current scale; sending the second transmitted feature of the current scale to the second channel addition node of the adjacent smaller scale processing path; performing a second processing on the second intermediate feature of the current scale to obtain the second feature of the current scale; The first scale, or largest scale, processing path includes: The first channel addition node of the maximum scale processing path receives the first transmission feature sent by the adjacent smaller scale processing path, performs channel addition on the first transmission feature sent by the adjacent smaller scale processing path and the first feature of the maximum scale to obtain a first fusion feature of the maximum scale; performs a second processing on the first fusion feature of the maximum scale to obtain a second feature of the maximum scale; performs a convolution operation on the second feature of the maximum scale to obtain a second transmission feature of the maximum scale, and sends the second transmission feature of the maximum scale to the second channel addition node of the adjacent smaller scale processing path.

7. The photovoltaic hot spot detection method based on visible light and thermal infrared image fusion according to claim 6 is characterized in that: The location and category of the photovoltaic hot spot obtained by combining the preset multi-scale second feature with the detection head include: Inputting the preset multi-scale second features into a detection head, encoding the input features through a convolutional layer, and outputting the bounding box coordinate offset, target confidence, and category probability distribution in the second features of each scale; A sigmoid-activated coordinate decoding strategy is used to convert the bounding box coordinate offset into a prediction box. The target confidence and category probability distribution are coupled and calculated to obtain a normalized detection confidence score; Redundancy optimization of prediction frames is achieved through a non-maximum suppression algorithm: the prediction frame with the highest confidence in the second feature of the predicted target at each scale is selected according to the detection confidence score, and redundant frames that highly overlap with the prediction frame with the highest confidence are eliminated to obtain the preferred prediction frame of all predicted targets; the location and category of the photovoltaic hot spot are obtained based on the preferred prediction frame.

8. A photovoltaic hot spot detection system based on visible light and thermal infrared image fusion, used in the method according to any one of claims 1 to 7, characterized in that: The system includes a first module, a second module, a third module and a fourth module; The first module is used to perform preset multi-scale feature extraction on the visible light image and the thermal infrared image respectively to obtain preset multi-scale visible light image features and thermal infrared image features; The second module is used to perform a first processing on visible light image features and thermal infrared image features of the same scale to obtain preset multi-scale first features; The first processing includes: extracting difference features between the visible light image feature and the thermal infrared image feature to obtain a visible light image-specific feature and a thermal infrared image-specific feature; fusing the visible light image-specific feature with the thermal infrared image to obtain a third feature; fusing the thermal infrared image-specific feature with the visible light image feature to obtain a fourth feature; and adaptively fusing the third feature with the fourth feature to obtain a first feature; The third module is used to perform multi-scale feature complementary fusion on the preset multi-scale first features according to the improved pyramid network, and perform a second processing on each fusion result to obtain the preset multi-scale second features; The second processing includes: extracting local features of the input features to obtain a fifth feature; capturing global context information of the input features to obtain a sixth feature; adding the fifth feature and the sixth feature to obtain a second feature; The fourth module is used to obtain the location and category of the photovoltaic hot spot based on the preset multi-scale second feature in combination with the detection head.

Images