Target perception-based photovoltaic visible light and infrared image fusion method

Abstract

The invention discloses a photovoltaic visible light and infrared image fusion method based on target perception. The method comprises the following steps: constructing a multispectral photovoltaic module scene data set comprising at least 1284 pairs of visible light and infrared spectrum alignment images; information of an infrared image and information of visible light in a Y channel of a YCrCb color space are fused through an image feature enhancement (LTENet) network, useful information in two spectral images is extracted and fused through cooperative work of an encoder, a feature sensing module and a local target enhancement module, and an enhanced Y channel image is generated; a multi-channel fusion network (MCFNet) is utilized to combine the enhanced visible light and infrared features, and a high-quality fusion image containing two kinds of spectral information is generated. The method effectively solves the problems of lack of image data and insufficient information fusion of the photovoltaic module under different spectrums, provides more accurate and reliable image data support for monitoring and maintenance of the photovoltaic module, and has a wide application prospect.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a method for fusing photovoltaic visible light and infrared images based on target perception. Background Art

[0002] Solar photovoltaic power generation has become a field of great concern and development worldwide due to its outstanding features such as safety and environmental protection. However, long-term operation and adverse weather conditions can cause various problems in photovoltaic modules, impairing their normal functions and power generation efficiency. To ensure stable operation of the equipment, daily inspections and timely maintenance are essential priorities.

[0003] Traditional inspection methods, such as manual inspection with handheld infrared cameras, are not only time-consuming and labor-intensive, but also inefficient. With the advancement of technology, drones combined with deep learning image processing technology have become the mainstream large-scale inspection method. Although this method has improved the inspection efficiency to a certain extent, some defects are still difficult to accurately capture in single-spectrum images. For example, shadow occlusion is almost undetectable in visible light images, while the characteristics of glass cracks in infrared images are relatively weak.

[0004] maintain

[0005] The commonly used image fusion techniques currently include traditional techniques and data-driven deep learning methods. Traditional methods mainly rely on image processing and digital signal processing techniques. Classic fusion techniques include methods based on multi-scale decomposition, subspace clustering, sparse representation and optimization. The images synthesized by the above methods can meet the needs of subsequent tasks in specific scenarios. However, the transformations or representations used by traditional methods are becoming more and more complex, which is not conducive to real-time computing applications; and the hand-crafted activity level measurements and fusion rules cannot adapt to complex scenarios. More efficient and adaptive solutions need to be developed to meet the changing needs of this field. Image fusion based on data-driven methods mainly includes the following steps: initial feature extraction, feature fusion, and image reconstruction. Although contemporary deep learning-based techniques have skillfully integrated key and complementary information from visible and infrared source images, there are still inherent obstacles in the field of photovoltaic diagnosis such as hot spot identification and classification. This is mainly because high-quality paired data sets are difficult to capture; in addition, contemporary fusion models mainly emphasize global features and textures, but the failure of photovoltaic modules is usually local, resulting in less than ideal fusion results.

[0006] Through literature search of the prior art, it is found that the direction of this type of patents is mainly focused on designing feature extraction networks and feature fusion networks to fuse photovoltaic visible light and infrared images. In the Chinese patent "Photovoltaic visible light and infrared image fusion method based on generative adversarial network", Wu Hao et al. proposed a method for image fusion of target visible light image and target infrared image based on target generative adversarial network. This method aligns and fuses the initial visible light and infrared images respectively by constructing two models, homography network and target generative adversarial network. However, the invention failed to fully consider the application scenario of photovoltaic diagnosis and was insufficient in capturing local features. In addition, in the Chinese patent "Photovoltaic defect detection method by fusion of infrared and visible light images", Qiu Ling et al. generated image feature description based on multiple attention layers, used the feature matching method without detection to perform image registration through local neighborhood and large context information, obtained the transmission transformation image according to the registration result, and weighted it with visible light in equal proportion to obtain the image fusion result. However, this method relies too much on the accuracy of registration, which may cause distortion of the fusion result; in addition, the selection of weighted fusion parameters limits its applicability and accuracy under different environmental conditions.

[0007] In summary, the existing photovoltaic image fusion methods have some problems, such as difficulty in capturing appropriate feature alignment and matching pairs, failure to consider the actual landing scene requirements, and insufficient model adaptability. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a photovoltaic visible light and infrared image fusion method based on target perception, aiming to solve the problems of lack of image data and insufficient information fusion of photovoltaic (PV) modules under visible light and infrared spectra. By constructing a multi-spectral photovoltaic module scene dataset, an image feature enhancement network (LTENet) and a visible light and infrared image fusion network (MCFNet) are adopted to achieve effective fusion and enhancement of image information of photovoltaic modules under different spectra, providing more accurate and reliable image data support for the monitoring and maintenance of photovoltaic modules.

[0009] The technical solution of the present invention is as follows:

[0010] A photovoltaic visible light and infrared image fusion method based on target perception is characterized in that it includes the following steps:

[0011] Step S1: construct a multi-spectral photovoltaic module scene (MSPVS) dataset, which consists of at least 1284 pairs of aligned visible light and infrared spectrum images, and each pair of images has spatial consistency and a specific spatial resolution;

[0012] Step S2: Image enhancement, using LTENet network to fuse the infrared image and the visible light information in the Y channel of the YCrCb color space. The LTENet network includes an encoder, two parallel feature perception modules and two parallel local target enhancement modules, wherein:

[0013] The encoder is used to receive inputs from infrared and visible light images and extract mixed features containing information from the two domains respectively;

[0014] Feature perception module, used to identify and extract specific area features of visible light and infrared images from mixed features;

[0015] The local target enhancement module is used to adaptively adjust the image features to ensure that the contrast and texture information of the original image are maintained during the fusion process, and to generate an enhanced Y channel image;

[0016] Step S3: Feature fusion, using MCFNet to combine the enhanced visible light and infrared features to generate a fused image containing two spectral information, the MCFNet includes a multi-channel attention module and a decoder, wherein:

[0017] Multi-channel attention module, which uses a multi-head self-attention mechanism to fuse feature information from different channels;

[0018] The decoder is responsible for decoding the self-attention feature map output by the multi-channel attention module into the final fused image.

[0019] By converting visible light and infrared images into YCrCb color space, extracting Y channel information as the input of LTENet, generating enhanced Y channel image through the collaborative work of encoder, FPBlock and LTEBlock, and combining the enhanced visible light and infrared features using MCFNet to generate a high-quality fused image containing two spectral information; then recombining with the Cr and Cb channels of the original image and converting back to RGB color space to obtain the final enhanced image.

[0020] Further, in step S1, the MSPVS dataset consists of at least 1284 pairs of aligned images of visible light and infrared spectra, each pair of images having a spatial resolution of 480×640 pixels, which are captured by an image capture device.

[0021] Further, in step S2, the encoder of the LTENet network includes at least four convolutional layers, wherein the kernel size of the first convolutional layer is 1×1, which is used to adjust the number of channels of the input image, and the kernel sizes of the next three convolutional layers are 3×3, which are used to further extract image features, and the activation function is LReLU.

[0022] Furthermore, each of the feature perception modules includes a mainstream and two residual streams, the mainstream deploys two general 1×1 convolutional layers with LReLU, and the two residual streams respectively integrate the original input and the output after the first group of 1×1 convolutional layers to extract the strong texture of the global features.

[0023] Furthermore, the local target enhancement module adaptively adjusts the image features to ensure that the contrast and texture information of the original image are maintained during the fusion process. The formula is as follows:

[0024]

[0025] Among them, conv(·) represents several convolution operations, concat(·) represents cascade, and ⊕ refers to the sum of elements. is the Sobel operator is the Laplace operator, which is used to amplify the fused features. and They represent the enhanced visible light features and infrared image features respectively.

[0026] Further, in step S3, the multi-channel attention module generates Q, K and V feature maps respectively through three 1×1 convolutional layers, wherein the Q feature map and the K feature map are used to calculate the attention weights, and the V feature map is weighted summed according to these weights to generate a self-attention feature map.

[0027] Further, in step S3, the decoder includes multiple 3×3 convolutional layers and at least one 1×1 convolutional layer, and the activation functions of all layers except the last layer are LReLU, and the activation function of the last layer is Tanh, so as to output a fused image with a desired size and number of channels.

[0028] Furthermore, it also includes enhanced loss design and local perceptual loss hybrid optimization, wherein the enhanced loss design is used to constrain the fused image to retain important target information of the infrared image, and the local perceptual loss hybrid optimization includes auxiliary intensity loss, gradient loss and color consistency loss, which are used to achieve feature enhancement and color balance.

[0029] Furthermore, the color consistency loss uses discrete cosine similarity to measure the similarity of color distribution between the fused image and the original visible image to reduce color distortion caused by image enhancement and inject the intensity of the visible image into the fused image.

[0030] Compared with the prior art, the advantages of the present invention are

[0031] 1) The LTENet network is used to fuse the infrared image and the visible light information in the Y channel of the YCrCb color space. The LTENet network is uniquely designed and includes an encoder, two parallel feature perception modules, and two parallel local target enhancement modules. The encoder can extract mixed features containing information from two domains; the feature perception module can accurately identify and extract features of specific regions; and the local target enhancement module adaptively adjusts image features to ensure that the contrast and texture information of the original image are maintained during the fusion process.

[0032] 2) By designing the feature perception (FPBlock) module, the ability to accurately extract specific area features from the mixed features of infrared and visible light images is achieved. This enables the model to intelligently focus on key target areas, effectively avoiding redundant fusion of information and loss of details. It not only improves the ability to retain image details, but also enhances the ability of the fused image to express key information, providing strong support for the accurate monitoring of photovoltaic modules.

[0033] 3) In order to solve the problem that traditional image fusion methods lack details under different lighting conditions, a local target and contrast enhancement (LTEBlock) module was designed. This module can effectively maintain image details and enhance salient targets in complex lighting environments by adaptively adjusting the contrast and texture of the image. This not only improves the visual quality of the fused image, but also enables the model to maintain stable performance under both strong and weak light conditions, providing a solid guarantee for the reliable monitoring of photovoltaic modules.

[0034] 4) The introduced multi-channel attention (MCABlock) module effectively focuses the model's attention on different feature dimensions through the multi-channel attention mechanism. It not only enhances the model's ability to process different information, but also significantly improves the quality of the fused image. In complex background and target scenes, the MCABlock module can better maintain the complementary information of visible light and infrared images, and achieve a more accurate and comprehensive image fusion effect. This technical effect provides more comprehensive and reliable image data support for the monitoring and maintenance of photovoltaic modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of a photovoltaic visible light and infrared image fusion method based on target perception according to an embodiment of the present invention;

[0036] Figure 2 LTENet architecture diagram in the present invention;

[0037] Figure 3 MCFNet architecture diagram in the present invention;

[0038] Figure 4Fusion results of GTF, DenseFuse, FusionGAN, U2Fusion, SDNet, GANMcC, SwinFusion and LTA-FNet;

[0039] Figure 5 : Quantitative results on visual information fidelity, standard deviation, quality index, information entropy, structural similarity and mutual information for 25 image pairs in the MSPV dataset;

[0040] Figure 6 : Visualization results of ablation experiments on 5 representative image pairs in the MSPV dataset, from top to bottom: visible light, infrared, LTA-FNet, results without LTEBlock architecture, and results without MCABlock architecture. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments, but this should not limit the protection scope of the present invention.

[0042] This embodiment provides a photovoltaic visible light and infrared image fusion method based on target perception, integrating LTENet and MCFNet to form an end-to-end image fusion system. Through the feature extraction and enhancement of LTENet and the fusion strategy of MCFNet, visible light and infrared image information can be efficiently fused to generate high-quality fused images.

[0043] See also Figure 1 , Figure 1 : is a flow chart of a photovoltaic visible light and infrared image fusion method based on target perception according to an embodiment of the present invention. As shown in the figure, the method specifically includes the following steps:

[0044] Step S1. Constructing a multi-spectral photovoltaic module scene (MSPVS) dataset: to address the lack of available image data on photovoltaic (PV) modules under these two spectra.

[0045] The dataset of this embodiment consists of 1284 pairs of aligned images of visible light and infrared spectrum, ensuring the consistency of the two spectral images in space and time. The spatial resolution of each pair of images is 480×640 pixels, and they are captured by the HIKMICRO device model HM-TPK20. ir represents infrared image, I vr represents a visible light image, Represents the Y channel of the visible light image.

[0046] Step S2. Image enhancement: An image feature enhancement (LTENet) network is used to fuse the information of the infrared image and the visible light in the Y channel of the YCrCb color space, thereby highlighting the brightness and details of the image while reducing the interference of color information.

[0047] The LTENet network consists of an encoder, two parallel feature perception modules (FPBlock), and two parallel local target enhancement modules (LTEBlock), which work together to extract and fuse useful information from two spectral images, such as Figure 2 shown.

[0048] The architecture and specific processing flow of the LTENet network are as follows:

[0049] Encoder: Receives input from infrared and visible light images and extracts mixed features σ containing information from the two domains. ir and σ vi , the formula is as follows:

[0050]

[0051] Among them, E(:) represents Encoder.

[0052] Feature perception module (FPBlock): receives input of mixed features containing information from two domains, and identifies and extracts specific area features of visible light and infrared images from the mixed features to obtain infrared features Visible Features The formula is as follows:

[0053]

[0054] Among them, FP(:) stands for FPBlock.

[0055] Local object enhancement module (LTEBlock): It can adaptively adjust image features to ensure that the contrast and texture information of the original image are maintained during the fusion process:

[0056]

[0057] Among them, conv(·) represents several convolution operations, conat(·) represents cascade, ⊕ refers to the sum of elements, and the Sobel operator is used at the same time. and the Laplacian operator To amplify the fused features.

[0058] The enhanced visible and infrared signatures are combined only in the channel dimension I c , the formula is as follows:

[0059]

[0060] The encoder of this embodiment includes four convolutional layers, where the kernel size of the first convolutional layer is 1×1, which is used to adjust the number of channels of the input image. The kernel size of the next three convolutional layers is 3×3, which is used to further extract image features. The activation function is the leaky linear rectifier function (LReLU), such as Figure 2 (a) is shown below. There are two FPBlocks, which are designed to extract richer feature information. FPBlock is a variant of Resblock. Each FPBlock has a main stream and two residual streams. The main stream deploys two general 1×1 convolutional layers with LReLU. The two residual streams integrate the original input and the output after the first group of 1×1 convolutional layers to extract strong textures of global features, such as Figure 2 (b) LTEBlock is responsible for further extracting and enhancing the texture features of the image. LTEBlock includes a Sobel operator, a Laplacian operator, two convolutional layers with a kernel size of 1×1, and two LReLUs. The Sobel operator can retain the strong texture of the feature, while the Laplacian operator further extracts the weak texture of the feature. The outputs of the two are then added together to enhance the texture features. The channel dimension difference is then eliminated through a 1×1 convolutional layer and two layers of LReLU, as shown in Figure 1. Figure 2 (c).

[0061] In the image enhancement process, the visible light and infrared images are first converted to the YCrCb color space, and the Y channel information is extracted. Then, the Y channel information of the two spectra is used as the input of LTENet. LTENet extracts and fuses the useful information in the two spectral images through the collaborative work of the encoder, FPBlock, and LTEBlock to generate an enhanced Y channel image. Finally, the enhanced Y channel image is recombined with the Cr and Cb channels of the original image and converted back to the RGB color space to obtain the final enhanced image. Through the fusion and enhancement processing of LTENet, the brightness and details of the image can be effectively highlighted, while reducing the interference of color information, providing more accurate and reliable image data for subsequent PV module monitoring tasks.

[0062] Step S3. Feature fusion: The enhanced visible and infrared features are combined using MCFNet to generate a high-quality fused image containing both spectral information.

[0063] The visible light and infrared image fusion (MCFNet) network consists of a multi-channel attention module (MCABlock) and a decoder (Decoder). The two modules work together to achieve effective image fusion. The specific architecture and process are as follows:

[0064] Multi-channel attention module (MCABlock): A multi-head self-attention mechanism is used to fuse the feature information of different channels. Compared with the traditional channel attention mechanism, multi-head self-attention can focus on the contextual information in each channel, thereby retaining more details and features.

[0065] The multi-head self-attention mechanism extends this process to multiple attention heads, each of which can focus on different aspects of information. This is achieved by dividing the model into multiple subspaces, each of which is responsible for processing different information. Specifically, MCABlock receives the output features from the LTENet network as input and generates Q (Query), K (Key), and V (Value) feature maps respectively through three 1×1 convolutional layers.

[0066] In the multi-head attention process, the Q feature map and the K feature map are used to calculate the attention weights, and the V feature map is weighted summed according to these weights to generate the self-attention feature map. This process can be expressed as:

[0067]

[0068] in, It is I in Q, K, V layer c Input; W i Q , W i K and W i V denote the weight vectors obtained through back-propagation training of Q, K and V layers respectively; d k d v d model It's K i 、V i and the dimensions of the model; h is the head number; It's K i The function softmax(·) represents the softmax function, which assigns a probability value to each output class, indicating the possibility of belonging to each category.

[0069] Decoder: Responsible for decoding the self-attention feature map output by MCABlock into the final fused image. It is usually composed of a series of convolutional layers that gradually extract and refine features to finally generate a fused image. Figure 3As shown, in this embodiment, the decoder includes five 3×3 convolutional layers and one 1×1 convolutional layer. Except for the last layer, the activation functions of all layers are LReLU (Leaky Rectified Linear Unit). The activation function of the last layer is Tanh to output a fused image with the required size and number of channels.

[0070] During the fusion process, MCFNet only processes the Y channel of the infrared image and the visible light image in the YCrCb color space to highlight the brightness information of the image, reduce the interference of color information, and retain sufficient details and structural information. The fused Y channel image is recombined with the Cr and Cb channels of the original image to generate the final fused image. Through color space conversion technology (such as YCrCb to RGB conversion), the fused image is converted to RGB space for easy human eye perception.

[0071] The reconstruction process of MCANet is as follows:

[0072]

[0073] Among them, σ f The self-attention feature map output by MCABlock is passed through the decoder D to obtain the final fused image. It is worth noting that this is only the result of fusion of the Y channel of the visible light image.

[0074] Next, the color difference component information of the Cr and Cb channels is integrated to obtain the fused image in the entire YCrCb color space and converted to the RGB color space to obtain the fused image I f In addition, the design enhances the loss and local perception loss Perform hybrid optimization.

[0075] In order to ensure the effect of LTENet in enhancing the features of visible light and infrared images, the present invention designs an enhancement loss, the purpose of which is to constrain the fused image to retain the important target information of the infrared image. The enhancement loss formula is as follows:

[0076]

[0077] Where H and W are the height and width of the input image, respectively, and ||·|| represents the 1-norm.

[0078] In order to enable the trained model to be used for image fusion, feature enhancement and color balance, the local perception loss formula is designed as follows:

[0079]

[0080] in, represents an auxiliary intensity loss, ensuring that the most favorable intensity distribution is preserved for the fused image. It is a gradient loss that aims to assimilate the additional texture detail information in the input image into the fused output. represents the color consistency loss, which aims to alleviate the color distortion that may occur during the enhancement and fusion process. λ1, λ2, and λ3 are hyperparameters used to strike a balance between the above losses.

[0081] MCFNet dynamically retains the intensity information from the source image through various channels. An additional auxiliary intensity loss is added to the framework to ensure that the optimal intensity distribution of the fused output is preserved, as shown below:

[0082]

[0083] Here, max(·) represents the maximum selection in terms of element value.

[0084] Gradient loss is introduced to force the fused image to contain more texture information, ensuring that the fused image maintains the best intensity distribution while retaining rich texture details. It is defined as follows:

[0085]

[0086] The gradient is calculated using the Sobel operator, where |·| refers to the absolute value operation.

[0087] Since visible light images are easily affected by different colors of light, image enhancement methods are prone to color distortion and over-amplification of the color information of the original light source in the scene. Considering that the Euclidean distance in the RGB space calculates the original numerical difference and cannot accurately reflect the perception law of the human eye, the present invention selects the discrete cosine distance that measures the direction of the color vector to constrain the color distribution of the fused image. In addition, the intensity of the visible image is injected into the fused image through discrete cosine similarity. In general, the designed color consistency loss formula is as follows:

[0088]

[0089] Where C represents the channel number of the image. cos(·,·) illustrates the pixel-by-pixel computation of discrete cosine similarity between the fused image and the original visible image in R, G, B channels. N represents the number of pixels in the image. i is the element of R, G, B. The design of color consistency loss has two purposes. On the one hand, in the fused image, the color distortion caused by image enhancement can be well reduced. On the other hand, the visual information can be aggregated from R, G, B channels into the fused image, so that the fused image can also capture more scene detail information.

[0090] Comparative Experiments: To comprehensively evaluate the effectiveness of the proposed methodology, qualitative and quantitative analysis methods are used to compare with cutting-edge fusion algorithms.

[0091] Five PV module images with different fault types are selected, including leaf cover, simulated bird droppings, dust accumulation, building shading, and broken glass, representing various hot spot defects that may appear on PV modules. By intuitively displaying the processing results of these images under different fusion methods, the versatility and effectiveness of various methods in processing different defects can be evaluated. Figure 4 As shown, different defects produce different results. For example, in the presence of shadow-masking defects, it becomes challenging to identify shadow features in visible light images. In the case of dust defects, GTF and DenseFuse fail to fully preserve the texture details of visible light images. FusionGAN, U2Fusion, SDNet, GANMcC, and SwinFuion weaken the texture information of visible and infrared targets accordingly. In the case of broken glass defects, the infrared image hardly shows the fracture defect. In contrast, the proposed method is able to better combine the two modalities, especially performing well in processing glass texture details.

[0092] A detailed comparison is conducted on the MSPVS dataset between the newly introduced LTA-FNet and seven pre-existing influential methods in terms of six indicators: visual information fidelity, standard deviation, quality index, information entropy, structural similarity, and mutual information. Figure 5 Quantitative results of six complementary metrics on 25 image pairs are given in . It is worth noting that the proposed method shows significant superiority in all six metrics. The best mutual information metric means that LTA-FNet transfers the most information from the source images and can adjust the fused image according to the lighting conditions. In addition, the proposed method obtains the highest standard deviation and visual information fidelity, indicating that the fused image has high contrast and satisfactory visual effects. Moreover, LTA-FNet shows the best quality indicators, that is, more edge information is retained in the fusion result, which is due to the proposed local object perception module LTEBlock. The experimental results highlight the superiority of the proposed method on the diagnosis of hot spots of photovoltaic modules in the produced dataset.

[0093] Ablation experiment: In order to verify the effectiveness of important modules in LTA-FNet, ablation experiments were performed on these components using the MSPVS dataset. The experimental results are shown in the figure. Figure 6As shown in the figure. It can be seen that the fusion result without the LTEBlock module obviously lacks local features and background details. The fused image is affected by spectral pollution and texture blur in daytime scenes. In addition, the fusion result cannot maintain the contrast of thermal targets, nor can it compensate the background area with the detail information in the night infrared image. On the contrary, the model combined with LTEBlock can adaptively achieve contrast preservation and texture preservation according to the lighting conditions.

[0094] Ablation experiments are also performed to examine the impact of MCABlock, another key element in the network that enhances and fuses local features and background details. The use of MCABlock in the feature extractor is omitted, resulting in the fusion of visible and infrared image features without extracting multi-channel features. From the results, it can be seen that the fused image retains the texture details of the visible image in the daytime scene, but weakens the salient objects. In addition, the fused image is similar to the infrared image, but ignores some detail information in the visible image at night. In contrast, the proposed network simultaneously maintains the intensity distribution of salient objects and the texture details of the background area. Therefore, due to the unique design of including LTEBlock and MCABlock, the proposed LTA-FNet shows proficient feature and detail fusion performance under various challenging conditions.

[0095] Downstream diagnosis: The results of image fusion are usually applied to downstream tasks such as object detection. The performance of object detection well reflects the semantic information fused into the image. In order to verify the performance of LTA-FNet in downstream tasks, object detection experiments were conducted. The most advanced object detection network YOLOX was selected as the detector, and 60 pairs of images were randomly selected from the MSRS dataset as the test set, which includes almost all urban scenes. The objective evaluation results of object detection are shown in Table 1, which shows the mean average precision (mAP) of different methods at different IoU thresholds. mAP is used to measure detection performance, and the higher the mAP, the better the detection performance.

[0096] Table 1: Average precision of detection on 60 pairs of images in MSRS

[0097]

[0098] The experimental environment of this embodiment: system Windows 11, equipped with NVIDIA GeForce RTX 3090 GPU and 3.2GHz Intel(R) Xeon(R) Gold 6330 CPU, deep learning framework Pytorch2.0.1.

[0099] Through ablation experiments and downstream diagnosis experiments, the researchers verified the effectiveness of the LTEBlock and MCABlock modules in LTA-FNet, and demonstrated the superior performance of LTA-FNet in image fusion and target detection tasks. These experimental results not only prove the feasibility and practicality of LTA-FNet, but also provide important references for subsequent research and applications.

Claims

1. A photovoltaic visible light and infrared image fusion method based on target perception, characterized in that: The following steps are involved: Step S1: construct a multi-spectral photovoltaic module scene (MSPVS) dataset, which consists of at least 1284 pairs of aligned visible light and infrared spectrum images, and each pair of images has spatial consistency and a specific spatial resolution; Step S2: Image enhancement, using LTENet network to fuse the infrared image and the visible light information in the Y channel of the YCrCb color space. The LTENet network includes an encoder, two parallel feature perception modules and two parallel local target enhancement modules, wherein: The encoder is used to receive inputs from infrared and visible light images and extract mixed features containing information from the two domains respectively; Feature perception module, used to identify and extract specific area features of visible light and infrared images from mixed features; The local target enhancement module is used to adaptively adjust the image features to ensure that the contrast and texture information of the original image are maintained during the fusion process, and to generate an enhanced Y channel image; Step S3: Feature fusion, using MCFNet to combine the enhanced visible light and infrared features to generate a fused image containing two spectral information, the MCFNet includes a multi-channel attention module and a decoder, wherein: Multi-channel attention module, which uses a multi-head self-attention mechanism to fuse feature information from different channels; The decoder is responsible for decoding the self-attention feature map output by the multi-channel attention module into the final fused image.

2. The photovoltaic visible light and infrared image fusion method based on target perception according to claim 1 is characterized in that: By converting visible light and infrared images into YCrCb color space, extracting Y channel information as the input of LTENet, generating enhanced Y channel image through the collaborative work of encoder, FPBlock and LTEBlock, and combining the enhanced visible light and infrared features using MCFNet to generate a high-quality fused image containing two spectral information; then recombining with the Cr and Cb channels of the original image and converting back to RGB color space to obtain the final enhanced image.

3. The photovoltaic visible light and infrared image fusion method based on target perception according to claim 1 is characterized in that: In step S1, the MSPVS dataset consists of at least 1284 pairs of aligned images of visible light and infrared spectra, each pair of images having a spatial resolution of 480×640 pixels, which are captured by an image capture device.

4. The photovoltaic visible light and infrared image fusion method based on target perception according to claim 1 is characterized in that: In step S2, the encoder of the LTENet network includes at least four convolutional layers, wherein the kernel size of the first convolutional layer is 1×1, which is used to adjust the number of channels of the input image, and the kernel size of the next three convolutional layers is 3×3, which is used to further extract image features, and the activation function is LReLU.

5. The photovoltaic visible light and infrared image fusion method based on target perception according to claim 4 is characterized in that: Each of the feature perception modules includes a mainstream and two residual streams. The mainstream deploys two general 1×1 convolutional layers with LReLU. The two residual streams respectively integrate the original input and the output after the first group of 1×1 convolutional layers to extract the strong texture of the global features.

6. The photovoltaic visible light and infrared image fusion method based on target perception according to claim 1 is characterized in that: The local target enhancement module adaptively adjusts the image features to ensure that the contrast and texture information of the original image are maintained during the fusion process. The formula is as follows: Among them, conv(·) represents several convolution operations, concat(·) represents cascade, and ⊕ refers to the sum of elements. is the Sobel operator is the Laplace operator, which is used to amplify the fused features. and They represent the enhanced visible light features and infrared image features respectively.

7. The photovoltaic visible light and infrared image fusion method based on target perception according to claim 1 is characterized in that: In step S3, the multi-channel attention module generates Q, K and V feature maps respectively through three 1×1 convolutional layers, wherein the Q feature map and the K feature map are used to calculate the attention weights, and the V feature map is weighted summed according to these weights to generate a self-attention feature map.

8. The photovoltaic visible light and infrared image fusion method based on target perception according to claim 1 is characterized in that: In step S3, the decoder comprises a plurality of 3×3 convolutional layers and at least one 1×1 convolutional layer, and the activation functions of all layers except the last layer are LReLU, and the activation function of the last layer is Tanh, so as to output a fused image with a desired size and number of channels.

9. The photovoltaic visible light and infrared image fusion method based on target perception according to any one of claims 1 to 8, characterized in that: It also includes enhanced loss design and local perceptual loss hybrid optimization, wherein the enhanced loss design is used to constrain the fused image to retain important target information of the infrared image, and the local perceptual loss hybrid optimization includes auxiliary intensity loss, gradient loss and color consistency loss, which are used to achieve feature enhancement and color balance.

10. The photovoltaic visible light and infrared image fusion method based on target perception according to claim 9, characterized in that: The color consistency loss uses discrete cosine similarity to measure the similarity of color distribution between the fused image and the original visible image to reduce the color distortion caused by image enhancement and inject the intensity of the visible image into the fused image.

Images