An image multi-scale fusion method, device, equipment, medium and product

Abstract

This application discloses an image multi-scale fusion method, apparatus, device, medium, and product, relating to the field of computer vision. The method includes: extracting image information data from acquired image data using a low-pass filter; fusing low-frequency information based on a preset fusion ratio to obtain fused low-frequency image information; denoising high-frequency information using an FFT Transformer, extracting multi-scale depth features using a deep neural network, merging image channels using the L1 norm, determining the fusion weights for visible light and infrared images in each region, then weighted fusion of high-frequency information to obtain fused high-frequency image information; and determining fused visible light and infrared dual-source image information based on the fused low-frequency and high-frequency image information. This application aims to improve image fusion performance.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a method, apparatus, device, medium, and product for multi-scale image fusion. Background Technology

[0002] The reliability and efficiency of photovoltaic (PV) systems are crucial to the global energy transition. With the rapid development of PV technology, defect detection of PV modules has become a key aspect of ensuring the long-term, efficient operation of PV systems. Traditional PV defect detection methods largely rely on manual inspection or image processing technology based on single sensors. However, as the complexity of PV panel surface defects increases, single-type image sensors often cannot comprehensively and effectively detect all defects, especially under different environmental conditions (such as rain, low light, or high temperatures).

[0003] Visible light imaging is widely used due to its mature technology, low acquisition cost, readily available equipment, and ease of use. Visible light images offer rich colors, providing complete color information perceptible to the human eye, making the image more intuitive and easier to understand. It also effectively reveals details of surface defects such as cracks, stains, and scratches. However, image quality can deteriorate significantly under low light, strong light, or shadow conditions, leading to inaccurate detection results. Infrared imaging can capture the temperature distribution on the surface of photovoltaic modules, especially at electrical connections or hot spots, revealing localized overheating caused by uneven current, aging, or poor contact. Infrared images rely on external lighting, enabling effective detection in low light or at night, making them highly adaptable. However, the resolution of infrared images is generally lower than that of visible light images, potentially lacking sufficient detail and making it difficult to accurately identify small-scale surface defects. Infrared images also do not provide intuitive color information.

[0004] In infrared imaging or infrared sensor data acquisition, in addition to thermal radiation information, various high-frequency noises are often mixed into the signal. These noises may originate from the electronic components inside the sensor (such as electronic noise, shot noise, and quantization noise), or they may be introduced by environmental interference (such as rapid fluctuations in background heat sources, mechanical vibration, or electromagnetic interference). Since these noises usually manifest as short-term, rapidly changing high-frequency components, it is difficult to distinguish them from low-frequency or mid-frequency thermal information if the raw time-domain signal is analyzed and processed directly.

[0005] Furthermore, although traditional image processing and fusion methods have achieved good fusion performance, they also have some drawbacks. First, traditional feature extraction and fusion strategies usually rely on manually designed features and lack adaptability. Second, traditional methods suffer from performance degradation under conditions such as noise, illumination changes, and viewpoint changes.

[0006] Therefore, improving image fusion performance is of paramount importance. Summary of the Invention

[0007] The purpose of this application is to provide an image multi-scale fusion method, apparatus, device, medium, and product that can improve image fusion performance.

[0008] To achieve the above objectives, this application provides the following solution:

[0009] Firstly, this application provides an image multi-scale fusion method, including:

[0010] Acquire image information data; the image information data includes: visible light images and infrared images;

[0011] Image information data is extracted using a low-pass filter to obtain extracted information; the extracted information includes: high-frequency information and low-frequency information.

[0012] Based on a preset fusion ratio, the low-frequency information is fused to obtain the fused image information of the low-frequency part.

[0013] An FFT Transformer is used to denoise the high-frequency information, and a deep neural network is used to extract multi-scale deep features to obtain the high-frequency feature activation value corresponding to each pixel.

[0014] Based on all high-frequency feature activation values, the L1 norm is used to merge the image channels, and the fusion weights of the visible light image and the infrared light image in each region are determined.

[0015] Based on the fusion weight, the high-frequency information is weighted and fused to obtain the fused image information of the high-frequency part;

[0016] Based on the fused image information of the low-frequency component and the fused image information of the high-frequency component, the fused image information of the visible light and infrared dual light sources is determined.

[0017] In one embodiment, the low-pass filter is a Gaussian filter; the Gaussian filter is a linear filter based on weights defined by the Gaussian function, which is updated by convolution operation and weighted average.

[0018] The mathematical expression for performing a convolution operation is:

[0019] ;

[0020] in, The image pixel values ​​for which convolution operations are performed; In pixel coordinates Image pixel values ​​at that location; These are the weights of the Gaussian filter; The x-coordinate is the pixel coordinate. The vertical coordinate is the pixel coordinate. and All of these are iterative variables in the Gaussian filtering process; In order to be in The pixel value at that location.

[0021] In one embodiment, an FFT Transformer is used to denoise the high-frequency information, and a deep neural network is used to extract multi-scale deep features to obtain the high-frequency feature activation value corresponding to each pixel. Specifically, this includes:

[0022] The FFT Transformer is used to denoise the high-frequency information, resulting in denoised high-frequency feature information. The mathematical expression for the FFT Transformer denoising process is as follows:

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] in, This refers to the high-frequency component information corresponding to the infrared image; This refers to the denoised high-frequency feature information corresponding to the visible light image; This refers to the denoised high-frequency feature information corresponding to the infrared image; These are the attention weights of visible light features after self-attention processing; The attention weights are the infrared light features after self-attention operation; It is a norm; It is a feedforward neural network; This is a self-attention mechanism based on Fast Fourier Transform; This is a self-attention mechanism based on the fast inverse Fourier transform; For Fast Fourier Transform; This is the inverse fast Fourier transform;

[0028] A deep neural network is used to extract multi-scale deep features from the denoised high-frequency feature information to obtain the high-frequency feature activation value corresponding to each pixel. The deep neural network is trained using a stochastic gradient algorithm with the goal of minimizing the loss function.

[0029] The backbone of the deep neural network uses ResNet34; the structure of ResNet34 includes: an initial convolutional module and four residual stages; each residual stage contains a set number of residual blocks, and each residual block consists of two 3×3 convolutional layers, batch normalization and the ReLU activation function.

[0030] In one embodiment, based on all high-frequency feature activation values, the L1 norm is used to merge the image channels, and the fusion weights of the visible light image and the infrared light image corresponding to each region are determined, specifically including:

[0031] Based on all high-frequency feature activation values, the L1 norm is used to merge the image channels to obtain pixel activation values;

[0032] Based on the local block averaging operator, the fusion weights of the visible light image and the infrared light image in each region are determined according to the pixel activation values; the mathematical expression for the fusion weights is:

[0033] ;

[0034] ;

[0035] in, The fusion weights for each region of the visible light image; The fusion weights for each region of the infrared image; The channels of the visible light image are merged using the L1 norm to obtain pixel activation values; The infrared image channels are merged using the L1 norm to obtain pixel activation values.

[0036] In one embodiment, the mathematical expression corresponding to the fused image information of the high-frequency component is:

[0037] ;

[0038] in, This refers to the fused image information of the high-frequency components; The fusion weights for each region of the visible light image; This refers to the high-frequency portion of the visible light image; The fusion weights for each region of the infrared image; This refers to the high-frequency information corresponding to the infrared image.

[0039] In one embodiment, the mathematical expression corresponding to the image information fused from both visible and infrared light sources is:

[0040] ;

[0041] in, This involves fusing image information from both visible and infrared light sources. This refers to the fused image information for the low-frequency components; This refers to the fused image information of the high-frequency components; The proportion of low-frequency information fusion; This refers to the proportion of high-frequency information fusion.

[0042] Secondly, this application provides an image multi-scale fusion apparatus, comprising:

[0043] An image information data acquisition module is used to acquire image information data; the image information data includes: visible light images and infrared images;

[0044] An image extraction module is used to extract image information from the image information data using a low-pass filter to obtain extracted information; the extracted information includes: high-frequency information and low-frequency information.

[0045] The low-frequency fusion module is used to fuse low-frequency information based on a preset fusion ratio to obtain fused image information of the low-frequency part.

[0046] The high-frequency extraction module is used to denoise the high-frequency information using an FFT Transformer and to extract multi-scale deep features using a deep neural network to obtain the high-frequency feature activation value corresponding to each pixel.

[0047] The fusion weight determination module is used to merge the channels of the image based on all high-frequency feature activation values ​​using the L1 norm, and to determine the fusion weights of the visible light image and the infrared light image in each region.

[0048] The high-frequency fusion module is used to perform weighted fusion of high-frequency information based on fusion weights to obtain fused image information of the high-frequency components.

[0049] The image fusion module is used to determine the fused image information of visible light and infrared dual light sources based on the fused image information of the low-frequency part and the fused image information of the high-frequency part.

[0050] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image multi-scale fusion method described above.

[0051] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image multi-scale fusion method described above.

[0052] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the image multi-scale fusion method described above.

[0053] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0054] This application provides an image multi-scale fusion method, apparatus, device, medium, and product. It employs a low-pass filter to extract image information from acquired image data; fuses low-frequency information; uses an FFT Transformer to denoise high-frequency information; extracts multi-scale depth features using a deep neural network; merges image channels using the L1 norm; and determines the fusion weights for visible light and infrared images in each region. Then, it weights and fuses the high-frequency information to obtain the fused high-frequency image information. Based on the fused low-frequency and high-frequency image information, it determines the fused image information of visible light and infrared dual light sources. This application improves image fusion performance by directly fusing low-frequency information, removing high-frequency noise while preserving background information such as the environment. It then uses a deep neural network to calculate fusion weights for the prominent high-frequency features (i.e., high-frequency information) before weighted fusion, thereby determining the fused image information of visible light and infrared dual light sources. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of an image multi-scale fusion method;

[0057] Figure 2 This is a diagram of the ResNet network model structure.

[0058] Figure 3 Here is a detailed structural diagram of the entire fusion method;

[0059] Figure 4 This is a schematic diagram illustrating the effect of multi-scale image fusion in photovoltaic images;

[0060] Figure 5 This is a structural diagram of an image multi-scale fusion device;

[0061] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] In one exemplary embodiment, such as Figure 1 As shown, an image multi-scale fusion method is provided, including:

[0065] Step 100: Acquire image information data. Image information data includes: visible light images and infrared images.

[0066] Step 200: Use a low-pass filter to extract image information from the image data to obtain extracted information. The extracted information includes high-frequency information and low-frequency information.

[0067] Step 300: Based on the preset fusion ratio, fuse the low-frequency information to obtain the fused image information of the low-frequency part.

[0068] Step 400: Use FFT Transformer to denoise the high-frequency information and use deep neural network to extract multi-scale deep features to obtain the high-frequency feature activation value corresponding to each pixel.

[0069] Step 500: Based on all high-frequency feature activation values, merge the channels of the image using the L1-norm and determine the fusion weights of the visible light image and the infrared light image in each region.

[0070] Step 600: Based on the fusion weight, the high-frequency information is weighted and fused to obtain the fused image information of the high-frequency part.

[0071] Step 700: Determine the fused image information of visible light and infrared dual light sources based on the fused image information of the low-frequency part and the fused image information of the high-frequency part.

[0072] In one embodiment, the low-pass filter is a Gaussian filter; the Gaussian filter is a linear filter based on weights defined by the Gaussian function, which is updated through convolution operation and weighted average. The mathematical expression corresponding to the convolution operation is:

[0073] .

[0074] in, The image pixel values ​​for which convolution operations are performed; In pixel coordinates Image pixel values ​​at that location; These are the weights of the Gaussian filter; The x-coordinate is the pixel coordinate. The vertical coordinate is the pixel coordinate. and All of these are iterative variables in the Gaussian filtering process; In order to be in The pixel value at that location.

[0075] An FFT Transformer is used to denoise the high-frequency information, and a deep neural network is used to extract multi-scale deep features to obtain the high-frequency feature activation values ​​corresponding to each pixel, specifically including:

[0076] The FFT Transformer is used to denoise the high-frequency information, resulting in denoised high-frequency feature information. The mathematical expression for the FFT Transformer denoising process is as follows:

[0077] .

[0078] .

[0079] .

[0080] .

[0081] in, This refers to the high-frequency component information corresponding to the infrared image; This refers to the denoised high-frequency feature information corresponding to the visible light image; This refers to the denoised high-frequency feature information corresponding to the infrared image; These are the attention weights of visible light features after self-attention processing; The attention weights are the infrared light features after self-attention operation; It is a norm; It is a feedforward neural network; This is a self-attention mechanism based on Fast Fourier Transform; This is a self-attention mechanism based on the fast inverse Fourier transform; For Fast Fourier Transform; This is the inverse fast Fourier transform.

[0082] A deep neural network is used to extract multi-scale deep features from the denoised high-frequency feature information to obtain the high-frequency feature activation value corresponding to each pixel. The deep neural network is trained using the stochastic gradient algorithm with the goal of minimizing the loss function.

[0083] The backbone of the deep neural network uses ResNet34; the structure of ResNet34 includes: an initial convolutional module and four residual stages; each residual stage contains a set number of residual blocks, and each residual block consists of two 3×3 convolutional layers, batch normalization and the ReLU activation function.

[0084] In one embodiment, based on all high-frequency feature activation values, the L1 norm is used to merge the image channels, and the fusion weights of the visible light image and the infrared light image corresponding to each region are determined, specifically including:

[0085] Based on all high-frequency feature activation values, the L1 norm is used to merge the image channels to obtain pixel activation values.

[0086] Based on the local block averaging operator, the fusion weights of the visible light image and the infrared light image in each region are determined according to the pixel activation values; the mathematical expression for the fusion weights is:

[0087] .

[0088] .

[0089] in, The fusion weights for each region of the visible light image; The fusion weights for each region of the infrared image; The channels of the visible light image are merged using the L1 norm to obtain pixel activation values; The infrared image channels are merged using the L1 norm to obtain pixel activation values.

[0090] The mathematical expression corresponding to the fused image information in the high-frequency component is:

[0091] .

[0092] in, This refers to the fused image information of the high-frequency components; The fusion weights for each region of the visible light image; This refers to the high-frequency portion of the visible light image; The fusion weights for each region of the infrared image; This refers to the high-frequency information corresponding to the infrared image.

[0093] The mathematical expression corresponding to the image information fused from visible light and infrared dual light sources is:

[0094] .

[0095] in, This involves fusing image information from both visible and infrared light sources. This refers to the fused image information for the low-frequency components; This refers to the fused image information of the high-frequency components; The proportion of low-frequency information fusion; This refers to the proportion of high-frequency information fusion.

[0096] Dual-light fusion, combining visible light and infrared images, is an innovative technology that enhances detection accuracy and efficiency. It leverages the advantages of both methods, overcoming the limitations of single image modes and providing more comprehensive defect detection capabilities. In photovoltaic defect detection, visible light images offer rich surface detail information, while infrared images provide crucial temperature distribution information, particularly advantageous for detecting defects caused by electrical issues. By fusing these two images, the detection system can not only identify surface defects but also promptly detect potential hotspots, significantly improving detection accuracy and efficiency.

[0097] Image fusion typically consists of three key steps: feature extraction, fusion strategy, and image reconstruction. Feature extraction, the first step, aims to extract useful feature information from the input images. These features typically include edges, textures, colors, shapes, and higher-level semantic information. Feature extraction can be performed in various ways, including using traditional image processing algorithms such as the Sobel operator and Canny edge detection, or deep learning methods. The goal of the fusion strategy is to effectively combine the features of multiple input images to obtain a fused image with higher information content. Fusion strategies can be selected based on task requirements; common fusion strategies include pixel-level fusion, feature-level fusion, and end-to-end fusion based on deep learning. Image reconstruction, the final step, aims to transform the fused feature information into the final fused image. The quality of this step determines the visual effect and practical application performance of the final image.

[0098] Deep learning methods (deep neural networks) can automatically extract effective low-level and high-level features from raw data or automatically select the most suitable fusion strategy based on the task objective (such as classification, detection, or segmentation) through an end-to-end learning process.

[0099] The Fast Fourier Transform (FFT) provides an efficient method for frequency domain analysis: it decomposes a time-domain signal into several different frequency components and provides the amplitude or energy distribution in each frequency band. Specifically, the acquired infrared time-domain waveform is first preprocessed using a window function (such as a Hanning or Hamming window) to reduce spectral leakage; then, the FFT is performed on the windowed data to obtain the spectrum. At this point, high-frequency noise will form a significant and concentrated energy spike in the high-frequency band of the spectrum, while the effective infrared thermal information is concentrated in the lower frequency range.

[0100] Once the frequency range of the noise is identified, digital filters can be designed to suppress or remove these high-frequency components. Common filtering schemes include low-pass filters (directly attenuating signals above a certain cutoff frequency), band-stop filters (suppressing signals within a specific high-frequency band), and even adaptive filters (adjusting filter coefficients based on the real-time noise power spectrum). After applying the filter to the frequency domain signal, the inverse fast fourier transform (IFFT) is used to convert the processed frequency domain result back to the time domain, resulting in a significantly denoised infrared signal. Through this process of "frequency domain analysis → noise identification → frequency domain filtering → time domain reconstruction," not only can high-frequency noise be effectively suppressed, but the true thermal radiation information in infrared light can also be preserved to the greatest extent, thereby significantly improving the signal-to-noise ratio and accuracy of infrared imaging or temperature measurement.

[0101] Deep learning infuses fused images with rich semantic information. Two common approaches utilize deep learning: one involves partially incorporating deep convolutional networks (CNNs). The earliest work on this, proposed by Yu Liu et al. in 2017, trained the network using blurred background and foreground images, resulting in a binarized weight map. During testing, the original image was combined with the weight map to obtain a fused multifocal image. However, this network was a classification network and not suitable for fusion of infrared and visible light images. Later researchers gradually utilized deep neural networks, which performed well in classification tasks, to extract image features. The other approach is an end-to-end convolutional neural network fusion model for infrared and visible light images. Among these, autoencoder-based fusion methods typically consist of three parts: an encoder, a decoder, and a fusion module. DenseFuse is generally considered the first method to use a deep learning model for fusing infrared and visible light images. In the encoder, this method concatenates the feature maps extracted from each layer to the input of the next layer, increasing information flow and making the network easier to train.

[0102] Feature extraction and processing tasks play a crucial role in image fusion, and different features and processing methods directly affect fusion performance. In contrast, most deep learning-based methods directly use deep features without feature extraction and noise reduction. In some cases, this leads to a decrease in fusion performance. The main innovation of this application lies in directly fusing low-frequency features, removing high-frequency noise while preserving background information such as the environment, and then fusing the prominent high-frequency features of the image by calculating fusion weights through a deep neural network.

[0103] Specifically, in practical applications, the method mentioned in this application includes the following steps:

[0104] Step S1: Construct a low-pass filter to extract the low-frequency and high-frequency components of the image; fuse the low-frequency components of the two images, and denoise the high-frequency components using an FFT Transformer before fusing them.

[0105] Step S2: Build a pre-trained deep neural network on the ImageNet dataset to extract high-frequency features.

[0106] Step S3: Calculate the high-frequency fusion weights and reconstruct the fused image.

[0107] Figure 3 This is a detailed structural diagram of the entire fusion method. Step S1 is described below:

[0108] First, a low-pass filter is constructed to extract low-frequency information from the image. A Gaussian filter is used, a linear filter commonly used in image and signal processing for smoothing or denoising. Based on the mathematical principle of the Gaussian distribution, it reduces noise or detail by weighted averaging of pixel values ​​in the image, making the image smoother. The core idea of ​​a Gaussian filter is to use a Gaussian function to define the filter weights. The Gaussian function is a bell-shaped curve, with the largest weight at the center and gradually decreasing weights away from the center. This weight distribution ensures that the filter's influence on surrounding pixels gradually decreases, thus achieving a smoothing effect. The weight distribution of the Gaussian filter is given by the Gaussian function. The standard form of a two-dimensional Gaussian function is:

[0109] .

[0110] in, It is a Gaussian function; The variance is a Gaussian distribution.

[0111] In image processing, Gaussian filters are typically applied to images through convolution operations. Let the pixel values ​​of the image be denoted as , i.e., at pixel coordinates . The image pixel value at that location is The weights of the Gaussian filter are Then the convolution operation can be represented as:

[0112] .

[0113] Each pixel in the image is updated based on the weighted average of its surrounding pixels, thus achieving a smoothing effect. This applies to visible light images. With infrared images The low-frequency and high-frequency information of the image is clearly obtained after passing through a Gaussian filter:

[0114] 。

[0115] .

[0116] in, This is the extracted information corresponding to the visible light image after Gaussian filter processing; This refers to the extracted information corresponding to the infrared image after Gaussian filter processing; This refers to the low-frequency information corresponding to the visible light image; This refers to the high-frequency portion of the visible light image; This refers to the low-frequency portion of the infrared image; This refers to the high-frequency information corresponding to the infrared image.

[0117] The low-frequency information extracted from the visible light image is fused with the low-frequency information extracted from the infrared image using a preset fusion ratio, i.e., the two parameters. as well as By controlling the fusion ratio separately, the fused image information of the low-frequency component is obtained. .

[0118] .

[0119] The high-frequency information is processed by using a Fast Fourier Transform (FFT) Transformer to remove high-frequency noise, resulting in denoised high-frequency feature information.

[0120] .

[0121] .

[0122] .

[0123] .

[0124] Step S2, the ResNet network model structure diagram is as follows: Figure 2As shown, the specific method is as follows:

[0125] For high-frequency feature information in images (such as edges, textures, etc.), deep neural networks are used to further learn image information. The backbone of the deep neural network adopts ResNet34. The structure of ResNet34 can be divided into an initial convolutional module and four residual stages, each containing a set number of residual blocks. Each residual block consists of two 3×3 convolutional layers, batch normalization, and the ReLU (Rectified Linear Unit) activation function. The input is directly added to the convolutional output through a shortcut connection to alleviate the gradient vanishing problem caused by network deepening.

[0126] Specifically, the input image is H(224)×W(224)×C(3), where H and W are the height and width of the image, respectively, and C is the number of image channels. First, it goes through a 7×7 large receptive field convolutional layer (stride 2, number of output channels 64), and the feature map size becomes 112×112×64. Then, it is downsampled through a 3×3 max pooling layer (stride 2), and the size is reduced to 56×56×64. After entering the first residual stage (conv2_x), there are 3 residual blocks, each with 64 input and output channels and a spatial resolution of 56×56. The second residual stage (conv3_x) contains 4 residual blocks, where the first block uses a convolution with a stride of 2 for downsampling, reducing the feature map size to 28×28 while expanding the number of channels to 128. The remaining blocks maintain a size of 28×28×128. The third residual stage (conv4_x) consists of 6 residual blocks, each with 256 channels and a feature map size of 14×14. The fourth residual stage (conv5_x) contains 3 residual blocks with 512 channels, further reducing the feature map size to 7×7×512. Finally, global average pooling is used to pool the 7×7×512 feature maps into a 1×1×512 feature vector for subsequent classification or feature fusion tasks. Through the process of "initial large convolution → max pooling → multi-stage residual block → global average pooling", ResNet34 can not only efficiently extract image features at different scales and levels, but also use residual connections to maintain information flow and suppress gradient vanishing, thereby better learning high-frequency details in images.

[0127] Deep neural networks are trained using a stochastic gradient descent algorithm with a set loss objective. The relevant loss function is defined as follows:

[0128] 。

[0129] in, The loss function; This refers to the number of image channels; For serial numbers; True labels in one-hot format; These are predicted values.

[0130] Will and The input is fed into a deep neural network, and the activation values ​​of each ResNet residual block before the max-pooling layer are used to extract multi-scale depth features, including visible light features. and infrared light characteristics .

[0131] .

[0132] 。

[0133] Step S3, the specific method is as follows:

[0134] For the high-frequency feature activation values ​​extracted in S2, the image channels are merged using L1-Norm (L1 norm) to obtain the absolute value of the high-frequency feature activation values ​​at each pixel. as well as :

[0135] 。

[0136] 。

[0137] in, This indicates that the L1-Norm is calculated on the vector of high-frequency feature activation values ​​along the 2-dimensional subscript. Due to the local invariance of images, the weights should be calculated based on local regions of the image content. Different regions in an image have different importance, and the algorithm weights and fuses them according to the feature responses of these regions, so that the fused image can retain the key regional information of each image. Therefore, a local block-based averaging operator is used to calculate the final feature fusion weights, using the weighted sum of the surrounding 3×3 neighborhood, representing the response of the two feature maps at that location. The general expression is as follows:

[0138] .

[0139] in, This represents the pixel activation value after L1-Norm processing; These are the iterative characteristic values ​​in the averaging process; These are the iteration coefficients; It is a general expression for the absolute value of the high-frequency feature activation value at each pixel.

[0140] Then, the fusion weights of the visible light image and the infrared image in each region are calculated separately. .

[0141] .

[0142] .

[0143] For high-frequency feature fusion of images, these weights are used to perform weighted fusion of the original images. Specifically, each pixel of each image is multiplied by its corresponding weight.

[0144] .

[0145] Finally, use two parameters. as well as By controlling the fusion ratio of the low-frequency and high-frequency fused image information respectively, the final visible light and infrared dual-source fused image is obtained. :

[0146] .

[0147] Because ImageNet primarily uses natural scene images, a dataset of visible light and infrared dual-source images of photovoltaic panels was constructed using drone footage. Finally, the model was fine-tuned to fully adapt to the processing scenarios of photovoltaic panels. Figure 4 As shown.

[0148] Unlike previous methods that simply superimpose or weightedly fuse low-frequency and high-frequency components of visible and infrared images, this application innovatively introduces a learnable Transformer module based on Fast Fourier Transform (FFT) to refine the processing of high-frequency noise in both types of images. Specifically, in addition to the multi-head self-attention and feedforward networks of the standard Transformer encoder, this module integrates Fast Fourier Transform and Inverse Fourier Transform operators: First, a Fourier transform is performed on the input multi-channel feature map to project spatial domain features into the frequency domain, facilitating the capture and differentiation of different frequency components; then, a frequency domain filter with learnable parameters dynamically suppresses high-frequency noise while retaining mid- and low-frequency information helpful for target detection or segmentation; finally, an Inverse Fourier Transform is performed to map the cleaned frequency domain features back to the spatial domain, fuse them with the original features, and feed them into the subsequent fusion network. By setting specific optimization objectives, this method can automatically learn the optimal frequency domain filtering strategy during training, thereby significantly reducing image noise while maintaining rich detail. The results show that the module not only enhances the complementarity of visible and infrared light features, but also improves target recognition and segmentation performance in complex environments.

[0149] In one exemplary embodiment, such as Figure 5 As shown, an image multi-scale fusion device is provided, comprising:

[0150] The image information data acquisition module is used to acquire image information data, including visible light images and infrared images.

[0151] The image extraction module is used to extract image information from the image information data using a low-pass filter to obtain extracted information; the extracted information includes high-frequency information and low-frequency information.

[0152] The low-frequency fusion module is used to fuse low-frequency information based on a preset fusion ratio to obtain fused image information of the low-frequency part.

[0153] The high-frequency extraction module is used to denoise the high-frequency information using an FFT Transformer and to extract multi-scale deep features using a deep neural network to obtain the high-frequency feature activation value corresponding to each pixel.

[0154] The fusion weight determination module is used to merge the channels of the image based on all high-frequency feature activation values ​​using the L1-norm, and to determine the fusion weights of the visible light image and the infrared light image in each region.

[0155] The high-frequency fusion module is used to weight and fuse high-frequency information based on fusion weights to obtain fused image information of the high-frequency components.

[0156] The image fusion module is used to determine the fused image information of visible light and infrared dual light sources based on the fused image information of the low-frequency part and the fused image information of the high-frequency part.

[0157] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores multi-scale image fusion data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an image multi-scale fusion method.

[0158] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0160] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0161] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0164] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-scale image fusion method, characterized in that, include: Acquire image information data; The image information data includes: visible light images and infrared images; Image information data is extracted using a low-pass filter to obtain extracted information; the extracted information includes: high-frequency information and low-frequency information. Based on a preset fusion ratio, the low-frequency information is fused to obtain the fused image information of the low-frequency part. An FFT Transformer is used to denoise the high-frequency information, and a deep neural network is used to extract multi-scale deep features to obtain the high-frequency feature activation value corresponding to each pixel. Based on all high-frequency feature activation values, the L1 norm is used to merge the image channels, and the fusion weights of the visible light image and the infrared light image in each region are determined. Based on the fusion weight, the high-frequency information is weighted and fused to obtain the fused image information of the high-frequency part; Based on the fused image information of the low-frequency component and the fused image information of the high-frequency component, the fused image information of the visible light and infrared dual light sources is determined. The backbone of the deep neural network uses ResNet34; ResNet34 can not only extract image features at different scales and levels, but also use residual connections to maintain information flow and suppress gradient vanishing, thereby learning high-frequency details in the image. Based on all high-frequency feature activation values, the L1 norm is used to merge the image channels, and the fusion weights for the visible light image and the infrared light image in each region are determined, specifically including: Based on all high-frequency feature activation values, the L1 norm is used to merge the image channels to obtain pixel activation values; Based on the local block averaging operator, the fusion weights of the visible light image and the infrared light image in each region are determined according to the pixel activation values; the mathematical expression for the fusion weights is: ； ； in, The fusion weights for each region of the visible light image; The fusion weights for each region of the infrared image; The channels of the visible light image are merged using the L1 norm to obtain pixel activation values; The infrared image channels are merged using the L1 norm to obtain pixel activation values; The FFT Transformer refines high-frequency noise in two types of images. In addition to the multi-head self-attention and feedforward network of the standard Transformer encoder, it integrates Fast Fourier Transform and Inverse Fourier Transform operators: First, it performs a Fourier transform on the input multi-channel feature map, projecting the spatial domain features into the frequency domain to capture and distinguish different frequency components; then, it uses a frequency domain filter with learnable parameters to dynamically suppress high-frequency noise and retain mid- and low-frequency information that is helpful for target detection or segmentation; finally, it uses an Inverse Fourier Transform to map the cleaned frequency domain features back to the spatial domain.

2. The image multiscale fusion method of claim 1, wherein, The low-pass filter is a Gaussian filter; the Gaussian filter is a linear filter based on weights defined by the Gaussian function, which is updated by convolution operation and weighted average. The mathematical expression for performing a convolution operation is: ； in, The image pixel values ​​for which convolution operations are performed; In pixel coordinates Image pixel values ​​at that location; These are the weights of the Gaussian filter; The x-coordinate is the pixel coordinate. The vertical coordinate is the pixel coordinate. and All of these are iterative variables in the Gaussian filtering process; In order to be in The pixel value at that location.

3. The image multiscale fusion method of claim 1, wherein, An FFT Transformer is used to denoise the high-frequency information, and a deep neural network is used to extract multi-scale deep features to obtain the high-frequency feature activation values ​​corresponding to each pixel, specifically including: The FFT Transformer is used to denoise the high-frequency information, resulting in denoised high-frequency feature information. The mathematical expression for the FFT Transformer denoising process is as follows: ； ； ； ； in, This refers to the high-frequency component information corresponding to the infrared image; This refers to the denoised high-frequency feature information corresponding to the visible light image; This refers to the denoised high-frequency feature information corresponding to the infrared image; These are the attention weights of visible light features after self-attention processing; The attention weights are the infrared light features after self-attention operation; It is a norm; It is a feedforward neural network; This is a self-attention mechanism based on Fast Fourier Transform; This is a self-attention mechanism based on the fast inverse Fourier transform; For Fast Fourier Transform; This is the inverse fast Fourier transform; A deep neural network is used to extract multi-scale deep features from the denoised high-frequency feature information to obtain the high-frequency feature activation value corresponding to each pixel. The deep neural network is trained using a stochastic gradient algorithm with the goal of minimizing the loss function. The backbone of the deep neural network uses ResNet34; the structure of ResNet34 includes: an initial convolutional module and four residual stages; each residual stage contains a set number of residual blocks, and each residual block consists of two 3×3 convolutional layers, batch normalization and the ReLU activation function.

4. The image multiscale fusion method of claim 1, wherein, The mathematical expression corresponding to the fused image information in the high-frequency component is: ； in, This refers to the fused image information of the high-frequency components; The fusion weights for each region of the visible light image; This refers to the high-frequency portion of the visible light image; The fusion weights for each region of the infrared image; This refers to the high-frequency information corresponding to the infrared image.

5. The image multiscale fusion method of claim 1, wherein, The mathematical expression corresponding to the image information fused from visible light and infrared dual light sources is: ； in, This involves fusing image information from both visible and infrared light sources. This refers to the fused image information for the low-frequency components; This refers to the fused image information of the high-frequency components; The proportion of low-frequency information fusion; This refers to the proportion of high-frequency information fusion.

6. An image multi-scale fusion device, characterized in that, include: Image information data acquisition module, used to acquire image information data; The image information data includes: visible light images and infrared images; An image extraction module is used to extract image information from the image information data using a low-pass filter to obtain extracted information; the extracted information includes: high-frequency information and low-frequency information. The low-frequency fusion module is used to fuse low-frequency information based on a preset fusion ratio to obtain fused image information of the low-frequency part. The high-frequency extraction module is used to denoise the high-frequency information using an FFT Transformer and to extract multi-scale deep features using a deep neural network to obtain the high-frequency feature activation value corresponding to each pixel. The fusion weight determination module is used to merge the channels of the image based on all high-frequency feature activation values ​​using the L1 norm, and to determine the fusion weights of the visible light image and the infrared light image in each region. The high-frequency fusion module is used to perform weighted fusion of high-frequency information based on fusion weights to obtain fused image information of the high-frequency components. The image fusion module is used to determine the fused image information of visible light and infrared dual light sources based on the fused image information of the low-frequency part and the fused image information of the high-frequency part. The backbone of the deep neural network uses ResNet34; ResNet34 can not only extract image features at different scales and levels, but also use residual connections to maintain information flow and suppress gradient vanishing, thereby learning high-frequency details in the image. Based on all high-frequency feature activation values, the L1 norm is used to merge the image channels, and the fusion weights for the visible light image and the infrared light image in each region are determined, specifically including: Based on all high-frequency feature activation values, the L1 norm is used to merge the image channels to obtain pixel activation values; Based on the local block averaging operator, the fusion weights of the visible light image and the infrared light image in each region are determined according to the pixel activation values; the mathematical expression for the fusion weights is: ； ； in, The fusion weights for each region of the visible light image; The fusion weights for each region of the infrared image; The channels of the visible light image are merged using the L1 norm to obtain pixel activation values; The infrared image channels are merged using the L1 norm to obtain pixel activation values; The FFT Transformer refines high-frequency noise in two types of images. In addition to the multi-head self-attention and feedforward network of the standard Transformer encoder, it integrates Fast Fourier Transform and Inverse Fourier Transform operators: First, it performs a Fourier transform on the input multi-channel feature map, projecting the spatial domain features into the frequency domain to capture and distinguish different frequency components; then, it uses a frequency domain filter with learnable parameters to dynamically suppress high-frequency noise and retain mid- and low-frequency information that is helpful for target detection or segmentation; finally, it uses an Inverse Fourier Transform to map the cleaned frequency domain features back to the spatial domain.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the image multi-scale fusion method according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the image multi-scale fusion method according to any one of claims 1-5.

9. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image multi-scale fusion method according to any one of claims 1-5.

Images