Infrared and visible image fusion method based on resnet50 and double pyramid
By using ResNet50 and the dual pyramid method, the problems of detail loss and artifacts in infrared and visible light image fusion are solved, achieving efficient image fusion with low resource consumption, which is suitable for engineering applications.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGCHUN UNIV OF SCI & TECH
- Filing Date
- 2022-06-11
- Publication Date
- 2026-05-19
AI Technical Summary
Existing infrared and visible light image fusion methods suffer from loss of detailed texture information, blurred target edges, and artifacts. Furthermore, deep learning-based methods are time-consuming and energy-intensive, making them difficult to apply in practical engineering.
A ResNet50-based approach with dual pyramids is employed, which generates high-quality fused images through preprocessing, window sliding block division, feature map computation, multi-level pyramid decomposition, and weighted average fusion.
It achieves efficient image fusion, improves fusion accuracy and efficiency, reduces hardware resource requirements, and facilitates engineering applications.
Smart Images
Figure CN114897883B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of infrared and visible light image processing technology, and in particular to an infrared and visible light image fusion method based on ResNet50 and dual pyramids. Background Technology
[0002] In the field of multimodal image processing technology, with the development of optoelectronic technology, optical imaging in the infrared and visible light bands has become increasingly mature and popular. Visible light images possess rich appearance information and high spatial resolution, but they are easily affected by environmental factors such as illumination, smoke, fog, haze, or other adverse weather conditions during the imaging process, leading to image quality degradation. Infrared images primarily reflect the thermal radiation characteristics of objects; the higher the target temperature, the stronger the radiation, and the more significant the scene information description. Infrared images are less affected by environmental factors or camouflage during the imaging process, but they typically suffer from low resolution, lack of detail, and poor clarity. Image fusion technology can utilize the complementary characteristics of infrared and visible light images to effectively fuse the two images, generating a fused image that is comprehensive in information, highlights thermal radiation targets, and preserves texture information. Therefore, in multimodal image processing, utilizing image fusion technology is an important way to enhance the application value and scope of infrared and visible light images.
[0003] Existing infrared and visible light image fusion methods are mainly divided into three categories: fusion methods based on multi-scale transformation, fusion methods based on sparse representation, and fusion methods based on deep learning. Fusion methods based on multi-scale transformation decompose the source image into multi-scale representations, then merge the decomposed images using specific fusion rules, and finally reconstruct the merged image using inverse multi-scale transformation to obtain the fused image. This type of method can effectively preserve the background and target contour information of the source image, but lacks effective preservation of detailed texture information. Fusion methods based on sparse representation use an overcomplete dictionary learned to represent the source image with sparse coefficients, then merge the sparse coefficients using designed fusion rules, and finally reconstruct the fused image using the overcomplete dictionary and the merged sparse coefficients. This type of method is prone to edge blurring and artifacts, severely affecting the fusion effect. Fusion methods based on deep learning use convolutional neural networks to build an encoder and decoder, forming a holistic deep learning network. The encoder automatically extracts features from the source image, and the decoder fuses the feature images, finally generating a fused image, achieving end-to-end automatic image fusion. In terms of fusion performance, deep learning-based fusion methods have advantages over the previous two types of methods. However, the parameters of the encoder and decoder in deep learning networks are obtained through training using datasets and loss functions, thus requiring a large image dataset with properties similar to the images to be fused, which is difficult to obtain in practice. Furthermore, the training and fusion processes consume significant memory and are time-consuming and energy-intensive, hindering the practical engineering application of infrared and visible light image fusion technology. Therefore, there is an urgent need in this field for an infrared and visible light image fusion method that can overcome the above-mentioned shortcomings. Summary of the Invention
[0004] This invention provides an infrared and visible light image fusion method based on ResNet50 and dual pyramids to solve the problems of existing fusion techniques, such as multi-scale transformation-based fusion methods failing to effectively preserve detailed texture information, sparse representation-based fusion methods easily producing edge blurring and artifacts, and deep learning-based fusion methods consuming large amounts of memory and being time-consuming and energy-intensive. This invention avoids the loss of detailed texture information, target edge blurring, and artifacts in the fused image, while also having high fusion execution efficiency and low hardware resource consumption.
[0005] The technical solution adopted by this invention includes the following steps:
[0006] Step A: Image I from infrared and visible light sources X Perform normalization preprocessing to generate normalized source images. Using this image as input to a pre-trained ResNet50 residual network model, feature maps of four residual blocks are obtained. For infrared and visible light image indices, R is the infrared image index, V is the visible light image index, and b∈{1,2,3,4} is the residual block index;
[0007] Step B: Calculate the l1-norm of the feature map group of each residual block using a sliding window method to obtain the activity measure map of each residual block.
[0008] Step C: Activity measurement plots for each residual block Upsampling is performed to obtain the initial salient feature maps of each residual block. Its size and normalized source image The dimensions are consistent;
[0009] Step D: Calculate the spatial frequency of the normalized source image using a sliding window block method to obtain the spatial frequency map SF of the normalized source image. X and the initial salient feature map of each residual block. Perform dot product operations to generate salient feature maps for each residual block.
[0010] Step E: Statistical mapping of each residual block Gaussian pyramid decomposition is performed to obtain the salient feature maps of each residual block and the Gaussian pyramid images at different decomposition levels. The Gaussian pyramid weight map of each residual block at different decomposition levels is obtained using the Softmax operator. l is the decomposition level index, 0 ≤ l ≤ L. The maximum number of decomposition levels. This indicates rounding down, where H and W are the row and column values of the decomposed image, respectively, and min(·) indicates taking the minimum value;
[0011] Step F: Normalize the source image Perform Laplacian pyramid decomposition to obtain the normalized source image's Laplacian pyramid images (ILPs) at different decomposition levels. X(l) ;
[0012] Step G: Utilize the Gaussian pyramid weight map of each residual block at different decomposition levels Laplacian pyramid images ILP on different decomposition layers X(l) A weighted average operation is performed to obtain the Laplacian pyramid fused image of each residual block on the non-decomposition layer.
[0013] Step H: Apply the inverse Laplacian pyramid transform to fuse the Laplacian pyramid images of each residual block at different decomposition levels. Reconstruction is performed to obtain the fused image of each residual block. fused image of each residual block The final fused image FF is obtained by using a pixel-maximizing algorithm.
[0014] Step A of the present invention includes:
[0015] Sub-step A1: Perform normalization preprocessing on the infrared and visible light source images. The normalization calculation formula is as follows:
[0016]
[0017] In the formula, X∈{R,V} are the infrared and visible light image indices, respectively, where R is the infrared image index and V is the visible light image index. The normalized infrared and visible light images are called the normalized source images, I. X min Represents source image I X The minimum value of the middle pixel, I X max Represents source image I X The maximum value of the middle pixel;
[0018] Sub-step A2: Construct a ResNet50 residual network model. The input-to-output sequence is: input convolutional layer (conv1), residual block 1 (conv2_x), residual block 2 (conv3_x), residual block 3 (conv4_x), residual block 4 (conv5_x), and a fully connected layer (FC). The ResNet50 residual network model includes one input convolutional layer (conv1), four residual blocks, and one fully connected layer, in the order of input convolutional layer, residual block 1, residual block 2, residual block 3, residual block 4, and fully connected layer. The kernel size and stride of the input convolutional layer are 7×7×64 and 2, respectively. Residual block 1 contains three groups of convolutional kernels arranged in order of [1×1×64, 3×3×64, 1×1×256]. The output is the feature. Figure 1 contains 256 feature maps. Residual block 2 contains four sequentially arranged convolutional kernel groups [1×1×128, 3×3×128, 1×1×512], outputting feature map group 2 containing 512 feature maps. Residual block 3 contains six sequentially arranged convolutional kernel groups [1×1×256, 3×3×256, 1×1×1024], outputting feature map group 3 containing 1024 feature maps. Residual block 4 contains four sequentially arranged convolutional kernel groups [1×1×512, 3×3×512, 1×1×2048], outputting feature map group 4 containing 2048 feature maps. The fully connected layer performs global mean pooling on each feature map and then multiplies it with the weight vector to obtain the output of the fully connected layer. The entire network has 50 convolutional layers.
[0019] Normalized source image After passing through the ResNet50 residual network model, the four residual blocks in the ResNet50 model will output four feature map groups, represented as follows: For residual block indices, N∈{256,512,1024,2048} represents the index value of the number of feature maps in the four feature map groups output by the four residual blocks, where 1:N indicates the number of feature maps from the four residual blocks. Figure 1 To feature map N, the ResNet50 residual network model is as follows: Figure 3 As shown.
[0020] Step B of the present invention includes:
[0021] In the feature map group of residual blocks In this method, a 5×5×N window is selected using a sliding window approach, arranging it into 25 N-dimensional row vectors. N is the index of the number of feature maps in the four feature map groups output by the four residual blocks, N∈{256,512,1024,2048}. The l1-norm of each vector is calculated, which is the sum of the absolute values of each element in the vector, resulting in 25 l1-norm values. The average of these 25 l1-norm values is then calculated to obtain a pixel value. When the window sliding step is 1, after traversing the entire feature map group, the activity measure map of the residual blocks is obtained. This method can be used to obtain the activity measure map of four residual blocks, represented as follows: A schematic diagram of the activity measure map generation process is shown below. Figure 4 As shown in the figure, x 1,1 ,x 1,2 ,...,x 1,N Let x be the first row vector. 2,1 ,x 2,2 ,...,x 2,N This is the second row vector, and so on, x 25,1 ,x 25,2 ,...,x 25,N Let y1, y2, ..., y be the 25th row vector. N These are the l1-norm values of each row vector.
[0022] Step C of the present invention includes:
[0023] Activity measure plots of each residual block using bilinear interpolation algorithm Upsampling is performed to obtain the initial salient feature map of each residual block, represented as follows: The interpolation result is represented by pixels. The calculation formula for the bilinear interpolation algorithm is as follows:
[0024]
[0025] In the formula, The coordinates of the point to be interpolated. Points to be interpolated Pixel value at that location, and Points to be interpolated The coordinates of the four adjacent points are given, and their pixel values are respectively and After interpolation of all points to be interpolated is completed, the initial salient feature map of each residual block is obtained, represented as follows: Its size and normalized source image The dimensions are consistent.
[0026] Step D of the present invention includes:
[0027] In normalized source image In the example, a 5×5 window is selected using a sliding window partitioning method with a sliding step size of 1. Its spatial frequency is calculated using the following formula:
[0028]
[0029] In the formula The spatial frequency value of the k-th 5×5 window sliding block image. and These are the row and column frequency values of the 5×5 window sliding block image, respectively, where k is the window index, k∈{1,2,...,K}, and K is the normalized source image. The formulas for calculating the number of window sliders, row frequency value, and column frequency value contained in the data are as follows:
[0030]
[0031] In the formula and Let be the row frequency value and column frequency value of the k-th 5×5 window sliding block image, respectively, and (i,j), (i,j+1), and (i+1,j) be the pixel coordinates within the 5×5 window sliding block. and These are the pixel values at the pixel coordinates (i,j), (i,j+1), and (i+1,j) within the 5×5 window slider, respectively.
[0032] When the window slider traverses the entire normalized source image Then, the spatial frequency map SF, which reflects the sharpness of the normalized source image, is obtained. X Then the spatial frequency diagram SF X Initial salient feature map of each residual block Performing a dot product operation yields the salient feature map of each residual block, represented as follows: The calculation formula is as follows:
[0033]
[0034] In the formula, SF X , and Normalized source images The spatial frequency plot, the initial salient feature plot of each residual block, and the salient feature plot of each residual block are shown. A schematic diagram of the salient feature plot generation process for each residual block is provided. Figure 5 As shown.
[0035] Step E of the present invention includes:
[0036] Saliency maps of each residual block Gaussian pyramid decomposition is performed separately to obtain the salient feature maps of each residual block on the Gaussian pyramid images of different decomposition layers. The Softmax operator is used to obtain the Gaussian pyramid weight maps of each residual block on different decomposition layers.
[0037] Saliency map of each residual block The Gaussian pyramid decomposition process is represented in pixel output form as follows:
[0038]
[0039]
[0040] In the formula, This is the salient feature map of the b-th residual block. Let W be the Gaussian pyramid image of the 0th layer of the salient feature map of the b-th residual block, where W is the Gaussian kernel weight matrix. l is the decomposition level index, 0 ≤ l ≤ L. The maximum number of decomposition levels. This indicates rounding down, where H and W are the row and column values of the decomposed image, respectively, and min(·) indicates taking the minimum value. Let be the pixel coordinates of the salient feature map of the b-th residual block in the Gaussian pyramid image at layer l;
[0041] In the Gaussian pyramid decomposition process, the salient feature maps of the residual blocks are first... As the image of the 0th layer of the Gaussian pyramid, it is represented as A Gaussian kernel weight matrix is used to perform convolution on the image, and then the convolution result image is downsampled to remove even-numbered rows and columns, resulting in the first-level Gaussian pyramid image, represented as follows. Using this image as input, repeated convolution and downsampling operations yield the second Gaussian pyramid image, denoted as... This process is repeated L times, forming an L+1-layer pyramid-shaped image structure with progressively halved resolution, called a Gaussian pyramid. The Gaussian pyramid image of the salient feature map of each residual block is represented as follows:
[0042] Gaussian pyramid images of each residual block The Gaussian pyramid weight map of each residual block is generated using the Softmax operator, and is represented as follows: The calculation formula is as follows:
[0043]
[0044] In the formula and These are, respectively, the Gaussian pyramid image of the infrared image of the l-th layer in the b-th residual block, the Gaussian pyramid image of the visible light image of the l-th layer in the b-th residual block, the Gaussian pyramid weight map of the infrared image of the l-th layer in the b-th residual block, and the Gaussian pyramid weight map of the visible light image of the l-th layer in the b-th residual block.
[0045] Step F of the present invention includes:
[0046] Normalized source image Perform Laplacian pyramid decomposition to obtain the normalized source image. Laplacian pyramid images at different decomposition levels. The decomposition process of the Laplacian pyramid includes two parts: Gaussian pyramid decomposition and prediction residuals. Normalized source image. The Gaussian pyramid decomposition process is represented in pixel output form as follows:
[0047]
[0048]
[0049] In the formula, For the normalized source image, GI X(0) Normalized source image Image of the 0th level of the Gaussian pyramid, (p l ,q l () is the normalized source image The pixel coordinates of the l-th layer of the Gaussian pyramid image, GI X(l) (p l ,q l The image of the l-th layer of the Gaussian pyramid, which is a normalized source image, is located at (p). l ,q l The pixel value at (2p) l +λ,2q l +θ) represents the normalized source image. The pixel coordinates of the (l-1)th layer of the Gaussian pyramid image, GI X(l-1) (2p l +λ,2q l +θ) represents the normalized source image. The image of the l-1 level of the Gaussian pyramid is in (2p l +λ,2q l The pixel value at +θ);
[0050] During the decomposition process, the source image is first normalized. As the image of the 0th layer of the Gaussian pyramid, denoted as GI X(0) The Gaussian pyramid image GI is obtained by convolving it with a Gaussian kernel weight matrix, and then downsampling the convolution result image to remove even-numbered rows and columns. X(1) Using this image as input, repeated convolution and downsampling operations are performed to obtain the second-layer Gaussian pyramid image GI. X(2) This process is repeated L times, forming an L+1-layer pyramid-shaped image structure with progressively halved resolution, known as a Gaussian pyramid, which normalizes the source image. The Gaussian pyramid image is represented as:
[0051] The algorithm for predicting residuals is as follows:
[0052]
[0053] ILP X(L) =GI X(L)
[0054] In the formula, GI X(L) Normalized source image Image of the Lth layer of the Gaussian pyramid, ILP X(L) The normalized source image is the Lth layer Laplacian pyramid image, (α) l ,β l () is the normalized source image Pixel position coordinates of the l-th layer Laplacian pyramid image and Gaussian pyramid image, ILP X(l) (α l ,β l () is the normalized source image The image of the l-th layer of the Laplace pyramid in (α) l ,β l The pixel value at position ) , GI X(l) (α l ,β l The image of the l-th layer of the Gaussian pyramid, which is a normalized source image, is located at (α). l ,β l The pixel value at () Normalized source image The pixel coordinates of the (l+1)th layer of the Laplacian pyramid image. Normalized source image The image of the l+1 level of the Laplace pyramid is in Pixel value at;
[0055] In the process of predicting residuals, the source image is first normalized. Image of the Lth layer of the Gaussian pyramid GI X(L) As the Lth level Laplacian pyramid image, it is represented as ILP X(L) The image is upsampled, and then convolved with a Gaussian kernel weight matrix. The L(1)th layer Gaussian pyramid image is then subtracted from the convolved image to obtain the L(1)th layer Laplacian pyramid image, denoted as ILP. X(L-1) Using this image as input, the upsampling, convolution, and subtraction operations are repeated to obtain the L-2 level Laplacian pyramid image, denoted as ILP. X(L-2) This process is repeated L times to form an L+1-layer pyramid-shaped image structure with progressively doubling resolution, known as the Laplacian pyramid, which normalizes the source image. The Gaussian pyramid image is represented as ILP X(l) .
[0056] Step G of the present invention includes:
[0057] The Laplacian pyramid image of the normalized source image is weighted and fused using the Gaussian pyramid weight map of each residual block to obtain the fused Laplacian pyramid image of each residual block. The calculation formula is as follows:
[0058]
[0059] In the formula and The l-th layer Gaussian pyramid weight map of the b-th residual block is used to fuse the l-th layer Laplacian pyramid image of the infrared and visible light images in the normalized source image. (ILP) R(l) The ILP is the l-th layer Laplacian pyramid image of the infrared image in the normalized source image. V(l) This is the l-th layer Laplacian pyramid image of the visible light image in the normalized source image. This is the merged image of the l-th layer of the b-th residual block's Laplacian pyramid.
[0060] Step H of the present invention includes:
[0061] Laplacian pyramid fused image of each residual block using inverse Laplacian pyramid transform Reconstruction is performed to obtain the fused image of each residual block, and then the pixel enlargement algorithm is used to obtain the final fused image FF;
[0062] The Laplace pyramid inverse transform algorithm is represented in pixel-wise form as follows:
[0063]
[0064]
[0065] In the formula, This is the fused Laplacian pyramid image of the Lth layer of the b-th residual block. For the L-th layer of the b-th residual block, (ξ) is the inverse Laplacian pyramid image. (l) ,ζ (l) () represents the pixel coordinates of the fused Laplacian pyramid image and the inverse Laplacian pyramid image of the b-th residual block at layer l. The Laplacian pyramid fusion image of the l-th layer of the b-th residual block in (ξ) (l) ,ζ (l) The pixel value at () The inverse Laplacian pyramid image of the l-th layer of the b-th residual block in (ξ) (l) ,ζ (l) The pixel value at () Let be the pixel coordinates of the (l+1)th layer of the Laplacian pyramid image of the b-th residual block. For the image of the (l+1)th layer of the b-th residual block, in Pixel value at;
[0066] In the inverse Laplacian transform process, the Lth-level Laplacian pyramid inverse transform image of each residual block is first upsampled. Then, a Gaussian kernel weight matrix is used to convolve it. The convolution result image is added to the (L-1)th-level Laplacian pyramid fused image to obtain the (L-1)th-level Laplacian pyramid inverse transform image. This image is used as input, and the upsampling, convolution, and summation operations are repeated to obtain the (L-2)th-level Laplacian pyramid inverse transform image. This process is repeated L times to obtain the 0th-level Laplacian pyramid inverse transform image of each residual block, denoted as... This is the fused image of each residual block;
[0067] fused image of each residual block The final fused image is obtained by using a pixel-maximizing algorithm. The calculation formula is as follows:
[0068]
[0069] In the formula, max{·} represents the maximum pixel value operation, and FF is the final merged image. These are the fused images of the first, second, third, and fourth residual blocks, respectively.
[0070] This invention first employs a pre-trained ResNet50 residual network model to automatically extract infrared and visible light image features from each residual block. Combining the spatial frequency images of the infrared and visible light images, a salient feature map for each residual block is formed. Then, a Laplacian pyramid is used to perform multi-level decomposition on both the infrared and visible light images, generating Laplacian pyramid images. A Gaussian pyramid is then used to perform multi-level decomposition on the salient feature maps of each residual block, forming a Gaussian pyramid weight map for each residual block. The Laplacian pyramid images are then weighted and averaged to generate a fused Laplacian pyramid image for each residual block. Finally, an inverse Laplacian pyramid transform is used to reconstruct the fused Laplacian pyramid images of each residual block, resulting in a fused image for each residual block. Finally, pixel enlargement is performed on the fused images of each residual block to obtain the final fused image. This method combines the advantages of deep learning networks and multi-scale decomposition methods, enabling effective fusion of infrared and visible light images without requiring a dedicated training image set or complex hardware, thus effectively improving image fusion accuracy and enhancing the practicality of infrared and visible light image fusion.
[0071] The present invention has the following beneficial effects:
[0072] (1) The ResNet50 model of the pre-trained residual network is used to extract the features of infrared and visible light images, realize the semantic representation of images at three levels of low / medium / high, and express the features of infrared and visible light images more comprehensively and accurately. There is no need to train network parameters and training datasets, which saves the need for complex hardware configuration performance and facilitates the engineering application of infrared and visible light image fusion.
[0073] (2) The Laplacian pyramid is used to decompose the infrared and visible light images into multiple layers, effectively separating the spatial overlapping features of the images. The Gaussian pyramid is used to decompose the salient feature map into multiple layers, generating the Gaussian pyramid weight map. The Laplacian pyramid image is then fused by weighted average to improve the fusion accuracy and efficiency, providing a better foundation for other computer vision tasks. Attached Figure Description
[0074] Figure 1 These are test infrared and visible light images according to an embodiment of the present invention, wherein (a) is an infrared image; and (b) is a visible light image.
[0075] Figure 2 The comparison method of the embodiments of the present invention and the method of the present invention are compared. Figure 1The images shown are fusion results of infrared and visible light images, where (a) is the fusion result image compared with method one; (b) is the fusion result image compared with method two; and (c) is the fusion result image of the method of the present invention.
[0076] Figure 3 This is a diagram of the ResNet50 residual network model according to an embodiment of the present invention;
[0077] Figure 4 This is a schematic diagram of the generation of the activity measurement map of each residual block in an embodiment of the present invention;
[0078] Figure 5 This is a schematic diagram illustrating the generation of spatial frequency maps of infrared and visible light images in an embodiment of the present invention;
[0079] Figure 6 This is a flowchart of an infrared and visible light image fusion method based on ResNet50 and dual pyramids according to an embodiment of the present invention;
[0080] Figure 7 This is an overall block diagram of the infrared and visible light image fusion method based on ResNet50 and dual pyramids according to an embodiment of the present invention. Detailed Implementation
[0081] Figure 6 and Figure 7 The flowchart and overall block diagram of the infrared and visible light image fusion method based on ResNet50 and dual pyramids according to embodiments of the present invention are shown below. Figure 6 and Figure 7 It includes the following steps:
[0082] Step A: Image I from infrared and visible light sources X Perform normalization preprocessing to generate normalized source images. Using this image as input to a pre-trained ResNet50 residual network model, feature maps of four residual blocks are obtained. For infrared and visible light image indexes, R is the infrared image index, V is the visible light image index, and b∈{1,2,3,4} is the residual block index; including:
[0083] Sub-step A1: Perform normalization preprocessing on the infrared and visible light source images. The normalization calculation formula is as follows:
[0084]
[0085] In the formula, X∈{R,V} are the infrared and visible light image indices, respectively, where R is the infrared image index and V is the visible light image index. The normalized infrared and visible light images are called the normalized source images, I. Xmin Represents source image IX The minimum value of the middle pixel, I Xmax Represents source image I X The maximum value of the middle pixel;
[0086] Sub-step A2: Construct a ResNet50 residual network model. The input to output sequence is: input convolutional layer (conv1), residual block 1 (conv2_x), residual block 2 (conv3_x), residual block 3 (conv4_x), residual block 4 (conv5_x), and fully connected layer (FC). The ResNet50 residual network model includes one input convolutional layer (conv1), four residual blocks, and one fully connected layer, in the order of input convolutional layer, residual block 1, residual block 2, residual block 3, residual block 4, and fully connected layer. The input convolutional layer has a kernel size of 7×7×64 and a stride of 2. Residual block 1 contains three sequentially arranged [1×1×64, 3×3×64, 1×1×256] convolutional kernel groups, and the output is feature map group 1, containing 256 feature maps. Residual block 2 contains four sequentially arranged [1×1×128, 3×3×128, 1×1×512] convolutional kernel groups, and the output is feature map group 2, containing 512 feature maps. Residual block 3 contains six sequentially arranged [1×... The network consists of 1×256, 3×3×256, and 1×1×1024 convolutional kernel groups, with the output being feature map group 3 containing 1024 feature maps. The residual block 4 contains four sequentially arranged [1×1×512, 3×3×512, and 1×1×2048] convolutional kernel groups, with the output being feature map group 4 containing 2048 feature maps. The fully connected layer performs global mean pooling on each feature map and then multiplies it with the weight vector to obtain the output of the fully connected layer. The entire network has a total of 50 convolutional layers.
[0087] Normalized source image After passing through the ResNet50 residual network model, the four residual blocks in the ResNet50 model will output four feature map groups, represented as follows: For residual block indices, N∈{256,512,1024,2048} represents the index value of the number of feature maps in the four feature map groups output by the four residual blocks, where 1:N indicates the number of feature maps from the four residual blocks. Figure 1 To feature map N, the ResNet50 residual network model is as follows: Figure 3 As shown;
[0088] Step B: Calculate the l1-norm of the feature map group of each residual block using a sliding window method to obtain the activity measure map of each residual block. include:
[0089] In the feature map group of residual blocks In this method, a 5×5×N window is selected using a sliding window approach, arranging it into 25 N-dimensional row vectors. N is the index of the number of feature maps in the four feature map groups output by the four residual blocks, N∈{256,512,1024,2048}. The l1-norm of each vector is calculated, which is the sum of the absolute values of each element in the vector, resulting in 25 l1-norm values. The average of these 25 l1-norm values is then calculated to obtain a pixel value. When the window sliding step is 1, after traversing the entire feature map group, the activity measure map of the residual blocks is obtained. This method can be used to obtain the activity measure map of four residual blocks, represented as follows: A schematic diagram of the activity measure map generation process is shown below. Figure 4 As shown in the figure, x 1,1 ,x 1,2 ,...,x 1,N Let x be the first row vector. 2,1 ,x 2,2 ,...,x 2,N This is the second row vector, and so on, x 25,1 ,x 25,2 ,...,x 25,N Let y1, y2, ..., y be the 25th row vector. N These are the l1-norm values of each row vector;
[0090] Step C: Activity measurement plots for each residual block Upsampling is performed to obtain the initial salient feature maps of each residual block. Its size and normalized source image Consistent in size; including:
[0091] Activity measure plots of each residual block using bilinear interpolation algorithm Upsampling is performed to obtain the initial salient feature map of each residual block, represented as follows: The interpolation result is represented by pixels. The calculation formula for the bilinear interpolation algorithm is as follows:
[0092]
[0093] In the formula, The coordinates of the point to be interpolated. Points to be interpolated Pixel value at that location, and Points to be interpolated The coordinates of the four adjacent points are given, and their pixel values are respectively and After interpolation of all points to be interpolated is completed, the initial salient feature map of each residual block is obtained, represented as follows: Its size and normalized source image The dimensions are consistent;
[0094] Step D: Calculate the spatial frequency of the normalized source image using a sliding window block method to obtain the spatial frequency map SF of the normalized source image. X and the initial salient feature map of each residual block. Perform dot product operations to generate salient feature maps for each residual block. include:
[0095] In normalized source image In the example, a 5×5 window is selected using a sliding window partitioning method with a sliding step size of 1. Its spatial frequency is calculated using the following formula:
[0096]
[0097] In the formula The spatial frequency value of the k-th 5×5 window sliding block image. and These are the row and column frequency values of the 5×5 window sliding block image, respectively, where k is the window index, k∈{1,2,...,K}, and K is the normalized source image. The formulas for calculating the number of window sliders, row frequency value, and column frequency value contained in the data are as follows:
[0098]
[0099] In the formula and Let be the row frequency value and column frequency value of the k-th 5×5 window sliding block image, respectively, and (i,j), (i,j+1), and (i+1,j) be the pixel coordinates within the 5×5 window sliding block. and These are the pixel values at the pixel coordinates (i,j), (i,j+1), and (i+1,j) within the 5×5 window slider, respectively.
[0100] When the window slider traverses the entire normalized source image Then, the spatial frequency map SF, which reflects the sharpness of the normalized source image, is obtained. X Then the spatial frequency diagram SF X Initial salient feature map of each residual block Performing a dot product operation yields the salient feature map of each residual block, represented as follows: The calculation formula is as follows:
[0101]
[0102] In the formula, SF X , and Normalized source images The spatial frequency plot, the initial salient feature plot of each residual block, and the salient feature plot of each residual block are shown. A schematic diagram of the salient feature plot generation process for each residual block is provided. Figure 5 As shown;
[0103] Step E: Statistical mapping of each residual block Gaussian pyramid decomposition is performed to obtain the salient feature maps of each residual block and the Gaussian pyramid images at different decomposition levels. The Gaussian pyramid weight map of each residual block at different decomposition levels is obtained using the Softmax operator. l is the decomposition level index, 0 ≤ l ≤ L. The maximum number of decomposition levels. This indicates rounding down, where H and W are the row and column values of the decomposed image, respectively, and min(·) represents taking the minimum value; including:
[0104] Saliency maps of each residual block Gaussian pyramid decomposition is performed separately to obtain the salient feature maps of each residual block on the Gaussian pyramid images of different decomposition layers. The Softmax operator is used to obtain the Gaussian pyramid weight maps of each residual block on different decomposition layers.
[0105] Saliency map of each residual block The Gaussian pyramid decomposition process is represented in pixel output form as follows:
[0106]
[0107]
[0108] In the formula, This is the salient feature map of the b-th residual block. Let W be the Gaussian pyramid image of the 0th layer of the salient feature map of the b-th residual block, where W is the Gaussian kernel weight matrix. l is the decomposition level index, 0 ≤ l ≤ L. The maximum number of decomposition levels. This indicates rounding down, where H and W are the row and column values of the decomposed image, respectively, and min(·) indicates taking the minimum value. Let be the pixel coordinates of the salient feature map of the b-th residual block in the Gaussian pyramid image at layer l;
[0109] In the Gaussian pyramid decomposition process, the salient feature maps of the residual blocks are first... As the image of the 0th layer of the Gaussian pyramid, it is represented as A Gaussian pyramid image is obtained by convolving it with a Gaussian kernel weight matrix, followed by downsampling (removing even-numbered rows and columns) of the convolution result. This yields the first-level Gaussian pyramid image, represented as follows: Using this image as input, repeated convolution and downsampling operations yield the second Gaussian pyramid image, denoted as... This process is repeated L times, forming an L+1-layer pyramid-shaped image structure with progressively halved resolution, called a Gaussian pyramid. The Gaussian pyramid image of the salient feature maps of each residual block is represented as follows:
[0110] Gaussian pyramid images of each residual block The Gaussian pyramid weight map of each residual block is generated using the Softmax operator, and is represented as follows: The calculation formula is as follows:
[0111]
[0112] In the formula and These are, respectively, the Gaussian pyramid image of the infrared image of the l-th layer in the b-th residual block, the Gaussian pyramid image of the visible light image of the l-th layer in the b-th residual block, the Gaussian pyramid weight map of the infrared image of the l-th layer in the b-th residual block, and the Gaussian pyramid weight map of the visible light image of the l-th layer in the b-th residual block;
[0113] Step F: Normalize the source image Perform Laplacian pyramid decomposition to obtain the normalized source image's Laplacian pyramid images (ILPs) at different decomposition levels. X(l) ;include:
[0114] Normalized source image Perform Laplacian pyramid decomposition to obtain the normalized source image. Laplacian pyramid images at different decomposition levels. The decomposition process of the Laplacian pyramid includes two parts: Gaussian pyramid decomposition and prediction residuals. Normalized source image. The Gaussian pyramid decomposition process is represented in pixel output form as follows:
[0115]
[0116]
[0117] In the formula, For the normalized source image, GI X(0) Normalized source image Image of the 0th level of the Gaussian pyramid, (p l ,q l () is the normalized source image The pixel coordinates of the l-th layer of the Gaussian pyramid image, GI X(l) (p l ,q l The image of the l-th layer of the Gaussian pyramid, which is a normalized source image, is located at (p). l ,q l The pixel value at (2p) l +λ,2q l +θ) represents the normalized source image. The pixel coordinates of the (l-1)th layer of the Gaussian pyramid image, GI X(l-1) (2p l +λ,2q l +θ) represents the normalized source image. The image of the l-1 level of the Gaussian pyramid is in (2p l +λ,2q l The pixel value at +θ);
[0118] During the decomposition process, the source image is first normalized. As the image of the 0th layer of the Gaussian pyramid, denoted as GI X(0) The Gaussian pyramid image GI is obtained by convolving it with a Gaussian kernel weight matrix and then downsampling the convolution result (removing even-numbered rows and columns). X(1) Using this image as input, repeated convolution and downsampling operations are performed to obtain the second-layer Gaussian pyramid image GI. X(2) This process is repeated L times, forming an L+1-layer pyramid-shaped image structure with progressively halved resolution, known as a Gaussian pyramid, which normalizes the source image. The Gaussian pyramid image is represented as GI X(l) ;
[0119] The algorithm for predicting residuals is as follows:
[0120]
[0121] ILP X(L) =GI X(L)
[0122] In the formula, GI X(L) Normalized source image Image of the Lth layer of the Gaussian pyramid, ILP X(L) The normalized source image is the Lth layer Laplacian pyramid image, (α) l ,β l () is the normalized source image Pixel position coordinates of the l-th layer Laplacian pyramid image and Gaussian pyramid image, ILP X(l) (α l ,β l() is the normalized source image The image of the l-th layer of the Laplace pyramid in (α) l ,β l The pixel value at position ) , GI X(l) (α l ,β l The image of the l-th layer of the Gaussian pyramid, which is a normalized source image, is located at (α). l ,β l The pixel value at (). Normalized source image The pixel coordinates of the (l+1)th layer of the Laplacian pyramid image. Normalized source image The image of the l+1 level of the Laplace pyramid is in Pixel value at;
[0123] In the process of predicting residuals, the source image is first normalized. Image of the Lth layer of the Gaussian pyramid GI X(L) As the Lth level Laplacian pyramid image, it is represented as ILP X(L) The image is upsampled, and then convolved with a Gaussian kernel weight matrix. The L(1)th layer Gaussian pyramid image is then subtracted from the convolved image to obtain the L(1)th layer Laplacian pyramid image, denoted as ILP. X(L-1) Using this image as input, the upsampling, convolution, and subtraction operations are repeated to obtain the L-2 level Laplacian pyramid image, denoted as ILP. X(L-2) This process is repeated L times to form an L+1-layer pyramid-shaped image structure with progressively doubling resolution, known as the Laplacian pyramid, which normalizes the source image. The Gaussian pyramid image is represented as ILP X(l) ;
[0124] Step G: Utilize the Gaussian pyramid weight map of each residual block at different decomposition levels Laplacian pyramid images ILP on different decomposition layers X(l) A weighted average operation is performed to obtain the Laplacian pyramid fused image of each residual block on the non-decomposition layer. include:
[0125] The Laplacian pyramid image of the normalized source image is weighted and fused using the Gaussian pyramid weight map of each residual block to obtain the fused Laplacian pyramid image of each residual block. The calculation formula is as follows:
[0126]
[0127] In the formula and The l-th layer Gaussian pyramid weight map of the b-th residual block is used to fuse the l-th layer Laplacian pyramid image of the infrared and visible light images in the normalized source image. (ILP) R(l) The ILP is the l-th layer Laplacian pyramid image of the infrared image in the normalized source image. V(l) This is the l-th layer Laplacian pyramid image of the visible light image in the normalized source image. This is the fused Laplacian pyramid image of the l-th layer of the b-th residual block;
[0128] Step H: Apply the inverse Laplacian pyramid transform to fuse the Laplacian pyramid images of each residual block at different decomposition levels. Reconstruction is performed to obtain the fused image of each residual block. fused image of each residual block The final fused image FF is obtained by using a pixel-maximizing algorithm; it includes:
[0129] Laplacian pyramid fused image of each residual block using inverse Laplacian pyramid transform Reconstruction is performed to obtain the fused image of each residual block, and then the pixel enlargement algorithm is used to obtain the final fused image FF;
[0130] The Laplace pyramid inverse transform algorithm is represented in pixel-wise form as follows:
[0131]
[0132]
[0133] In the formula, This is the fused Laplacian pyramid image of the Lth layer of the b-th residual block. For the L-th layer of the b-th residual block, (ξ) is the inverse Laplacian pyramid image. (l) ,ζ (l) () represents the pixel coordinates of the fused Laplacian pyramid image and the inverse Laplacian pyramid image of the b-th residual block at layer l. The Laplacian pyramid fusion image of the l-th layer of the b-th residual block in (ξ) (l) ,ζ (l) The pixel value at () The inverse Laplacian pyramid image of the l-th layer of the b-th residual block in (ξ) (l) ,ζ (l) The pixel value at () Let be the pixel coordinates of the (l+1)th layer of the Laplacian pyramid image of the b-th residual block. For the image of the (l+1)th layer of the b-th residual block, in Pixel value at;
[0134] In the inverse Laplacian transform process, the Lth-level Laplacian pyramid inverse transform image of each residual block is first upsampled. Then, a Gaussian kernel weight matrix is used to convolve it. The convolution result image is added to the (L-1)th-level Laplacian pyramid fused image to obtain the (L-1)th-level Laplacian pyramid inverse transform image. This image is used as input, and the upsampling, convolution, and summation operations are repeated to obtain the (L-2)th-level Laplacian pyramid inverse transform image. This process is repeated L times to obtain the 0th-level Laplacian pyramid inverse transform image of each residual block, denoted as... This is the fused image of each residual block;
[0135] fused image of each residual block The final fused image is obtained by using a pixel-maximizing algorithm. The calculation formula is as follows:
[0136]
[0137] In the formula, max{·} represents the maximum pixel value operation, and FF is the final merged image. These are the fused images of the first, second, third, and fourth residual blocks, respectively.
[0138] The following further illustrates application examples of the present invention: To verify the effectiveness of the infrared and visible light image fusion method proposed in this invention, the solution of this invention was tested on multiple real test datasets. Here, a set of datasets is selected as follows: Figure 1 The infrared image shown in (a) is as follows: Figure 1(b) shows a visible light image fusion experiment with an image size of 620×450. Furthermore, the fusion result obtained by the method of this invention is compared with the fusion results obtained by two existing methods: [CHLiu,Y.Qi,WRDing. Infrared and visible image fusion method based on saliency detection in sparsedomain[J].Infrared Physics&Technology,2017,83:94-102] (Comparison Method 1) [[Li H,Wu XJ,Kittler J.Infrared and Visible Image Fusion using a Deep Learning Framework[C].24th International Conference on Pattern Recognition(ICPR),Aug.20–24,2018,Beijing,China.New York:IEEE,2018,2705-2710.] (Comparison Method 2). The corresponding test results are as follows: Figure 2 As shown in (a)(b)(c).
[0139] Comparison Method 1: The infrared and visible light images are decomposed using a joint sparse representation model, saliency detection is performed in the sparse coefficient domain, and the sparse coefficients are fused using a weighted average algorithm to obtain the fused image.
[0140] Comparison Method 2: The infrared and visible light images are decomposed into base layer images and detail layer images using a low-pass filter. The base layer image is then fused using an equal-weighted average algorithm. The deep features of the detail layer image are extracted using a VGG19 network to generate a weight map. The detail layer images are then fused using a weighted average. Finally, the fused base layer image and detail layer image are added together to obtain the final fused image.
[0141] Figure 2 (a), (b), and (c) represent comparison method one, comparison method two, and the method of the present invention, respectively. Figure 1 (a) and (b) represent the fusion results of a set of infrared and visible light images; Figure 2In the fusion results shown, compared with the fusion result of method one, the edge detail information in the fused image is severely lost and there is some gray-scale distortion; compared with the fusion result of method two, the overall brightness and local contrast of the fused image are low, the energy is severely lost, which is not conducive to visual observation; in the fusion result of the method of the present invention, the structural information of the fused image is clear and complete, the complementary information of infrared and visible light images is well preserved, the detail information is rich, and the brightness and contrast are moderate.
[0142] To better evaluate the performance of fusion methods, in addition to the subjective evaluations mentioned above, objective evaluation metrics are used to objectively analyze the fusion results of different methods. The objective evaluation metrics used in this invention include the edge information transmission amount Q. AB / F Structural Information Similarity (SSIM), Differential Correlation Coefficient (SCD), and Human Visual System Perceptual Quality Evaluation Coefficient (Q) HVS The feature mutual information (FMIw) extracted using wavelet extraction is used. A higher value for the above evaluation index indicates better image quality, and vice versa. Objective evaluation index data are shown in Table 1. As can be seen from the objective evaluation data in Table 1, the resulting image obtained by the fusion method of this invention is superior to both comparative methods one and two in all objective index evaluations. This demonstrates that the method of this invention can consistently obtain high-quality fusion results.
[0143] Table 1 Objective Evaluation Indicator Data
[0144]
[0145] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. Based on the above description, those skilled in the art should have a clear understanding of the infrared and visible light image fusion method based on ResNet50 and dual pyramids of the present invention.
[0146] This invention presents an infrared and visible light image fusion method based on ResNet50 and dual pyramids. It utilizes a pre-trained ResNet50 residual network to automatically extract multi-layer features from infrared and visible light images. The spatial frequency images of the infrared and visible light images are combined to form salient feature maps for each residual block. Laplacian pyramids and Gaussian pyramids are used to perform multi-layer decomposition on the normalized source image and the salient feature maps of each residual block, respectively. Then, a weighted average fusion is performed. Finally, the fused images of each residual block are processed by pixel enlargement to obtain the fused image. This method combines the advantages of deep learning networks and multi-scale decomposition, enabling effective fusion of infrared and visible light images. Furthermore, it eliminates the need for specialized training image sets and complex hardware, effectively improving image fusion accuracy and efficiency, and facilitating the engineering application of infrared and visible light image fusion methods.
Claims
1. A method for fusing infrared and visible light images based on ResNet50 and dual pyramids, characterized in that, Includes the following steps: Step A: Image processing of infrared and visible light sources Perform normalization preprocessing to generate normalized source images. Using this image as input to a pre-trained ResNet50 residual network model, feature maps of four residual blocks are obtained. , This is an index for infrared and visible light images. For infrared image indexing, For visible light image indexing, For residual block index; Step B: Calculate the l1-norm of the feature map group of each residual block using a sliding window method to obtain the activity measure map of each residual block. ; Step C: Activity measurement plots for each residual block Upsampling is performed to obtain the initial salient feature maps of each residual block. Its size is similar to that of the normalized source image. The dimensions are consistent; Step D: Calculate the spatial frequency of the normalized source image using a sliding window block method to obtain the spatial frequency map of the normalized source image. and the initial salient feature map of each residual block. Perform dot product operations to generate salient feature maps for each residual block. ; Step E: Statistical mapping of each residual block Gaussian pyramid decomposition is performed to obtain the salient feature maps of each residual block and the Gaussian pyramid images at different decomposition levels. The Gaussian pyramid weight map of each residual block at different decomposition levels is obtained using the Softmax operator. , For decomposition layer index, , The maximum number of decomposition levels. This indicates rounding down, where H and W are the row and column values of the decomposed image, respectively. This indicates taking the minimum value; Step F: Normalize the source image Perform Laplacian pyramid decomposition to obtain Laplacian pyramid images of the normalized source image at different decomposition levels. ; Step G: Utilize the Gaussian pyramid weight map of each residual block at different decomposition levels Laplacian pyramid images at different decomposition levels A weighted average operation is performed to obtain the Laplacian pyramid fused image of each residual block on the non-decomposition layer. ; Step H: Apply the inverse Laplacian pyramid transform to fuse the Laplacian pyramid images of each residual block at different decomposition levels. Reconstruction is performed to obtain the fused image of each residual block. The fused image of each residual block The final fused image is obtained by using a pixel-maximizing algorithm. .
2. The infrared and visible light image fusion method based on ResNet50 and dual pyramids according to claim 1, characterized in that, Step A includes: Sub-step A1: Perform normalization preprocessing on the infrared and visible light source images. The normalization calculation formula is as follows: ； In the formula, For infrared and visible light image indexing, For infrared image indexing, For visible light image indexing, The normalized infrared and visible light images are called the normalized source images. Represents the source image The minimum value of the middle pixel. Represents the source image The maximum value of the middle pixel; Sub-step A2: Construct a ResNet50 residual network model. The input-to-output sequence is: input convolutional layer (conv1), residual block 1 (conv2_x), residual block 2 (conv3_x), residual block 3 (conv4_x), residual block 4 (conv5_x), and a fully connected layer (FC). The ResNet50 residual network model includes one input convolutional layer (conv1), four residual blocks, and one fully connected layer, in the order of input convolutional layer, residual block 1, residual block 2, residual block 3, residual block 4, and fully connected layer. The kernel size and stride of the input convolutional layer are 7×7×64 and 2, respectively. Residual block 1 contains three groups of convolutional kernels arranged in order of [1×1×64, 3×3×64, 1×1×256]. The output is the feature. Figure 1 contains 256 feature maps. Residual block 2 contains four sequentially arranged convolutional kernel groups [1×1×128, 3×3×128, 1×1×512], outputting feature map group 2 containing 512 feature maps. Residual block 3 contains six sequentially arranged convolutional kernel groups [1×1×256, 3×3×256, 1×1×1024], outputting feature map group 3 containing 1024 feature maps. Residual block 4 contains four sequentially arranged convolutional kernel groups [1×1×512, 3×3×512, 1×1×2048], outputting feature map group 4 containing 2048 feature maps. The fully connected layer performs global mean pooling on each feature map and then multiplies it with the weight vector to obtain the output of the fully connected layer. The entire network has 50 convolutional layers. Normalized source image After passing through the ResNet50 residual network model, the four residual blocks in the ResNet50 model will output four feature map groups, represented as follows: , For residual block index, This is the index value of the number of feature maps in the four feature map groups output by the four residual blocks, where 1:N represents the number of feature maps from feature map 1 to feature map N.
3. The infrared and visible light image fusion method based on ResNet50 and dual pyramids according to claim 1, characterized in that, Step B includes: In the feature map group of residual blocks In the middle, a window sliding block selection method is used. The window is arranged into 25 N-dimensional row vectors, where N is the index value of the number of feature maps in the four feature map groups output by the four residual blocks. The l1-norm of each vector is calculated, which is the sum of the absolute values of all elements in the vector, resulting in 25 l1-norm values. Then, the average of these 25 l1-norm values is calculated to obtain a pixel value. With a window step size of 1, after traversing the entire feature map group, the activity measure map of the residual blocks is obtained. Using this method, activity measure maps of four residual blocks can be obtained, represented as follows: Activity measurement diagram For the first row vector, This is the second row vector, and so on. For the 25th row vector, These are the l1-norm values of each row vector.
4. The infrared and visible light image fusion method based on ResNet50 and dual pyramids according to claim 1, characterized in that, Step C includes: Activity measure plots of each residual block using bilinear interpolation algorithm Upsampling is performed to obtain the initial salient feature map of each residual block, represented as follows: The interpolation result is represented by pixels. The calculation formula for the bilinear interpolation algorithm is: ； In the formula, The coordinates of the point to be interpolated. Points to be interpolated Pixel value at that location, , , and Points to be interpolated The coordinates of the four adjacent points are given, and their pixel values are respectively , , and After all interpolation points have been interpolated, the initial salient feature map of each residual block is obtained, represented as follows: Its size is similar to that of the normalized source image. The dimensions are consistent.
5. The infrared and visible light image fusion method based on ResNet50 and dual pyramids according to claim 1, characterized in that, Step D includes: In normalized source image In the middle, a window sliding block selection method is used. For a given window with a sliding step of 1, calculate its spatial frequency using the following formula: ； In the formula For the kth Spatial frequency values of the window slider image. and They are respectively The row and column frequency values of the window slider image. For window indexing, , Normalized source image The formulas for calculating the number of window sliders, row frequency value, and column frequency value contained in the data are as follows: ， ； In the formula and For respectively the first indivual The row and column frequency values of the window slider image. , and for The pixel coordinates within the window slider. , and They are respectively Pixel coordinates within the window slider , and Pixel value at; When the window slider traverses the entire normalized source image Then, a spatial frequency map reflecting the sharpness of the normalized source image is obtained. Then the spatial frequency diagram Initial salient feature map of each residual block Performing a dot product operation yields the salient feature map of each residual block, represented as follows: The calculation formula is as follows: ； In the formula , and Normalized source images Spatial frequency plot, initial salient feature plot of each residual block, and salient feature plot of each residual block.
6. The infrared and visible light image fusion method based on ResNet50 and dual pyramids according to claim 1, characterized in that, Step E includes: Saliency maps of each residual block Gaussian pyramid decomposition is performed separately to obtain the salient feature maps of each residual block on the Gaussian pyramid images of different decomposition layers. The Softmax operator is used to obtain the Gaussian pyramid weight maps of each residual block on different decomposition layers. Saliency map of each residual block The Gaussian pyramid decomposition process is represented in pixel output form as follows: ； ； In the formula, This is the salient feature map of the b-th residual block. The image of the 0th layer of the Gaussian pyramid is the salient feature map of the b-th residual block. The Gaussian kernel weight matrix is... , For decomposition layer index, , The maximum number of decomposition levels. This indicates rounding down, where H and W are the row and column values of the decomposed image, respectively. This indicates taking the minimum value. Let be the pixel coordinates of the salient feature map of the b-th residual block in the Gaussian pyramid image at layer l; In the Gaussian pyramid decomposition process, the salient feature maps of the residual blocks are first... As the image of the 0th layer of the Gaussian pyramid, it is represented as The Gaussian pyramid image is obtained by convolving it with a Gaussian kernel weight matrix, and then downsampling the convolutional image to remove even-numbered rows and columns, resulting in the first-level Gaussian pyramid image, represented as follows. Using this image as input, repeated convolution and downsampling operations yield the second-layer Gaussian pyramid image, represented as... This process is repeated L times, forming an L+1-layer pyramid-shaped image structure with progressively halved resolution, called a Gaussian pyramid. The Gaussian pyramid image of the salient feature map of each residual block is represented as follows: ; Gaussian pyramid images of each residual block The Gaussian pyramid weight map of each residual block is generated using the Softmax operator, and is represented as follows: The calculation formula is as follows: ， ； In the formula , , and These are, respectively, the Gaussian pyramid image of the infrared image of the l-th layer in the b-th residual block, the Gaussian pyramid image of the visible light image of the l-th layer in the b-th residual block, the Gaussian pyramid weight map of the infrared image of the l-th layer in the b-th residual block, and the Gaussian pyramid weight map of the visible light image of the l-th layer in the b-th residual block.
7. The infrared and visible light image fusion method based on ResNet50 and dual pyramids according to claim 1, characterized in that, Step F includes: Normalized source image Perform Laplacian pyramid decomposition to obtain the normalized source image. Laplacian pyramid images at different decomposition levels. The decomposition process of the Laplacian pyramid includes two parts: Gaussian pyramid decomposition and prediction residuals. Normalized source image. The Gaussian pyramid decomposition process is represented in pixel output form as follows: ； ； In the formula, For the normalized source image, Normalized source image Image of the 0th level of the Gaussian pyramid. Normalized source image The pixel coordinates of the l-th layer Gaussian pyramid image. The l-th layer Gaussian pyramid image of the normalized source image in Pixel value at that location, Normalized source image The pixel coordinates of the (l-1)th layer of the Gaussian pyramid image. Normalized source image Image of the L-1 level of the Gaussian pyramid Pixel value at; During the decomposition process, the source image is first normalized. As the image of the 0th layer of the Gaussian pyramid, it is represented as The Gaussian pyramid image is obtained by convolving it with a Gaussian kernel weight matrix, and then downsampling the convolution result to remove even-numbered rows and columns. Using this image as input, repeated convolution and downsampling operations are performed to obtain the second-layer Gaussian pyramid image. This process is repeated L times to form an L+1-layer pyramid-shaped image structure with progressively halved resolution, called a Gaussian pyramid, which normalizes the source image. The Gaussian pyramid image is represented as follows; The algorithm for predicting residuals is as follows: ； ； In the formula, Normalized source image Image of the Lth layer of the Gaussian pyramid. The L-th layer Laplacian pyramid image is a normalized source image. Normalized source image The pixel coordinates of the l-th layer Laplacian pyramid image and Gaussian pyramid image. Normalized source image The image of the lth layer of the Laplace pyramid in Pixel value at that location, The l-th layer Gaussian pyramid image of the normalized source image in Pixel value at that location, Normalized source image The pixel coordinates of the (l+1)th layer of the Laplacian pyramid image. Normalized source image The image of the l+1 level of the Laplace pyramid is in Pixel value at; In the process of predicting residuals, the source image is first normalized. Image of the Lth level of the Gaussian pyramid As the Lth level Laplacian pyramid image, it is represented as The image is upsampled, and then convolved with a Gaussian kernel weight matrix. The L(1)th layer Gaussian pyramid image is then subtracted from the convolved image to obtain the L(1)th layer Laplacian pyramid image, denoted as... Using this image as input, the upsampling, convolution, and subtraction operations are repeated to obtain the (L-2)th layer Laplacian pyramid image, denoted as... This process is repeated L times to form an L+1-layer pyramid-shaped image structure with progressively doubling resolution, known as the Laplacian pyramid, which normalizes the source image. The Gaussian pyramid image is represented as .
8. The infrared and visible light image fusion method based on ResNet50 and dual pyramids according to claim 1, characterized in that, Step G includes: The Laplacian pyramid image of the normalized source image is weighted and fused using the Gaussian pyramid weight map of each residual block to obtain the fused Laplacian pyramid image of each residual block. The calculation formula is as follows: ； In the formula and The l-th layer Gaussian pyramid weight map of the b-th residual block is used to fuse the l-th layer Laplacian pyramid image of the normalized source image from the infrared and visible light images. This is the l-th layer Laplacian pyramid image of the infrared image in the normalized source image. This is the l-th layer Laplacian pyramid image of the visible light image in the normalized source image. This is the merged image of the l-th layer of the b-th residual block's Laplacian pyramid.
9. The infrared and visible light image fusion method based on ResNet50 and dual pyramids according to claim 1, characterized in that, Step H includes: Laplacian pyramid fused image of each residual block using inverse Laplacian pyramid transform Reconstruction is performed to obtain the fused image of each residual block, and then the pixel enlargement algorithm is used to obtain the final fused image FF; The Laplace pyramid inverse transform algorithm is represented in pixel-wise form as follows: ； ； In the formula, This is the fused Laplacian pyramid image of the Lth layer of the b-th residual block. This is the inverse Laplacian pyramid transform image of the Lth layer of the b-th residual block. Let be the pixel coordinates of the fused Laplacian pyramid image and the inverse Laplacian pyramid image of the b-th residual block at layer l. The Laplacian pyramid fusion image of the l-th layer of the b-th residual block in Pixel value at that location, The inverse Laplacian pyramid image of the l-th layer of the b-th residual block in Pixel value at that location, Let be the pixel coordinates of the (l+1)th layer of the Laplacian pyramid image of the b-th residual block. For the image of the (l+1)th layer of the b-th residual block, in Pixel value at; In the inverse Laplacian transform process, the Lth-level Laplacian pyramid inverse transform image of each residual block is first upsampled. Then, a Gaussian kernel weight matrix is used to convolve it. The convolution result image is added to the (L-1)th-level Laplacian pyramid fused image to obtain the (L-1)th-level Laplacian pyramid inverse transform image. This image is used as input, and the upsampling, convolution, and summation operations are repeated to obtain the (L-2)th-level Laplacian pyramid inverse transform image. This process is repeated L times to obtain the 0th-level Laplacian pyramid inverse transform image of each residual block, denoted as... This is the fused image of each residual block; fused image of each residual block The final fused image is obtained by using a pixel-maximizing algorithm. The calculation formula is as follows: ； In the formula, The operation is to find the maximum value for each pixel. For the final merged image, These are the fused images of the first, second, third, and fourth residual blocks, respectively.