An image fusion method and system based on image decomposition twin network
By decomposing an image into basic and detailed parts using an image decomposition Siamese network, and extracting and fusing features separately, the problem of poor image fusion quality in existing technologies is solved, and higher quality image fusion is achieved.
Patent Information
- Application Number
- CN202211582823.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-09
- Publication Date
- 2025-11-21
- Estimated Expiration
- 2042-12-09
AI Technical Summary
In existing deep learning-based image fusion methods, the feature extraction network model reduces the resolution, resulting in poor quality of the fused image and failing to effectively preserve specific information from images from different sources.
An image decomposition Siamese network is used to decompose the image into basic and detailed parts. Features are extracted by the Siamese network and VGG-16 network respectively, and then fused using channel attention and block averaging strategies to reconstruct the fused image.
It improves the quality of fused images, retains more image information, and enhances feature extraction and fusion effects.
Smart Images
Figure CN115797242B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of computer vision, and more particularly, relates to an image fusion method and system based on image decomposition twin network. BACKGROUND
[0002] With the rapid development of sensor technology, different types of sensors such as visible light, infrared, and laser radar are widely used in monitoring and warning, intelligence reconnaissance, target tracking systems on ground, vehicle, airborne, and satellite platforms. Fusing data from different types of sensors to obtain comprehensive information about targets and scenes and improve system performance is one of the important research directions.
[0003] In traditional image fusion methods, in order to ensure the feasibility of subsequent feature fusion, the same transformation is often used to extract features from different source images. Since this step does not take into account the feature differences between different source images, it may result in poor representation of the extracted features. On the other hand, the feature fusion strategy in traditional methods is too rough, limiting the fusion performance. Deep learning-based image fusion methods can better overcome these limitations in traditional methods. Deep learning-based methods can achieve differentiated feature extraction through different network branches, thereby obtaining more targeted features. Secondly, deep learning-based methods can learn more reasonable feature fusion strategies under the guidance of carefully designed loss functions, thereby achieving adaptive feature fusion. Thanks to these advantages, deep learning has promoted the development of image fusion technology and achieved better performance than traditional image fusion methods. However, the lack of real fusion images as ground truth for different source image fusion makes it difficult for deep learning-based image fusion methods to use supervised learning for training, and the loss function cannot effectively guide the training process, making it difficult to effectively preserve unique information in different source images. In addition, convolutional neural networks commonly used for image feature extraction contain pooling down-sampling operations, which reduce the resolution of features, and direct application in image fusion can lead to a decrease in the quality of the fused image. Therefore, designing appropriate deep network models, loss functions, and training methods for image fusion problems while preserving information in different source images and improving the quality of the fused image remains a problem to be solved. SUMMARY
[0004] To address the above deficiencies and / or improve improvements of prior art, the present application provides an image fusion method and system based on image decomposition twin network, which aims to solve the technical problems of existing deep learning-based image fusion methods that commonly use feature extraction network models that reduce resolution, resulting in poor quality of the fused image and the inability to effectively preserve specific information in different source images.
[0005] To achieve the above object, according to one aspect of the present application, an image fusion method based on image decomposition twin network is provided, comprising:
[0006] S1. Decomposing two frames of images to be fused into image basic parts and image detail parts respectively;
[0007] S2. Inputting the two frames of image basic parts into a twin network to obtain an activity level map of the image basic parts, performing channel attention processing on the activity level map to obtain two frames of image basic part weight maps, and fusing the two frames of image basic parts by using the image basic part weight maps to obtain an image basic part fusion result; the twin network comprises two branches; each branch comprises a plurality of convolution layers and a plurality of convolution layers with skip connection connected in series, and the feature maps output by the two branches are combined to obtain the activity level map of the image basic parts;
[0008] S3. Performing multi-layer feature extraction and fusion on the image detail parts to obtain an image detail part fusion result;
[0009] S4. Adding the image basic part fusion result and the image detail part fusion result to reconstruct a fusion image.
[0010] Further, S1 comprises: performing low-pass filtering on the two frames of images by using a Tikhonov regularization method respectively to obtain the image basic parts, and the calculation formula is:
[0011]
[0012] For the decomposed image basic parts, k is the sequence number of the image to be fused, I k is the source image k, g x and g y are gradient operators in horizontal and vertical directions respectively, ‖·‖F represents F-norm, and the parameter γ is used to balance the residual term and the penalty term;
[0013] The image detail parts are calculated by using the formula
[0014] For the decomposed image detail parts, I k is the source image k, is the decomposed image basic part, and k is the sequence number of the image to be fused.
[0015] Further, the calculation process of the channel attention processing in step S2 is as follows:
[0016] F' = Sigmoid(W fc2 (W fci (AveragePool(F)))) ⊙ F
[0017] where F' is the image base part weight map, F is the image base part activity level map, AveragePool(*) is the global average pooling layer processing, W fc 1(*) is the first linear layer processing, W fc2 (*) is the second linear layer processing, Sigmoid(*) is the Sigmoid activation function, and is the Hadamard product operation.
[0018] Further, the fusion process of the image base part weight map and the two image base parts is as follows:
[0019]
[0020] where F b (x,y) is the image base part fusion result at the coordinate position (x,y), F'(x,y) is the image base part weight value at the coordinate position (x,y), and respectively represent the values of the two image base parts to be fused at the coordinate position (x,y).
[0021] Further, the step S3 comprises:
[0022] The two image detail parts are respectively input into a VGG-6 network to extract multi-layer features of the two image detail parts;
[0023] The L1 norm method is used to calculate the multi-layer initial activity level maps of the two image detail parts from the multi-layer features;
[0024] A block average strategy is used to calculate the multi-layer activity level maps of the two image detail parts;
[0025] Softmax transformation and up-sampling are used to obtain the multi-layer weight maps of the two image detail parts;
[0026] The two image detail parts are weighted using the multi-layer weight maps of the two image detail parts to obtain initial detail part fusion results at different layers;
[0027] The maximum selection strategy is used to fuse the initial detail part fusion results to obtain a final image detail part fusion result.
[0028] Further, the fusion formula of the multi-layer weight maps of the two image detail parts and the two image detail parts is as follows:
[0029]
[0030] where represents the i-th layer image detail part fusion result at the coordinate position (x,y), a detail part of an image to be fused k representing a coordinate position (x, y), an i-th layer weight value of the image to be fused k representing a coordinate position (x, y), k being an image serial number.
[0031] The application further provides an image fusion system based on an image decomposition twin network, comprising:
[0032] an image decomposition module, which decomposes two images to be fused into image basic parts and image detail parts respectively;
[0033] a basic part fusion module, which inputs the two image basic parts into a twin network to obtain an activity level map of the image basic part, performs channel attention processing to obtain two image basic part weight maps, and fuses the two image basic parts by using the image basic part weight maps to obtain an image basic part fusion result; the twin network comprises two branches; each branch comprises a plurality of convolution layers and a plurality of convolution layers with skip connections connected in series, and the feature maps output by the two branches are combined to obtain the activity level map of the image basic part;
[0034] a detail part fusion module, which performs multi-layer feature extraction and fusion on the image detail parts to obtain an image detail part fusion result;
[0035] an image reconstruction module, which adds the image basic part fusion result and the image detail part fusion result to reconstruct a fused image
[0036] Overall, compared with the prior art, the above technical solutions conceived by the application can achieve the following beneficial effects.
[0037] (1) The application designs a twin network fusion algorithm for the image basic part, and the twin network structure comprises two branches, each branch is composed of a plurality of convolution layers without pooling downsampling and a plurality of convolution layers with skip connections connected in series, low-level features and high-level features of the two image basic parts are extracted respectively, the information of the image basic part after feature extraction is improved, a channel attention module is further added to enhance the features, the quality of the image basic part fusion result is improved, and the quality of the fused image is improved.
[0038] (2) The application uses VGG-16 to extract multi-layer features of the image, adopts a block average strategy, a softmax transformation and an upsampling to obtain multi-layer weights for fusing the multi-layer features respectively, and processes the obtained multi-layer fusion features by a maximum selection strategy to obtain the fusion result of the image detail part. The multi-layer feature fusion process based on the maximum selection strategy is beneficial to retaining the image detail information and improving the quality of the fused image. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 The flow chart of the image fusion method based on the image decomposition twin network provided by the embodiment of the present application is shown in
[0040] Figure 2 The architecture schematic diagram of the image fusion method based on the image decomposition twin network provided by the embodiment of the present application is shown in
[0041] Figure 3 The network structure schematic diagram of the basic part of the fused image provided by the embodiment of the present application is shown in
[0042] Figure 4 The network structure schematic diagram of the detail part of the fused image provided by the embodiment of the present application is shown in DETAILED DESCRIPTION
[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.
[0044] Figure 1 The flow chart of the image fusion method based on the image decomposition twin network provided by the embodiment of the present application is shown in Figure 1 , in combination with Figures 2-4 , the image fusion method based on the image decomposition twin network in the present embodiment will be described in detail, which comprises steps S1-S4.
[0045] In step S1, two frames of images to be fused are respectively decomposed into image basic parts and image detail parts. The Tikhonov regularization method is used for low-pass filtering on the two frames of images respectively to obtain the image basic parts, and the calculation formula is as follows:
[0046]
[0047] wherein, is the decomposed image basic part, k is the image sequence number to be fused, taking 0 or 1, I k is the source image k, g x = [-1 1], g y = [-1 1] T is the gradient operator in horizontal and vertical directions, ‖·‖ F F-norm, and the parameter γ is used to balance the residual term and the penalty term, taking 5.
[0048] The image detail part is calculated by the following formula:
[0049] The image detail part is calculated by the following formula:
[0050] in, For the decomposed image details, I k Let k be the source image. is the basic part of the decomposed image, and k is the index of the image to be fused, which can be 0 or 1.
[0051] Step S2: Input the basic parts of the two frames of images into the Siamese network to obtain the activity level map of the basic parts of the images, and then process the channel attention to obtain the weight map of the basic parts of the images. Use the weight map of the basic parts of the images to fuse the basic parts of the two frames of images to obtain the image basic parts fusion result.
[0052] According to an embodiment of the present invention, step S2 includes sub-steps S21-S23.
[0053] In sub-step S21, the basic parts of the two frames of images are divided into 16×16 image blocks with a step size of 2, and the image blocks are input into the Siamese network, as follows. Figure 3 As shown. See also Figure 3 The Siamese network comprises two branches, each with four convolutional layers. Each convolutional layer uses a 3×3 kernel with a stride of 1, and the activation function is ReLU. The first two convolutional layers have 32 and 64 kernels respectively, while the last two are convolutional layers with skip connections, each with 64 kernels. The feature maps output from the two branches are merged to obtain the activity level map of the basic image portion. In both branches of the Siamese network, no pooling downsampling operation is performed after each convolutional layer to maintain the resolution of the basic image features. Skip connections are then used to combine low-level features with high-level features, thus preserving more information from the original image's basic portion in the output activity level map.
[0054] In sub-step S22, channel attention processing is further applied to the activity level map of the basic image part output by the Siamese network to obtain the image basic part weight map.
[0055] In this embodiment, see Figure 3 In the channel attention process, firstly, a global average pooling step is performed on the activity level map of the basic image part. Next, two fully connected layers are used to learn the correlation between channels. Then, the sigmoid activation function is used to obtain weight coefficients. Finally, the weight coefficients are multiplied with the activity level map of the basic image part to obtain the image basic part weight map. The calculation process of channel attention is as follows:
[0056] F′=Sigmoid(W fc 2(W fci (AveragePool(F))))⊙F
[0057] where F' is the image base part weight map, F is the image base part activity level map, AveragePool(*) is the global average pooling layer processing, W fc 1(*) is the first linear layer processing, W fc2 (*) is the second linear layer processing, Sigmoid(*) is the Sigmoid activation function, and is the Hadamard product operation.
[0058] In the sub-step S23, the image base part fusion result is obtained by fusing the two image base parts obtained in the step S1 using the obtained image base part weight map. The fusion process is calculated as follows:
[0059]
[0060] where F b (x,y) is the image base part fusion result at the coordinate position (x,y), F'(x,y) is the image base part weight value at the coordinate position (x,y), and respectively represent the value of the base part of the image 0 to be fused and the image 1 to be fused at the coordinate position (x,y).
[0061] In the step S3, the two image detail parts obtained in the step S1 are respectively input into the VGG-6 network to extract the multi-layer features of the two image detail parts, the block average strategy is used to calculate the multi-layer activity level maps of the two image detail parts, the Softmax transformation and the up-sampling are used to obtain the multi-layer weight maps of the two image detail parts, the multi-layer image detail part fusion results are obtained by fusing the two image detail parts using the multi-layer weight maps of the two image detail parts, and the image detail part fusion result is obtained by fusing the multi-layer image detail part fusion results using the maximum selection strategy.
[0062] According to the embodiments of the present application, the step S3 includes the sub-steps S31-S33.
[0063] In the sub-step S31, referring to Figure 4 , the two image detail parts obtained in the step S1 are respectively input into two VGG-16 networks, the feature maps output by the 4 ReLU layers in the VGG-16 network are extracted to form the multi-layer features of the two image detail parts, and the feature dimensions of the feature maps are 64, 128, 256 and 512 respectively.
[0064] The multi-layer initial activity level maps of the two image detail parts are respectively calculated using the L1 norm method, and the calculation formula is as follows:
[0065]
[0066] where F an initial activity level map of the i-th layer of the detail part of the to-be-fused image k at the coordinate position (x, y), M is a feature dimension, M = 64 x 2 i-1 , i takes values of 1, 2, 3 and 4, k is a to-be-fused image detail part sequence number, taking values of 0 or 1, and ||·||1 represents an L1 norm.
[0067] The multi-layer activity level maps of the detail parts of the two frames of images are calculated by using a block averaging strategy, and a calculation formula of the block averaging strategy is as follows:
[0068]
[0069] wherein, an initial activity level map of the i-th layer of the detail part of the to-be-fused image k at the coordinate position (x, y), an initial activity level map of the i-th layer of the detail part of the to-be-fused image k at the coordinate position (x+β, y+α), s is an average block size parameter, the size of the average block is (2s+1) x (2s+1), s takes a value of 1, k is a to-be-fused image sequence number, taking values of 0 or 1, and i takes values of 1, 2, 3 and 4.
[0070] In the sub-step S32, referring to Figure 4 , a Softmax step is used to obtain the initial weight maps of each layer of the activity level maps of the detail parts of the two frames of images obtained in the S31, and a calculation formula is as follows:
[0071]
[0072] wherein, an initial weight value of the i-th layer of the detail part of the to-be-fused image k at the coordinate position (x, y), an initial activity level map of the i-th layer of the detail part of the to-be-fused image k at the coordinate position (x, y), k is an image sequence number, taking values of 0 or 1, and i takes values of 1, 2, 3 and 4.
[0073] After upsampling of the initial weight maps of each layer, the multi-layer weight maps of the image detail parts are obtained. An upsampling formula is as follows:
[0074]
[0075] wherein, a weight value of the i-th layer of the detail part of the to-be-fused image k at the coordinate position (x+p, y+q), an initial weight value of the i-th layer of the detail part of the to-be-fused image k at the coordinate position (x, y), k is an image sequence number, taking values of 0 or 1, and i takes values of 1, 2, 3 and 4.
[0076] In sub-step S33, the multi-layer weight map of the image detail part obtained in S32 is used to fuse the two image detail parts obtained in S1 respectively to obtain a multi-layer image detail part fusion result. The fusion calculation formula is as follows:
[0077]
[0078] wherein, represents the i-th layer image detail part fusion result at coordinate position (x, y), represents the detail part of image k to be fused at coordinate position (x, y), represents the i-th layer weight value of image k to be fused at coordinate position (x, y), k is the image sequence number, taking values of 0 or 1, and i takes values of 1, 2, 3 and 4.
[0079] The multi-layer image detail part fusion result is fused by using the maximum strategy to obtain the image detail part fusion result. The calculation formula of the maximum strategy fusion is as follows:
[0080]
[0081] wherein, F d (x, y) represents the image detail part fusion result at coordinate position (x, y), represents the i-th layer image detail part fusion result at coordinate position (x, y).
[0082] Step S4, the image basic part fusion result obtained in S2 is added to the image detail part fusion result obtained in S3 to reconstruct a fusion image. The calculation formula of the reconstruction is as follows:
[0083] F(x, y) = F d (x, y) + F b (x, y)
[0084] wherein, F(x, y) is the fusion result at coordinate position (x, y), F d (x, y) is the image detail part fusion result at coordinate position (x, y), and F b (x, y) is the image basic part fusion result at coordinate position (x, y).
[0085] In order to verify the image fusion result of the image fusion method based on the image decomposition twin network in the embodiment of the present application, a data set for image fusion is constructed based on the TNO image fusion data set. 50 pairs of infrared and visible light images are selected from the TNO image fusion data set, which are divided into a plurality of image blocks with a size of 16x16 with a step of 2, generating 400,000 training samples. When p1 comes from the infrared image and p2 comes from the corresponding visible light image at the same position, the label of {p1, p2} is set to 1 as a positive example, and otherwise the label of {p1, p2} is set to 0 as a negative example. 80% of them are selected as training samples, and the remaining 20% are selected as verification samples.
[0086] After the data set is prepared, model training is needed. When training the network model of the image basic part fusion based on the twin network, the cross-entropy loss function and the stochastic gradient descent optimization method are selected, the batch data size is set to 128, the model iteration number is 25000, and the learning rate is 0.0001.
[0087] The trained network is used to test 7 pairs of infrared and visible light images selected from the TNO data set, and other several classical image fusion methods are also used for testing. The average values of 8 performance indicators such as information entropy (EN), correlation coefficient (CC), mutual information (MI), standard deviation (SD), average gradient (AG), mean square error (MSE), difference correlation and (SCD) and peak signal-to-noise ratio (PSNR) are compared, and the experimental results are shown in Table 1.
[0088] Table 1 Comparison of average performance indicators obtained by various image fusion methods
[0089] Method EN CC MI SD AG MSE SCD PSNR CBF 6.7343 0.4799 13.4686 0.1268 0.0193 0.0256 1.2951 63.6399 JSR 6.7490 0.5553 13.4980 0.1500 0.0181 0.0615 1.6047 60.3482 DCHWT 6.7654 0.5335 13.5209 0.1242 0.0143 0.0249 1.3141 63.9527 GTF 6.7755 0.4784 13.5510 0.1358 0.0137 0.0300 0.9289 63.6970 MEFGAN 6.4260 0.5382 12.8521 0.1113 0.0088 0.0347 1.0914 62.9152 FusionGAN 6.4260 0.5382 12.8521 0.1113 0.0088 0.0347 1.0914 62.9152 The present embodiments 6.7510 0.6399 13.7469 0.1321 0.0155 0.0189 1.6413 65.9007
[0090] As can be seen from the experimental results in Table 1, among the 8 performance indicators, the present embodiment is optimal in 5 indicators such as correlation coefficient (CC), mutual information (MI), mean square error (MSE), difference correlation and (SCD) and peak signal-to-noise ratio (PSNR), and is also the method with the most optimal indicators among all methods. The present embodiment has certain advantages compared with other image fusion methods.
[0091] It is easy for those skilled in the art to understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. An image fusion method based on image decomposition twin network, characterized in that, Comprising: S1. decomposing two images to be fused into image basic parts and image detail parts respectively; S2. inputting the two image basic parts into a twin network to obtain an activity level map of the image basic parts, performing channel attention processing to obtain a two image basic part weight map, and fusing the two image basic parts by using the image basic part weight map to obtain an image basic part fusion result; the twin network comprises two branches; each branch comprises a plurality of convolution layers connected in series and a plurality of convolution layers with skip connection, and the feature maps output by the two branches are merged to obtain the activity level map of the image basic parts; S3. performing multi-layer feature extraction and fusion on the image detail parts to obtain an image detail part fusion result; S4. adding the image basic part fusion result and the image detail part fusion result to reconstruct a fused image; S1 comprises: using a Tikhonov regularization method to perform low-pass filtering on the two images respectively to obtain image basic parts, and the calculation formula is: for the decomposed image base part, for the image to be fused, for the source image , and are gradient operators in horizontal and vertical directions, respectively, denotes the F-norm, and parameter is used to balance the residual term and the penalty term; The image detail portion is calculated using the formula ImageDetailPortion = ImageDetailPortion + (ImageDetailPortion * DetailPortion for the decomposed image detail part, for the source image , for the decomposed image base part, is the image sequence number to be fused; The calculation process of the channel attention processing in step S2 is as follows: wherein, is an image base part weight map, is an activity level map of the image base part, is a global average pooling layer processing, is a first linear layer processing, is a second linear layer processing, is a Sigmoid activation function, is a Hadamard product operation; The fusion process of the image basic part weight map and the two image basic parts is as follows: wherein is a coordinate position of the image base portion fusion result, is a coordinate position of the image base portion weight value, and respectively represent values of two image base portions of the frames to be fused at the coordinate position . 2.The image fusion method based on image decomposition twin network according to claim 1, characterized in that, Step S3 comprises: The two image detail parts are input into a VGG-6 network to extract multi-layer features of the two image detail parts; An L1 norm method is used to calculate multi-layer initial activity level maps of the two image detail parts; A block average strategy is used to calculate multi-layer activity level maps of the two image detail parts; Softmax transformation and up-sampling are used to obtain multi-layer weight maps of the two image detail parts; The multi-layer weight maps of the two image detail parts are used to weight the two image detail parts to obtain initial detail part fusion results at different layers; A maximum selection strategy is used to fuse the initial detail part fusion results to obtain a final image detail part fusion result. 3.The image fusion method based on image decomposition twin network according to claim 2, characterized in that, The multi-layer weight map of the two image detail parts and the two image detail parts are fused according to the following formula: wherein, denotes a coordinate position of a first layer image detail portion fusion result, denotes a coordinate position of a detail portion of an image to be fused, denotes a coordinate position of a first layer weight value of an image to be fused, is an image sequence number.
4. An image fusion system based on image decomposition twin network, characterized in that, Comprising: An image decomposition module decomposes two images to be fused into image basic parts and image detail parts respectively; An image basic part fusion module inputs the two image basic parts into a twin network to obtain an activity level map of the image basic parts, performs channel attention processing to obtain a two image basic part weight map, and fuses the two image basic parts by using the image basic part weight map to obtain an image basic part fusion result; the twin network comprises two branches; each branch comprises a plurality of convolution layers connected in series and a plurality of convolution layers with skip connection, and the feature maps output by the two branches are merged to obtain the activity level map of the image basic parts; An image detail part fusion module performs multi-layer feature extraction and fusion on the image detail parts to obtain an image detail part fusion result; An image reconstruction module adds the image basic part fusion result and the image detail part fusion result to reconstruct a fused image; Wherein, decomposing two images to be fused into image basic parts and image detail parts comprises: The Tikhonov regularization method is used for low-pass filtering on two frames of images respectively to obtain image basic parts, and a calculation formula is as follows: is the decomposed image basic part, is the image sequence number to be fused, is the source image , and are the gradient operators in horizontal and vertical directions, respectively, ||·||F F denotes the F-norm, parameter is used to balance the residual term and the penalty term; The image detail portion is calculated using the formula ImageDetailPortion = ImageDetailPortion + (ImageDetailPortion * DetailPortion for the decomposed image detail part, for the source image , for the decomposed image base part, is the image sequence number to be fused; The calculation process of the channel attention processing is as follows: wherein, is an image base part weight map, is an activity level map of the image base part, is a global average pooling layer processing, is a first linear layer processing, is a second linear layer processing, is a Sigmoid activation function, is a Hadamard product operation; The fusion process of the image basic part weight map and the two frames of image basic parts is as follows: wherein, is a coordinate position of the image base portion fusion result, is a coordinate position of the image base portion weight value, and respectively represent values of two frames of image base portions to be fused at the coordinate position .
5. A computer readable medium, on which a computer program is stored, the computer program being executed by a processor to implement the steps of the method according to any one of claims 1-3.