An optimization-based dual-branch feature pyramid multispectral image fusion framework

Abstract

The application belongs to the field of image processing, and proposes a multispectral image fusion framework of an optimized double-branch feature pyramid, aiming to solve the problems of unnatural color and contrast mismatch that may occur when fusing infrared and visible light images under dark light conditions. First, a multi-scale residual module is used to strengthen feature extraction, and a Canny filter is integrated to capture feature information. Second, shallow and deep fusion modules are used to realize information fusion, and a weighted loss guide process of gradient and contrast is designed. Finally, a residual connection feature reconstruction module is used to splice and reconstruct the features of the fused image. The application can effectively realize the fusion of infrared and visible light images based on low light scenes, has excellent feature extraction capability and high-quality visual fusion effect, and can ensure high contrast, rich background and other features.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of infrared and visible light image fusion technology in low light scenes, and in particular to an optimized dual-branch feature pyramid multispectral image fusion framework. BACKGROUND

[0002] Image acquisition in low light scenes is a key challenge in the field of computer vision and image processing. In this case, the quality of visible light images is constrained by lighting conditions, making the target objects in the images unclear and difficult to identify. On the contrary, infrared light images, as a substitute for heat-sensitive and natural light-independent means, have significant advantages. However, the lack of color information in infrared light images alone limits their application in visual recognition tasks. Therefore, fusing visible and infrared light images is an effective technology to improve image quality and enhance target detection and recognition performance. In addition, infrared and visible light fusion technology can help overcome difficulties in low light and heavy fog conditions, and has wide application value in night vision, military, monitoring, search and rescue, etc. The core goal of image fusion technology is to combine image information from the same or different wavebands organically, thereby providing more abundant and more conducive to visual analysis information in the final fusion image. In order to achieve this goal, researchers have continuously developed and landed a variety of fusion technologies and algorithms. In traditional methods, a lot of effort is needed for artificial design of fusion rules. These methods have low generalization and long development cycle, high cost, and poor fusion effect in the face of complex background, especially low light. In recent years, deep learning-based methods have emerged, especially represented by convolutional neural networks, which are widely used in the field of image fusion. These deep learning methods learn the feature representation of different waveband images by training the network, and cleverly fuse them to obtain the fusion image. This progress has shown colorful potential in dealing with the challenges of low light conditions, especially in solving color unnaturalness and contrast mismatch. These methods learn the feature representation of different waveband images by training deep learning networks, and fuse them together. In this way, they can better handle the challenges of low light conditions, such as color unnaturalness and contrast mismatch. In addition, some research methods also introduce semantic, gradient and contrast loss functions to guide the fusion process, ensuring better preservation and enhancement of information in the image. Such technological innovation brings new vitality to the field of image fusion, enabling us to more comprehensively address visual challenges in low light conditions and provide better image fusion solutions for various application fields.

[0003] Despite the significant progress made in deep learning-based image fusion methods, there are still some technical challenges that need to be addressed at the current stage. First, the feature extraction capability of existing methods is relatively weak, and further research is needed to improve the quality and accuracy of feature extraction. In addition, the design of the loss function is usually relatively simple, lacking adaptability to complex scenes, which limits the quality of the fusion results. In practice, the fusion results often fail to meet the high-quality visual requirements, so more methods are needed to improve this aspect. These problems have become bottlenecks for deep learning image fusion methods, and more research and innovation are needed to overcome them. By fully exploiting the potential of deep learning and further improving the structure of the fusion model and the design of the loss function, it is hoped that these problems can be overcome to achieve better image fusion results. This will provide higher levels of visual information for various applications and push image fusion technology to new heights.

[0004] The purpose of the present application is to overcome the shortcomings of the prior art, and we provide a dual-branch feature pyramid-based optimization-based multispectral image fusion framework. This fusion method has excellent feature extraction capability, which can enhance the dominant features of different modal images, thereby achieving high-quality fused image generation. These fused images contain high-contrast texture details and rich scene information, meeting the high-standard requirements of multi-modal image fusion.

[0005] The technical solution of the present application to solve the above technical problems is:

[0006] A dual-branch feature pyramid-based optimization-based multispectral image fusion framework, comprising the following steps:

[0007] S1, simultaneously shooting 500 pairs of dual-modal images with infrared and visible light cameras to form a set of dual-modal images to be registered G, and the registered image set is H and the two modal images are denoted as I1 and I2;

[0008] S2, the infrared and visible light images are respectively taken as I1 and I2 as network input, and the relevant features are captured through the feature extraction module FEM and ;

[0009] S3, the features and are passed through the shallow fusion module CFM, the information proportion of the features and is adjusted in weight, and new features and are obtained;

[0010] S4, the obtained features , , and characteristics are added to obtain characteristics , and characteristics are taken as the input of the next feature extraction module;

[0011] S5, characteristics are input into a deep fusion module AFM to adjust the information proportion of characteristics and by weight, and obtain characteristics and ;

[0012] S6, characteristics and are added to obtain characteristics , and characteristics and are spliced as the input of the feature reconstruction sub-network;

[0013] S7, the feature extraction module FEM, the shallow fusion module CFM, the deep fusion module AFM and the feature reconstruction module FDM are combined into a multi-spectral image fusion framework based on an optimized double-branch feature pyramid, and the loss function Loss value between the output fusion image F and the input multi-modal images I1 and I2 is calculated. The image fusion framework is trained by back propagation to reduce the loss value;

[0014] S8, the double-modal image set H to be fused is put into the trained image fusion framework, and the fusion image F is obtained by testing.

[0015] Preferably, in step S1, the double-modal images include visible light images, polarized light images, and different waveband infrared light images.

[0016] Preferably, in step S2, the feature extraction module FEM is composed of convolution layers with convolution kernel sizes of 1×1 and 3×3, a Canny operator with a convolution kernel size of 3×3, a regularization BN layer and a nonlinear activation function LReLU jump connection, and is used to extract the features of the input image.

[0017] Preferably, in step S3, the shallow fusion module CFM has two modules, which are composed of a Canny operator with a convolution kernel size of 3×3, an average pooling layer, a deep convolution layer with a convolution kernel size of 3×3, and a Sigmoid activation function, and are used to output shallow fusion characteristics and adjust the feature extraction capability.

[0018] Preferably, in step S5, the deep fusion module AFM has two modules, which are composed of a Prewitt operator with a convolution kernel size of 3×3, an average pooling layer, a RepVGG, and a Sigmoid activation function, and are used to output the dominant information on the global level in the deep features.

[0019] Preferably, in step S6, the feature reconstruction sub-network adopts a feature reconstruction module FDM composed of convolution layers with convolution kernel sizes of 1*1 and 3*3 and a nonlinear activation function LReLU skip connection, for reconstructing a high-quality fusion image.

[0020] Preferably, in step S7, the loss function Loss is composed of a pixel-level loss L con and a gradient loss L adv , and is calculated by the following formula:

[0021]

[0022]

[0023]

[0024] wherein I f is a fusion image, I vi is a visible light image, I ir is a mid-wave infrared light image, refers to a Canny operator.

[0025] Compared with the prior art, the application has the following beneficial effects:

[0026] The multi-spectral image fusion framework based on the optimized double-branch feature pyramid of the application first obtains the features of the source images from the shallow layer to the deep layer by using a double-branch feature extraction module, wherein the feature encoding module is composed of convolution layers with convolution kernel sizes of 1*1 and 3*3, a Canny operator with a convolution kernel size of 3*3, a regularization BN layer and a nonlinear activation function LReLU skip connection. This simple combination can effectively retain more low-frequency information such as texture and background in the shallow layer features and more high-frequency information such as edges and contours in the deep layer features, and avoid excessive increase of redundant parameters, and better realize the extraction of the features of the double-mode images.

[0027] The multi-spectral image fusion framework based on the optimized double-branch feature pyramid of the application first extracts the feature maps of the i-th channel of different modal images by using a feature extraction module, and then processes the feature maps by using a fusion module from the shallow layer to the deep layer, one way of which is to adjust the extraction capability of the feature extraction module from a global perspective, and the other way of which is to input the feature maps as the input of a feature reconstruction module to reconstruct and restore a fusion image with comprehensive information.

[0028] The feature fusion method fuses the feature information of the original image, can fully integrate the infrared intensity and visible light edge information of different components on the channel, and can retain more fused fusion information.

[0029] The multispectral image fusion framework based on the optimized double-branch feature pyramid of the application adopts a pixel-level loss and a gradient loss function when constructing a loss function. The two loss functions constrain the similarity of the brightness, edges and background details of the fusion image to the source image, so that the fusion image not only has rich texture and detail information, but also maintains a high target contrast. BRIEF DESCRIPTION OF DRAWINGS

[0030] Fig. 1 is a flowchart of the multispectral image fusion framework based on the optimized double-branch feature pyramid of the application.

[0031] Fig. 2 is a structure of a feature extractor module FEM in the multispectral image fusion framework based on the optimized double-branch feature pyramid of the application.

[0032] Fig. 3 is a structure of a feature fusion module CFM in the multispectral image fusion framework based on the optimized double-branch feature pyramid of the application.

[0033] Fig. 4 is a structure of an attention feature fusion module AFM in the multispectral image fusion framework based on the optimized double-branch feature pyramid of the application.

[0034] Fig. 5 is a structure of a feature reconstruction module FDM in the multispectral image fusion framework based on the optimized double-branch feature pyramid of the application. DETAILED DESCRIPTION

[0035] The application will be further described in detail below in combination with embodiments, but the embodiments of the application are not limited thereto.

[0036] Referring to Fig. 1, the multispectral image fusion framework based on the optimized double-branch feature pyramid of the application includes the following steps:

[0037] S1, using cameras of different imaging modes to shoot 500 pairs of bimodal images to form a bimodal image set G to be registered, and performing rotation, sliding and cutting to augment the image set G, the number of the augmented image set G is 7260 groups, and the image set G is rigidly registered and grayed, and the two kinds of modal images after the above operations are denoted as I1 and I2 respectively;

[0038] S2, the infrared and visible light images are respectively taken as I1 and I2 as network inputs, and relevant features are captured through a feature extraction module FEM and ;

[0039] S3, the features and After the shallow fusion module CFM, the information proportion of the features and is adjusted in weight, to obtain new features and ;

[0040] S4, the obtained features , , and are added to obtain feature , and the feature is taken as the input of the next feature extraction module;

[0041] S5, the feature passes through the deep fusion module AFM, and the information proportion of the features and is adjusted in weight, to obtain features and ;

[0042] S6, the obtained features and are added to obtain feature , and the features and are spliced as the input of the feature reconstruction subnetwork;

[0043] S7, the feature extraction module FEM, the shallow fusion module CFM, the deep fusion module AFM and the feature reconstruction module FDM are combined into a multi-spectral image fusion framework based on an optimized double-branch feature pyramid, and the loss function Loss value between the output fusion image F and the input multi-modal image I1 and I2 is calculated. The image fusion framework is trained through back propagation to reduce the loss value;

[0044] S8, the set of dual-mode images to be fused M is put into the trained multi-modal image fusion framework, and the framework is trained in the platform NVIDIA GTX 3090 and the fusion image F is obtained by testing.

[0045] Referring to FIG. 2, the feature extraction module FEM is composed of convolution layers with convolution kernel sizes of 1x1 and 3x3, Canny operators with a convolution kernel size of 3x3, regularization BN layers and nonlinear activation functions LReLU jump connections, which are used to extract the features of the input image. This simple combination can effectively utilize the background edge details and target features, achieve better extraction of multi-modal image features, and avoid excessive increase of redundant parameters.

[0046] Referring to FIG. 3, the shallow fusion module CFM is composed of a Canny operator with a convolution kernel size of 3*3, an average pooling layer, a deep convolution layer with a convolution kernel size of 3*3, and a Sigmoid activation function, and is used for outputting shallow fusion features and adjusting feature extraction capability.

[0047] Referring to FIG. 4, the deep fusion module AFM is composed of a Prewitt operator with a convolution kernel size of 3*3, an average pooling layer, a RepVGG, and a Sigmoid activation function, and is used for outputting dominant information on a global level in deep features.

[0048] Referring to FIG. 5, the feature reconstruction module FDM is composed of convolution layers with a convolution kernel size of 1*1 and 3*3, respectively, and a nonlinear activation function LReLU jump connection, and this simple combination can ensure that important information is not lost, and quickly reconstruct the fusion image and reduce network parameters.

[0049] The multispectral image fusion framework based on the optimized double-branch feature pyramid of the application as a whole includes a feature extraction subnetwork, a fusion subnetwork, and a feature reconstruction subnetwork, wherein the feature extraction subnetwork is composed of 4 feature extraction FEM modules connected in residual, the fusion subnetwork is composed of 2 shallow fusion modules CFM and 2 deep fusion modules AFM, and the feature reconstruction subnetwork includes 4 feature reconstruction modules FDM and a Tanh layer jump connection combination.

[0050] The above is the preferred embodiment of the application, but the embodiments of the application are not limited by the above, any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the application, should be equivalent replacement, all included in the protection scope of the application.

Claims

1. A multispectral image fusion framework based on optimized bi-branch feature pyramids, comprising the following steps: S1. Use infrared and visible light cameras to simultaneously capture 500 pairs of dual-modal images to form a dual-modal image set G to be registered. The registered image set is H and the two modal images are denoted as I1 and I2 respectively. S2. The infrared and visible light images are used as network inputs (I1 and I2 respectively) and then processed by the feature extraction module (FEM) to capture relevant features. and ; S3, Features and After the shallow fusion module CFM, the features are... and The weights of the information proportions are adjusted to obtain new features. and ; S4, will obtain features , , and Add the features to obtain the features and features As input to the next feature extraction module; S5, Features After deep fusion module AFM, the features and The information proportions are weighted and adjusted to obtain features. and ; S6, will obtain features and Add the features to obtain the features and splice features and As input to the feature reconstruction subnetwork; S7. Combine the feature extraction module FEM, the shallow fusion module CFM, the deep fusion module AFM, and the feature reconstruction module FDM into a multispectral image fusion framework based on an optimized bi-branch feature pyramid. Calculate the loss function Loss value between the output fused image F and the input multimodal images I1 and I2. Train the image fusion framework through backpropagation to reduce the loss value. S8. Place the bimodal image set H to be fused into the trained image fusion framework and test to obtain the fused image F.

2. The multispectral image fusion framework based on optimized bi-branch feature pyramids according to claim 1, characterized in that, In step S1, the dual-modal image includes a visible light image, a polarized light image, and infrared light images of different wavelengths.

3. The multispectral image fusion framework based on optimized bi-branch feature pyramids according to claim 1, characterized in that, In step S2, the feature extraction module FEM consists of convolutional layers with kernel sizes of 1×1 and 3×3, a Canny operator with a kernel size of 3×3, a regularized BN layer, and a nonlinear activation function LReLU skip connection, used to extract features from the input image.

4. The multispectral image fusion framework based on optimized bi-branch feature pyramids according to claim 1, characterized in that, In step S3, the shallow fusion module CFM consists of two parts, which are composed of a Canny operator with a kernel size of 3×3, an average pooling layer, a deep convolutional layer with a kernel size of 3×3, and a Sigmoid activation function. They are used to output shallow fusion features and adjust feature extraction capabilities.

5. The multispectral image fusion framework based on optimized bi-branch feature pyramids according to claim 1, characterized in that, In step S5, there are two deep fusion modules (AFM), consisting of a Prewitt operator with a 3×3 convolution kernel, an average pooling layer, a RepVGG layer, and a Sigmoid activation function, which are used to output global-level advantage information in deep features.

6. The multispectral image fusion framework based on optimized bi-branch feature pyramids according to claim 1, characterized in that, In step S6, the feature reconstruction subnetwork uses a feature reconstruction module FDM consisting of convolutional layers with kernel sizes of 1×1 and 3×3 and nonlinear activation function LReLU skip connections, which is used to reconstruct a high-quality fused image.

7. The multispectral image fusion framework based on optimized bi-branch feature pyramids according to claim 1, characterized in that, In step S7, the loss function Loss is composed of the pixel-level loss function L. con and gradient loss function L adv The composition is calculated using the following formula: Among them, I f To fuse images, I vi For visible light images, I ir This is a mid-infrared light image. Refers to the gradient operator Canny.

Images