A method and system for implementing infrared and visible light image fusion

Abstract

The application relates to the technical field of digital image processing, and provides a method and system for realizing infrared and visible light image fusion, which comprises the following steps: extracting initial infrared features and initial visible light features; adopting a multi-channel parallel cross-vision Mamba network model to respectively perform global feature extraction on the initial infrared features and the initial visible light features, so as to obtain infrared global features and visible light global features; adopting a multi-channel parallel KAGtention network model to respectively perform local feature extraction on the initial infrared features and the initial visible light features, so as to obtain infrared local features and visible light local features; performing merging and mapping reconstruction on the infrared global features and the visible light global features to obtain global features; performing merging and mapping reconstruction on the infrared local features and the visible light local features to obtain local features; and generating a fusion image according to the global features and the local features. The application can significantly improve the quality of the fusion image, reduces the required computing resources while maintaining high precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, and particularly relates to a method and system for realizing infrared and visible light image fusion. BACKGROUND

[0002] Multi-modal image fusion technology, as an advanced image processing method, aims to integrate image data from different sensing technologies and imaging modes into a single output image with a large amount of information. The core goal of this technology is to integrate the advantages of various image sources and overcome the limitations of single images, thereby providing higher quality and information density images. Multi-modal image fusion technology has a wide range of applications, involving medical imaging, remote sensing, computer vision, and military surveillance, and many other important fields.

[0003] Currently, one of the typical multi-modal image fusion tasks is the fusion of infrared and visible light images. The goal of this task is to make full use of the data of two different wavelength images to produce more comprehensive and comprehensive information than a single image. Infrared images are favored for their unique ability to observe at night and penetrate smoke, while visible light images are known for their high-resolution detail display capabilities. However, there are still problems such as blurred details and texture loss in the process of infrared and visible light image fusion in the prior art, not only the parameter quantity of the model is large, but also the fusion effect is poor. Therefore, the main challenge in the field of multi-modal image fusion is how to effectively protect and enhance the subtle texture structure in the original image while deeply fusing the information of multiple modal images, while suppressing noise interference, reducing unnecessary artifacts, and reducing the parameter quantity of the fusion model, which is the key to promoting the development of this field. Specifically, the focus of the task is how to fuse infrared and visible light images to achieve the complementary advantages of both - that is, not only to retain the thermal radiation characteristics of infrared images, but also to maintain the detail clarity of visible light images. Further, the process aims to enhance key features in the source images, such as the shadows of leaves in low light conditions, the fine structure of building surfaces, and the texture of zebra stripes on the road, to improve the richness and recognition of overall visual information. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a method and system for realizing infrared and visible light image fusion.

[0005] In one aspect of the present application, a method for realizing infrared and visible light image fusion is provided, the method comprising:

[0006] performing feature extraction on the original infrared image and the original visible light image respectively to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image;

[0007] The two preset multi-channel parallel cross-vision Mamba network models arranged in parallel are respectively used for global feature extraction of the initial infrared feature and the initial visible light feature, to obtain infrared global features highlighting global information of the original infrared image and visible light global features highlighting global information of the original visible light image;

[0008] The two preset multi-channel parallel KAGtention network models arranged in parallel are respectively used for local feature extraction of the initial infrared feature and the initial visible light feature, to obtain infrared local features of the original infrared image and visible light local features of the original visible light image;

[0009] The infrared global features and the visible light global features are merged, and the merged global fusion features are mapped to the resolution of the original image, to obtain reconstructed global features;

[0010] The infrared local features and the visible light local features are merged, and the merged local fusion features are mapped to the resolution of the original image, to obtain reconstructed local features;

[0011] Image reconstruction is performed according to the reconstructed global features and the local features, to generate a fusion image.

[0012] Further, the multi-channel parallel cross-vision Mamba network model comprises a plurality of multi-channel parallel cross-vision Mamba modules arranged in series, each multi-channel parallel cross-vision Mamba module comprising a first normalization layer, two Mamba modules arranged in parallel and connected with output channels of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer arranged in sequence, and each Mamba module is provided with a first adaptive adjustment operation layer.

[0013] Further, the implementation process of the multi-channel parallel cross-vision Mamba module for global feature extraction comprises:

[0014] The first normalization layer is used for preprocessing the initial infrared feature or the initial visible light feature, and the output feature is divided into two feature vectors and Each feature vector comprises 1 / 2 features of the original channel C, and the operation process is as follows:

[0015]

[0016] The two feature vectors are respectively input into the two Mamba modules arranged in parallel for enhanced feature extraction, and the first adaptive adjustment operation layer is used for adaptive adjustment of the enhanced feature extraction results according to a preset adjustment factor, the feature vectors processed by the two Mamba modules are cross-merged to generate two merged feature vectors, and the operation process is as follows:

[0017]

[0018]

[0019] The two obtained merging feature vectors are connected through a residual connection mechanism by using a residual quick connection layer and normalized by using a second normalization layer, and projection operation is performed on the connected features after normalization to obtain the infrared global feature or the visible light global feature The operation process is as follows:

[0020]

[0021] Wherein, LN(·) represents layer normalization, Sp(·) represents Split operation, Mamba(·) represents Mamba operation, θ represents a preset adjustment factor, Concat(·) represents connection operation, Pro(·) represents projection operation, i represents the number of Mamba blocks, and R represents input features.

[0022] Further, the multi-channel parallel KAGtention network model comprises a plurality of multi-channel parallel KAGtention modules arranged in series, each multi-channel parallel KAGtention module comprising a third normalization layer, four KAGN modules arranged in parallel and connected with output channels of the third normalization layer, a residual quick connection layer, a fourth normalization layer, and a projection layer arranged in sequence, and each KAGN module is provided with a second adaptive adjustment operation layer.

[0023] Further, the implementation process of the multi-channel parallel KAGtention module for local feature extraction comprises:

[0024] The initial infrared feature or the initial visible light feature is preprocessed by using a third normalization layer, and the output feature is divided into four feature vectors and Each feature vector includes 1 / 4 features of the original channel C, and the operation process is as follows:

[0025]

[0026] The four feature vectors are respectively input into the four KAGN modules arranged in parallel for local detailed information feature extraction, and the second adaptive adjustment operation layer is used to adaptively adjust the local detailed information feature extraction results according to a preset adjustment factor, and the operation process is as follows:

[0027]

[0028] The four adaptive adjusted output features obtained are connected through a residual connection mechanism by using a residual quick connection layer and normalized by using a fourth normalization layer, and projection operation is performed on the connected features after normalization to obtain the infrared local feature or the visible light local feature The operation process is as follows:

[0029]

[0030]

[0031] wherein, LN(·) represents layer normalization, Sp(·) represents Split operation, KAGN(·) represents KAGN operation, θ represents a preset adjustment factor, Concat(·) represents connection operation, Pro(·) represents projection operation, j represents the number of layers of the KAGN module, and R represents input features.

[0032] Further, the KAGN module is composed of a group of univariate nonlinear functions:

[0033] The convolution operation model of the KAGN module is:

[0034]

[0035] wherein, I vi and I ir are visible light images and infrared images respectively, C represents the number of channels, H and W represent the height and width of image features respectively, k is the kernel size of the KAGN module, is a univariate nonlinear learnable function, has its own trainable parameters, a, b, and d represent the a-th, b-th, and d-th layers in the KAGN respectively, represent the univariate nonlinear learnable functions of the a-th, b-th, and d-th layers respectively, and the polynomial is as follows:

[0036]

[0037]

[0038] b(x) = SiLU(x) = x / (1 + e -x )

[0039] wherein, ω b represents a weight, b(x) represents a nonlinear activation function, and the polynomial Gram is an n-th order polynomial about x, for n = 0, 1, 2, …, N-1, which is constructed based on the correlation of the weight functions of two different orders about , wherein δ(·) represents the Dirac function.

[0040] Further, the mapping of the merged global fusion features to the resolution of the original image to obtain the reconstructed global features comprises: adopting a plurality of multi-channel parallel cross-vision Mamba modules arranged in series to iteratively map the merged global fusion features to the resolution of the original image to obtain the global features.

[0041] The mapping of the merged local fusion features to the resolution of the original image to obtain the reconstructed local features comprises: adopting a plurality of multi-channel parallel KAGtention modules arranged in series to iteratively map the merged local fusion features to the resolution of the original image to obtain the local features.

[0042] Further, the method further comprises:

[0043] The loss function is L:

[0044] L = λ1·L int + λ2·L grad + λ3·L dec

[0045] Wherein, L int represents the intensity loss, L grad represents the gradient loss, L dec represents the multi-branch feature decomposition loss, and λ1, λ2 and λ3 represent the weights of the corresponding loss functions, respectively.

[0046] The function form of L int is as follows:

[0047]

[0048] Wherein, H and W represent the height and width of the image features, respectively.

[0049] The function form of L grad is as follows:

[0050]

[0051] Wherein, ▽ represents the Sobel gradient operator.

[0052] The function form of L dec is as follows:

[0053]

[0054] Wherein, CC(·) represents the correlation coefficient operation, respectively, represent the visible light image I vi and the infrared image I irVisible light global features and infrared global features extracted by the multi-channel parallel cross-vision Mamba network model, and Respectively represent I vi and I ir Visible light local features and infrared local features extracted by the multi-channel parallel KAGattention network model.

[0055] Another aspect of the present invention provides a system for implementing infrared and visible light image fusion, the system comprising a first convolutional model, a PCVM encoder, a PKAGN encoder, a first channel addition module, a second channel addition module, a PCVM decoder, a PKAGN decoder, and a second convolutional model;

[0056] The first convolution model is used to extract features from the original infrared image and the original visible light image respectively, to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image;

[0057] A PCVM encoder is used to extract global features from the initial infrared features and the initial visible light features using two parallel preset multi-channel parallel cross-visual Mamba network models, thereby obtaining infrared global features that highlight the global information of the original infrared image and visible light global features that highlight the global information of the original visible light image.

[0058] The PKAGN encoder is used to extract local features from the initial infrared features and the initial visible light features using two parallel preset multi-channel parallel KAG attention network models, thereby obtaining the infrared local features of the original infrared image and the visible light local features of the original visible light image.

[0059] A first channel addition module is used to combine the infrared global feature and the visible light global feature to obtain a global fusion feature;

[0060] PCVM decoder, used to map the global fusion features to the resolution of the original image to obtain the reconstructed global features;

[0061] A second channel adding module is used to combine the infrared local features and the visible light local features to obtain a local fusion feature;

[0062] PKAGN decoder, used to map the local fusion features to the resolution of the original image to obtain the reconstructed local features;

[0063] The second convolutional model is used to reconstruct the image based on the reconstructed global features and local features to generate a fused image.

[0064] Further, the PCVM encoder is implemented by two parallel multi-channel cross- vision Mamba network models arranged in parallel, each of which comprises a plurality of multi-channel cross-vision Mamba modules arranged in series, each of which comprises a first normalization layer, two Mamba modules arranged in parallel and connected to the output channels of the first normalization layer, a residual shortcut layer, a second normalization layer, and a projection layer arranged in sequence, and each of the Mamba modules is provided with a first adaptive adjustment operation layer.

[0065] The PKAGN encoder is implemented by two parallel multi-channel KAGtention network models arranged in parallel, each of which comprises a plurality of multi-channel KAGtention modules arranged in series, each of which comprises a third normalization layer, four KAGN modules arranged in parallel and connected to the output channels of the third normalization layer, a residual shortcut layer, a fourth normalization layer, and a projection layer arranged in sequence, and each of the KAGN modules is provided with a second adaptive adjustment operation layer.

[0066] The method and system for infrared and visible light image fusion provided by the application can enhance the effect of image fusion, significantly improve the detail definition and texture performance in the fused image, and reduce the required computing resources while maintaining high precision, thereby embodying its efficiency and practicality in practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0067] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not intended to limit the scope of the application. Moreover, like reference numerals designate like parts throughout the several views in the drawings. In the drawings:

[0068] Figure 1 A network architecture diagram of a system for infrared and visible light image fusion provided by an embodiment of the application;

[0069] Figure 2 An implementation flowchart of the method for implementing infrared and visible light image fusion provided by the embodiment of the present application is provided.

[0070] FIG. 3(a) is a network structure diagram of a multi-channel parallel cross-vision Mamba module provided by the embodiment of the present application;

[0071] FIG. 3(b) is a network structure diagram of a Mamba module provided by the embodiment of the present application;

[0072] FIG. 4(a) is a network structure diagram of a plurality of multi-channel parallel KAGtention modules provided by the embodiment of the present application;

[0073] FIG. 4(b) is a network structure diagram of a KAGtention module provided by the embodiment of the present application;

[0074] Figure 5 FIG. 5 is a comparison diagram of fusion results of the method for implementing infrared and visible light image fusion provided by the present application and the existing image fusion method on the LLVIP dataset;

[0075] Figure 6 FIG. 6 is a bubble chart for comparing the efficiency and effect of the method for implementing infrared and visible light image fusion provided by the present application and the existing image fusion method on the LLVIP dataset. DETAILED DESCRIPTION

[0076] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0077] In order to reduce the model parameter amount while ensuring the high performance of the infrared and visible light image fusion model, the present application proposes an innovative method and system for implementing infrared and visible light image fusion. The present application can significantly reduce the parameter amount of the model, and at the same time, by using advanced image processing technology and optimized fusion strategy, not only the key features of the original image are retained, but also the overall visual effect of the fused image is significantly improved. The present application mainly consists of three key modules: a multi-branch heterogeneous encoder responsible for feature extraction and decomposition, a multi-branch heterogeneous decoder for image reconstruction and generation, and a set of carefully designed loss functions for optimizing the performance in the model training process.

[0078] Figure 1 A network architecture diagram of a system for implementing infrared and visible light image fusion is shown. As shown in FIG. 1, the system mainly consists of three key modules: a multi-branch heterogeneous encoder 101 responsible for feature extraction and decomposition, a multi-branch heterogeneous decoder 102 for image reconstruction and generation, and a set of carefully designed loss functions 103 for optimizing the performance in the model training process.Figure 1 As shown, the system includes a first convolution model, a PCVM encoder, a PKAGN encoder, a first channel addition module, a second channel addition module, a PCVM decoder, a PKAGN decoder and a second convolution model.

[0079] The first convolution model includes two convolution blocks, the outputs of which are connected to the inputs of the PCVM encoder and the PKAGN encoder, respectively, for extracting features from the original infrared image and the original visible light image, respectively, to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image;

[0080] The PCVM encoder is used to extract global features from the initial infrared features and the initial visible light features using two parallel preset multi-channel parallel cross-visual Mamba network models, thereby obtaining infrared global features that highlight the global information of the original infrared image and visible light global features that highlight the global information of the original visible light image.

[0081] The PKAGN encoder is used to extract local features from the initial infrared features and the initial visible light features using two parallel preset multi-channel parallel KAG attention network models, thereby obtaining the infrared local features of the original infrared image and the visible light local features of the original visible light image.

[0082] A first channel addition module is used to combine the infrared global feature and the visible light global feature to obtain a global fusion feature;

[0083] PCVM decoder, used to map the global fusion features to the resolution of the original image to obtain the reconstructed global features;

[0084] A second channel adding module is used to combine the infrared local features and the visible light local features to obtain a local fusion feature;

[0085] PKAGN decoder, used to map the local fusion features to the resolution of the original image to obtain the reconstructed local features;

[0086] The second convolutional model is used to reconstruct the image based on the reconstructed global features and local features to generate a fused image.

[0087] Further, the second convolutional model comprises two convolutional blocks and an element multiplication operation module connected with the outputs of the two convolutional blocks; one of the convolutional blocks is used for feature extraction on the global features output by the PCVM decoder to obtain first fusion features, and the other convolutional block is used for feature extraction on the local features output by the PKAGN decoder to obtain second fusion features, and the element multiplication operation module is used for element-wise multiplication operation on the first fusion features and the second fusion features to generate a final fusion image.

[0088] In Figure 1 In one specific embodiment shown, the PCVM encoder is implemented by two parallel multi-channel parallel cross vision Mamba network models arranged in parallel, each of which comprises a plurality of multi-channel parallel cross vision Mamba modules arranged in series. The output features of the previous multi-channel parallel cross vision Mamba module in the multi-channel parallel cross vision Mamba network model are used as the input of the next multi-channel parallel cross vision Mamba module to realize multi-level iterative feature extraction. In this embodiment, three multi-channel parallel cross vision Mamba modules arranged in series are taken as an example for illustration. Further, each multi-channel parallel cross vision Mamba module comprises a first normalization layer, two Mamba modules arranged in parallel connected with the output channels of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer arranged in series, and each Mamba module is provided with a first adaptive adjustment operation layer.

[0089] The Mamba module is implemented by an improved S4, i.e., a Mamba model, which is an advanced sequence modeling method and can maintain high efficiency when processing complex data. It dynamically adapts to different scenarios by combining input-dependent state space adjustment and linear time complexity design, effectively capturing long-term dependencies in sequences. Mamba exhibits higher efficiency and competitiveness when processing long sequences, providing a new and high-performance solution for sequence data analysis. The Mamba series model exhibits extraordinary performance in the field of computer vision.

[0090] In Figure 1In one specific embodiment shown, the PKAGN encoder is implemented by two multi-channel parallel KAGtention network models arranged in parallel, each of which includes a plurality of multi-channel parallel KAGtention modules arranged in series. The output features of the previous multi-channel parallel KAGtention module in the multi-channel parallel KAGtention network model are taken as the input of the next multi-channel parallel KAGtention module to realize multi-level iterative feature extraction. In this embodiment, three multi-channel parallel KAGtention modules arranged in series are taken as an example for illustration. Further, each multi-channel parallel KAGtention module includes a third normalization layer, four KAGN modules arranged in parallel connected to the output channels of the third normalization layer, a residual shortcut connection layer, a fourth normalization layer, and a projection layer arranged in sequence, and each KAGN module is provided with a second adaptive adjustment operation layer.

[0091] The KAGN module is implemented by a Kolmogorov-Arnold network (KAN), which is a new architecture based on the Kolmogorov-Arnold (KN) representation theorem and can efficiently and flexibly simulate complex functions with fewer parameters. Each weight of the network is represented by a learnable univariate function, which not only reduces the number of parameters but also enhances the interpretability of the model. With a solid mathematical theoretical foundation, KAN has shown great potential in image processing tasks.

[0092] In this embodiment, the visible light image and the infrared image are respectively denoted as I vi and I ir , and I F represents the fused image. The PCVM encoder and the PKAGN encoder each have three layers, and the input and output channel numbers are configured as (4, 8), (8, 16), and (16, 32) in turn.

[0093]

[0094]

[0095] wherein, represents the features extracted by the PCVM encoder from I vi and I ir , respectively, respectively. PCVM(·), PKAGN(·) represent the operation of Multi-Channel Parallel Cross Vision Mamba module and Multi-Channel Parallel KAGtention module respectively. Conv(·) represents the convolution operation of convolution block. The features from the two different encoders are further merged by channel addition module for subsequent processing.

[0096]

[0097] where Concat(·) represents the concatenation operation.

[0098] In this embodiment, the Multi-Channel Parallel Cross Vision Mamba (PCVM) module has parallel processing capability, which can process multiple feature maps simultaneously in the fusion process. This module can effectively integrate information from different modalities and contains a mechanism to identify complementary information and how to merge these information without compromising image quality. The Mamba module can dynamically select valuable features or information in the fusion process due to its selective state space characteristics, which is crucial for preserving key features of the image such as edges and textures.

[0099] The application of Multi-Channel Parallel KAGtention (PKAGN) module in infrared and visible image fusion is mainly to improve the quality and efficiency of the fused image. The Kolmogorov-Arnold model is a mathematical model that shows that any multivariate function can be approximated by a two-parameter function. In image processing, the KAN model can be used to describe the complex relationship between pixel values and other related factors in the image. In the process of infrared and visible image fusion, the use of PKAGN module can help achieve more efficient, stable and high-quality image fusion results.

[0100] The application of the PCVM module and the PKAGN module exhibits significant complementarity. The PCVM module focuses on capturing global information of an image, which not only deepens the understanding of the overall structure of the image, but also enhances the understanding of the mutual relationship between various objects in the image. In contrast, the PKAGN module particularly focuses on extracting local features of an image, and can efficiently identify subtle differences in the image, such as texture, edge and other local patterns. When the PCVM module and the PKAGN module are combined, the combination not only endows the model with strong global perception ability, enabling it to better understand and analyze the overall layout of the image, but also ensures that the model maintains high sensitivity to subtle details of the image, and can accurately capture important local information.

[0101] Figure 2 An implementation flowchart of the method for fusing infrared and visible light images according to an embodiment of the present application is shown. The method for fusing infrared and visible light images provided by the embodiment of the present application is applicable to the system for fusing infrared and visible light images as shown in Figure 1 The method for fusing infrared and visible light images provided by the embodiment of the present application is applicable to the system for fusing infrared and visible light images as shown in Figure 2 The method for fusing infrared and visible light images provided by the embodiment of the present application specifically includes the following steps:

[0102] S11, feature extraction is respectively performed on the original infrared image and the original visible light image to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image.

[0103] In the embodiment of the present application, the visible light image and the infrared image can be simultaneously acquired by using a collection device capable of simultaneously capturing the visible light image and the infrared image, or the visible light image and the infrared image can be respectively collected by using different collection devices. The embodiment of the present application does not make specific limitation on this.

[0104] Before entering the PCVM encoder or the PKAGN encoder, a convolution block is first set before the two types of encoders, wherein the convolution kernel size is 3x3, the input channel number is 1, and the output channel number is 4. Next, the output features of the convolution block enter the PCVM encoder or the PKAGN encoder for feature extraction.

[0105] S12, the initial infrared features and the initial visible light features are respectively extracted by two preset multi-channel parallel cross-vision Mamba network models arranged in parallel to obtain infrared global features of the original infrared image and visible light global features of the original visible light image. Specifically, the initial infrared features and the initial visible light features output by the convolution block arranged before the PCVM encoder are respectively input into two preset multi-channel parallel cross-vision Mamba network models arranged in parallel in the PCVM encoder, and the initial infrared features and the initial visible light features are respectively extracted by the corresponding multi-channel parallel cross-vision Mamba network models.

[0106] S13, the initial infrared features and the initial visible light features are respectively extracted by two preset multi-channel parallel KAGtention network models arranged in parallel to obtain infrared local features of the original infrared image and visible light local features of the original visible light image. Specifically, the initial infrared features and the initial visible light features output by the convolution block arranged before the PKAGN encoder are respectively input into two preset multi-channel parallel KAGtention network models arranged in parallel in the PKAGN encoder, and the initial infrared features and the initial visible light features are respectively extracted by the corresponding multi-channel parallel KAGtention network models.

[0107] S14, the infrared global features and the visible light global features are merged, and the merged global fusion features are mapped to the resolution of the original image to obtain reconstructed global features.

[0108] S15, the infrared local features and the visible light local features are merged, and the merged local fusion features are mapped to the resolution of the original image to obtain reconstructed local features.

[0109] S16, image reconstruction is performed according to the reconstructed global features and the local features to generate a fusion image. Specifically, a convolution block can be used to extract features from the global features to obtain first fusion features, another convolution block can be used to extract features from the local features to obtain second fusion features, and then the first fusion features and the second fusion features are subjected to element-wise multiplication to generate a final fusion image.

[0110] The method for realizing infrared and visible light image fusion provided by the application introduces a multi-channel cross-vision Mamba module and a multi-channel parallel KAGtention module to respectively extract global and local features, thereby enhancing the image fusion effect, enabling the application to maintain high precision while reducing the required computing resources, and embodying its efficiency and practicality in actual applications. Further, the application uses a multi-branch heterogeneous framework to process infrared and visible light images to fully utilize their respective characteristics and significantly improve the quality of the fused image. In the task of infrared and visible light image fusion, the application not only performs excellently in subjective visual evaluation, but also surpasses the prior art in objective quantitative indicators of infrared and visible light image fusion, has wide application potential, and is of great significance to further promoting the development of the image processing field.

[0111] In an embodiment of the application, the multi-channel parallel cross-vision Mamba network model includes a plurality of multi-channel parallel cross-vision Mamba modules arranged in series. As shown in FIG. 3(a), each multi-channel parallel cross-vision Mamba module includes a first normalization layer, two Mamba modules arranged in parallel and connected to the output channels of the first normalization layer, a residual fast connection layer, a second normalization layer, and a projection layer arranged in sequence, and each Mamba module is provided with a first adaptive adjustment operation layer. The specific network structure of the Mamba module is shown in FIG. 3(b). The Mamba module includes a first linear layer, a second linear layer, a first activation function operation module connected to the output channels of the first linear layer, a convolution block, a second activation function operation module, and a state space equation module connected in sequence to the output channels of the second linear layer, an element multiplication operation module connected to the output ends of the first activation function operation module and the state space equation module, and a third linear layer.

[0112] In the PCVM encoder, first, the input feature is preprocessed by the LN normalization layer, and then is divided into two feature vectors and The number of channels of each feature vector is C / 2. Then, and Each is sent into the corresponding multi-channel parallel cross-vision Mamba module for independent processing. Inside the multi-channel parallel cross-vision Mamba module, the processed output is integrated using residual connection and an adjustment factor, which is designed to enhance the model's ability to capture long-range spatial information. Then, the two processed feature vectors are cross-merged into a single feature vector through the Concat operation. Then, the feature vector is subjected to double transformation of the LN layer and the Projection layer to generate the output feature

[0113] Specifically, the implementation process of the multi-channel parallel cross-vision Mamba module for global feature extraction specifically includes:

[0114] The first normalization layer is used to pre-process the initial infrared feature or initial visible light feature, and the output feature is divided into two feature vectors and Each feature vector includes 1 / 2 features of the original channel C, and the operation process is as follows:

[0115]

[0116] The two feature vectors are input into two Mamba modules arranged in parallel for enhanced feature extraction, and the first adaptive adjustment operation layer is used to adaptively adjust the enhanced feature extraction results according to a preset adjustment factor. The feature vectors processed by the two Mamba modules are cross-merged to generate two merged feature vectors, and the operation process is as follows:

[0117]

[0118]

[0119] The residual fast connection layer is used to connect the two merged feature vectors obtained by the residual connection mechanism and normalize the connection features by the second normalization layer. The projection operation is performed on the normalized connection features to obtain the infrared global feature or the visible light global feature The operation process is as follows:

[0120]

[0121] Wherein, LN(·) represents layer normalization, Sp(·) represents Split operation, Mamba(·) represents Mamba operation, θ represents a preset adjustment factor, Concat(·) represents connection operation, Pro(·) represents projection operation, i represents the number of Mamba blocks, and R represents input features.

[0122] In one embodiment of the present application, the multi-channel parallel KAGtention network model comprises a plurality of multi-channel parallel KAGtention modules arranged in series. As shown in FIG. 4(a), each multi-channel parallel KAGtention module comprises a third normalization layer (not shown in the figure), four KAGN modules arranged in parallel and connected to the output channels of the third normalization layer, a residual fast connection layer, a fourth normalization layer, and a projection layer (not shown in the figure), which are arranged in sequence, and each KAGN module is provided with a second adaptive adjustment operation layer. The specific network structure of the KAGtention module is shown in FIG. 4(b). Referring to FIG. 4, the KAGtention module is composed of a group of single-variable nonlinear functions.

[0123] In the PKAGN encoder, the input features pass through the LN layer and are divided into four features and The number of channels of each feature is C / 4. These independent features are sent into the multi-channel parallel KAGtention network model for processing, and the output is combined with the adjustment factor through the residual connection, aiming to enhance the ability of the model to capture long-range spatial information. After this step, the four processed features are combined into a feature vector through the Concat operation. The vector is then converted through the LN layer and the Projection layer, thereby generating the final output feature.

[0124] Specifically, the implementation process of the multi-channel parallel KAGtention module for local feature extraction includes:

[0125] The third normalization layer is used to pre-process the initial infrared feature or the initial visible light feature, and the output feature is divided into four feature vectors and Each feature vector includes 1 / 4 of the original channel C, and the operation process is as follows:

[0126]

[0127] The four feature vectors are respectively input into the four KAGN modules arranged in parallel for local detailed information feature extraction, and the second adaptive adjustment operation layer is used to adaptively adjust the local detailed information feature extraction result according to the preset adjustment factor. The operation process is as follows:

[0128]

[0129] The residual fast connection layer is used to connect the four adaptively adjusted output features through the residual connection mechanism, and the fourth normalization layer is used for normalization processing. The projection operation is performed on the normalized connection features, and the infrared local feature or the visible light local feature is obtained. The operation process is as follows:

[0130]

[0131]

[0132] wherein LN(·) represents layer normalization, Sp(·) represents a Split operation, KAGN(·) represents a KAGN operation, θ represents a preset adjustment factor, Concat(·) represents a connection operation, Pro(·) represents a projection operation, j represents a number of layers of the KAGN module, and R represents an input feature.

[0133] Further, in the multi-channel parallel KAGtention network model, the KAGN module is composed of a set of univariate nonlinear functions, and is applied to the infrared and visible light images I vi ir ∈R c×m×n wherein C represents a number of channels, H and W represent a height and a width of the image respectively. Therefore, for a kernel with a size of k, the convolution operation model of the KAGN module under the KAN framework is as follows:

[0134]

[0135] wherein I vi ir are a visible light image and an infrared image respectively, C represents a number of channels, H and W represent a height and a width of the image feature respectively, and k is a kernel size of the KAGN module, are all univariate nonlinear learnable functions with their own trainable parameters, and a, b, d represent the a-th, b-th and d-th layers in the KAGN respectively, represent the a-th, b-th and d-th layers respectively, and the polynomial is as follows:

[0136]

[0137]

[0138] b(x) = SiLU(x) = x / (1+e -x )

[0139] wherein ω b represents a weight, b(x) represents a nonlinear activation function, and the polynomial Gram is an n-th order polynomial about x, and for n = 0, 1, 2, …, N-1, it is constructed based on two polynomials of different orders about ​​the relevance of the weight function, where δ(·) denotes the Dirac function. KAGN not only allows for efficient parameter fine-tuning of pre-trained models, but also reduces the number of parameters that need to be trained.

[0140] In KAGN, the regularization technique is applied to the Kolmogorov-Arnold convolution, which involves the computation of the polynomial Gram. Specifically, KAGN first computes the polynomial basis over the variable x, and then performs a weighted sum over it. Therefore, KAGN provides three different dropout application locations: before layer processing (Full), before computing the polynomial basis (Poly), and before applying the weights to the polynomial (Degree).

[0141] Further, mapping the merged global fusion feature to the resolution of the original image to obtain the reconstructed global feature includes: adopting a plurality of multi-channel parallel cross-vision Mamba modules arranged in series to iterate the merged global fusion feature multiple times to map the global fusion feature to the resolution of the original image to obtain the global feature. Mapping the merged local fusion feature to the resolution of the original image to obtain the reconstructed local feature includes: adopting a plurality of multi-channel parallel KAGtention modules arranged in series to iterate the merged local fusion feature multiple times to map the local fusion feature to the resolution of the original image to obtain the local feature.

[0142] In the embodiments of the present application, the multi-channel parallel cross-vision Mamba modules and the multi-channel parallel KAGtention modules in the PCVM decoder and the PKAGN decoder are the same as the network structure in the corresponding encoder, and the only difference is the number of input and output channels, which is used to realize channel merging. Specifically, first, the obtained global fusion feature and local fusion feature are respectively decomposed and assigned to the corresponding PCVM decoder and PKAGN decoder. Then, the decomposed feature maps are spliced along the channel dimension while ensuring that the spatial size and pixel value range of the feature maps remain consistent; finally, the spliced feature maps are input into the decoder, and the channel number is decreased layer by layer until the final image representation is output, wherein the input and output channel numbers of each layer of the decoder are (64, 32), (32, 16), and (16, 8). This process not only helps to enhance the utilization efficiency of the decoder for feature information, but also promotes the generation of more accurate and detailed output images.

[0143] In the decoder of the present invention, the decomposed features are concatenated along the channel dimension as input, and two types of decoder outputs (PCVM decoder and PKAGN decoder) are generated, as well as the final fused image. The decoder and encoder have the same number of layers, that is, 3 layers, and the input and output channel configurations of each layer are (64, 32), (32, 16), and (16, 8), respectively. The decoder feature representation is as follows:

[0144]

[0145] in, and Represent the features processed by the PCVM decoder and the PKAGN decoder respectively. This design ensures that the decoder can gradually restore the spatial details of the original image while maintaining sufficient channel depth to capture complex feature information. Therefore, the fused image finally obtained by the present invention is:

[0146]

[0147] The number of input and output channels of Conv(·) is (8,1).

[0148] The loss function is crucial in the infrared and visible light image fusion model. It measures the difference between the model output and the real target, and guides the model to optimize parameters by minimizing this difference, thereby effectively fusing image features and improving the quality of the output image. In this embodiment of the present invention, the constructed loss function L is defined as

[0149] L=λ1·L int +λ2·L grad +λ3·L dec

[0150] Among them, L int Indicates strength loss, L grad represents the gradient loss, L dec represents the multi-branch feature decomposition loss, λ1λ2 and λ3 represent the weights of the corresponding loss functions respectively;

[0151] L int The main role of image fusion is to maintain and optimize the brightness information and contrast of the image, ensuring that the fused image can not only reflect the thermal radiation characteristics of the infrared image, but also retain the good visual effects and details of the visible light image. int The function form is as follows:

[0152]

[0153] Where H and W represent the height and width of the image feature respectively;

[0154] Lgrad The role in the present application is to make the model keep the edge and detail information of the image during the training process. It helps the model learn more accurate boundary and texture features by punishing the consistency difference between the gradient information in the fusion image and the source image (infrared image and visible light image), so as to generate a more clear and detailed fusion image. grad The function form of the function is as follows:

[0155]

[0156] Wherein, ∇ represents the Sobel gradient operator;

[0157] In order to improve the performance and effect of the multi-modal image fusion task, the L dec By decomposing the feature space, the PCVM encoder and the PKAGN encoder can more efficiently capture the correlation and information exchange between different modalities. dec The function form of the function is as follows:

[0158]

[0159] Wherein, CC(·) represents the correlation coefficient operation, respectively represent the visible light image I vi and the infrared image I ir The visible light global feature and the infrared global feature extracted by the multi-channel parallel cross-vision Mamba network model, and respectively represent I vi and I ir The visible light local feature and the infrared local feature extracted by the multi-channel parallel KAGtention network model.

[0160] A specific example is given below to verify and illustrate the beneficial effects of the method for realizing infrared and visible light image fusion proposed in the present application.

[0161] In this example, the MSRS dataset containing 1083 images, 300 M3FD images and 300 LLVIP images are used to train the framework. The pixel size of these images is set to 64x64 as the training samples. During the training process, 30 epochs are set, in which the first 5 epochs keep a fixed learning rate, while the learning rate is adjusted by an exponential decay strategy in the following 25 epochs. The initial learning rate is set to 0.001, the momentum is set to 0.9, and the weight decay is set to 10-4. Through a series of enumeration experiments, the 10th epoch is determined as the best training period of the model. Alternatively, for the weight parameters corresponding to the loss function, λ1, λ2 and λ3 are set to 1, 100 and 100 respectively. This setting is based on the preliminary parameter tuning experiment.

[0162] Figure 5 Table 1 systematically summarizes and shows the quantitative evaluation results of ten comparative algorithms on six general evaluation indicators. The evaluation criteria adopt entropy (EN), spatial frequency (SF), average gradient (AG), sum of correlation differences (SCD), correlation coefficient (CC) and multi-scale structural similarity index measure (MS-SSIM). For each evaluation indicator, the average score is calculated based on 50 pairs of test images in the LLVIP dataset, and the optimal result is marked in bold, the suboptimal result is marked with an underscore, and the third best result is displayed in italics to enhance the visual contrast effect. The analysis results show that the method for realizing infrared and visible light image fusion proposed in the present application has reached the optimal level on the six evaluation indicators. This detailed data analysis not only highlights the outstanding performance of the present application in objective performance evaluation, but also provides strong evidence for the effectiveness and necessity of the introduced PCVM and PKAGN module design.

[0163] Table 1. Average index values obtained by existing fusion methods and the present application on the LLVIP dataset

[0164] EN↑ SF↑ AG↑ SD↑ SCD↑ MS_SSIM↑ CrossFuse Li ]]> 6.324 9.316 2.474 24.683 0.882 0.908 CrossFuse Wang ]]> 6.367 13.833 3.645 37.227 1.177 0.900 DSFusion

[52] 6.517 14.380 4.172 35.272 1.126 0.947 LRRNet

[53] 6.348 10.565 2.865 28.931 0.983 0.800 MBHFuse 7.255 14.596 4.118 46.574 1.669 1.045 MFEIF 6.783 8.583 2.451 39.635 1.543 1.034 PIAFusion 7.182 13.883 4.087 44.890 1.539 1.029 SemLA 6.849 9.631 2.730 36.571 1.345 0.890 YDTR 6.527 8.260 2.099 29.940 1.064 0.929 Ours 7.315 15.466 4.476 47.982 1.821 1.058

[0165] Figure 5 The present application provides a method for realizing infrared and visible light image fusion. The present application shows a comparison between the fusion results of existing image fusion methods and the method for realizing infrared and visible light image fusion provided by the present application on the LLVIP dataset.

[0166] Figure 6A bubble chart is shown to compare the efficiency and effect of the present application and the current SOTA method on the LLVIP dataset. The horizontal coordinate of the chart represents the average inference time of different frameworks on the LLVIP dataset, which reflects the calculation efficiency of the model; the vertical coordinate represents the sum of the six evaluation indexes after normalization, which is used to measure the overall performance of the model. In addition, the size of the circle in the chart and the number marked in the circle represent the parameter amount of each model, which reflects the complexity of the model and the demand for computing resources. From Figure 6 It can be seen from the above that the present application not only leads in comprehensive performance, but also has the smallest model parameter amount. This outstanding performance is mainly due to the PCVM and PKAGN proposed in the present application. The former enhances the feature extraction capability through an efficient cross-channel information exchange mechanism, thereby improving the model performance without increasing too many parameters; the latter realizes effective capture of important features through KAN while maintaining a relatively low computational complexity. The design of these two modules effectively balances the performance and efficiency of the model, enabling the present application to reduce the required computing resources while maintaining high accuracy, thereby embodying its efficiency and practicality in practical applications.

[0167] The method and system for realizing infrared and visible light image fusion provided by the present application introduce a multi-channel cross-vision Mamba module and a multi-channel parallel KAGtention module to perform global and local feature extraction respectively, thereby enhancing the image fusion effect and significantly improving the detail clarity and texture performance in the fused image. The present application can reduce the required computing resources while maintaining high accuracy, thereby embodying its efficiency and practicality in practical applications. Further, the present application uses a multi-branch heterogeneous framework to process infrared and visible light images respectively to fully utilize their respective characteristics and achieve significant effect in improving the quality of the fused image. In the task of infrared and visible light image fusion, the present application not only performs excellently in subjective visual evaluation, but also surpasses the prior art in objective quantitative indicators of infrared and visible light image fusion, thereby having wide application potential and being of great significance for further promoting the development of the image processing field.

[0168] Those skilled in the art can understand that although some embodiments herein include certain features rather than other features included in other embodiments, the combination of features of different embodiments means that it is within the scope of the present application and forms different embodiments. For example, any one of the claimed embodiments can be used in any combination.

[0169] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for realizing infrared and visible light image fusion, characterized in that: The method comprises: Performing feature extraction on the original infrared image and the original visible light image respectively to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image; A two-way pre-set multi-channel parallel cross-vision Mamba network model is used to extract global features from the initial infrared features and the initial visible light features, respectively, to obtain the infrared global features of the original infrared image and the visible light global features of the original visible light image; Two parallel preset multi-channel parallel KAG attention network models are used to extract local features of the initial infrared features and the initial visible light features respectively, obtaining the infrared local features of the original infrared image and the visible light local features of the original visible light image. Merging the infrared global feature and the visible light global feature, mapping the merged global fusion feature to the resolution of the original image, and obtaining a reconstructed global feature; Merging the infrared local features and the visible light local features, mapping the merged local fusion features to the resolution of the original image, and obtaining reconstructed local features; Image reconstruction is performed based on the reconstructed global features and local features to generate a fused image.

2. The method according to claim 1, characterized in that The multi-channel parallel cross-vision Mamba network model includes multiple multi-channel parallel cross-vision Mamba modules arranged in series, each multi-channel parallel cross-vision Mamba module includes a first normalization layer arranged in sequence, two Mamba modules arranged in parallel connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with a first adaptive adjustment operation layer.

3. The method according to claim 2, characterized in that The implementation process of the multi-channel parallel cross-vision Mamba module for global feature extraction includes: The first normalization layer is used to preprocess the initial infrared features or initial visible light features and split the output features into two feature vectors and Each feature vector includes 1 / 2 features of the original channel C, and the calculation process is as follows: The two feature vectors are input into two parallel Mamba modules for enhanced feature extraction. The enhanced feature extraction results are adaptively adjusted according to the preset adjustment factor through the first adaptive adjustment operation layer. The feature vectors processed by the two Mamba modules are cross-merged to generate two merged feature vectors. The operation process is as follows: The two merged feature vectors are connected by a residual fast connection mechanism using a residual connection layer and normalized by a second normalization layer, and the normalized connection features are projected to obtain the infrared global feature or the visible light global feature. The calculation process is as follows: Among them, LN(·) represents layer normalization, Sp(·) represents Split operation, Mamba(·) represents Mamba operation, θ represents the preset adjustment factor, Concat(·) represents concatenation operation, Pro(·) represents projection operation, i represents the number of layers of Mamba block, and R represents the input feature.

4. The method according to any one of claims 1 to 3, characterized in that The multi-channel parallel KAGtention network model includes multiple multi-channel parallel KAGtention modules arranged in series, each multi-channel parallel KAGtention module includes a third normalization layer arranged in sequence, four KAGN modules arranged in parallel connected to the output channel of the third normalization layer, a residual fast connection layer, a fourth normalization layer and a projection layer, and each KAGN module is provided with a second adaptive adjustment operation layer.

5. The method according to claim 4, characterized in that The implementation process of the multi-channel parallel KAGattention module for local feature extraction includes: The third normalization layer is used to preprocess the initial infrared features or initial visible light features and divide the output features into four feature vectors and Each feature vector includes 1 / 4 of the features of the original channel C. The calculation process is as follows: The four feature vectors are input into four parallel KAGN modules to extract local detail information features. The second adaptive adjustment operation layer adaptively adjusts the local detail information feature extraction results according to the preset adjustment factor. The operation process is as follows: The four adaptively adjusted output features are connected by a residual fast connection layer through a residual connection mechanism and normalized by a fourth normalization layer. The normalized connection features are projected to obtain the infrared local features or the visible light local features. The calculation process is as follows: Among them, LN(·) represents layer normalization, Sp(·) represents Split operation, KAGN(·) represents KAGN operation, θ represents the preset adjustment factor, Concat(·) represents concatenation operation, Pro(·) represents projection operation, j represents the number of layers of the KAGN module, and R represents the input feature.

6. The method according to claim 5, characterized in that The KAGN module consists of a set of univariate nonlinear functions: The convolution operation model of the KAGN module is: Among them, Iv i 、Ii r are visible light images and infrared images respectively, C represents the number of channels, H and W represent the height and width of image features respectively, k is the kernel size of the KAGN module, Each is a single variable nonlinear learnable function with trainable parameters. a, b, and d represent the a, b, and d layers in KAGN, respectively. Represents the univariate nonlinear learnable function of the a, b, and d layers, respectively, and the polynomial It is expressed as follows: b(x)=SiLU(x)=x / (1+e -x ) Among them, ω b represents weight, b(x) represents nonlinear activation function, polynomial Gram is an n-degree polynomial about x, for n = 0, 1, 2, ..., N-1, which is constructed based on two polynomials of different orders about The correlation of the weight function of , where δ(·) represents the Dirac function.

7. The method according to claim 4, characterized in that Mapping the merged global fusion features to the resolution of the original image to obtain the reconstructed global features includes: iterating the merged global fusion features multiple times using a plurality of multi-channel parallel cross-vision Mamba modules arranged in series to map the global fusion features to the resolution of the original image to obtain the global features; Mapping the merged local fusion features to the resolution of the original image to obtain the reconstructed local features includes: using multiple multi-channel parallel KAGattention modules arranged in series to iterate the merged local fusion features multiple times to map the local fusion features to the resolution of the original image to obtain the local features.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Construct the loss function, the loss function is L: L=λ1·L int +λ2·L grad +λ3·L dec Among them, L int Indicates strength loss, L grad represents the gradient loss, L dec represents the multi-branch feature decomposition loss, λ1λ2 and λ3 represent the weights of the corresponding loss functions respectively; L int The function form is as follows: Where H and W represent the height and width of the image feature respectively; L grad The functional form is as follows: in, Represents the Sobel gradient operator; L dec The functional form is as follows: Among them, CC(·) represents the correlation coefficient operation, Represent the visible light image I vi and infrared image I ir Visible light global features and infrared global features extracted by the multi-channel parallel cross-vision Mamba network model, and Respectively represent I vi and I ir Visible light local features and infrared local features extracted by the multi-channel parallel KAGattention network model.

9. A system for realizing infrared and visible light image fusion, characterized in that: The system includes a first convolution model, a PCVM encoder, a PKAGN encoder, a first channel addition module, a second channel addition module, a PCVM decoder, a PKAGN decoder and a second convolution model; The first convolution model is used to extract features from the original infrared image and the original visible light image respectively, to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image; The PCVM encoder is used to extract global features from the initial infrared features and the initial visible light features using two parallel preset multi-channel parallel cross-visual Mamba network models, thereby obtaining infrared global features that highlight the global information of the original infrared image and visible light global features that highlight the global information of the original visible light image. The PKAGN encoder is used to extract local features from the initial infrared features and the initial visible light features using two parallel preset multi-channel parallel KAG attention network models, thereby obtaining the infrared local features of the original infrared image and the visible light local features of the original visible light image. A first channel addition module is used to combine the infrared global feature and the visible light global feature to obtain a global fusion feature; PCVM decoder, used to map the global fusion features to the resolution of the original image to obtain the reconstructed global features; A second channel adding module is used to combine the infrared local features and the visible light local features to obtain a local fusion feature; PKAGN decoder, used to map the local fusion features to the resolution of the original image to obtain the reconstructed local features; The second convolutional model is used to reconstruct the image based on the reconstructed global features and local features to generate a fused image.

10. The system according to claim 9, characterized in that The PCVM encoder is implemented using two parallel multi-channel parallel cross-vision Mamba network models, each of which includes a plurality of multi-channel parallel cross-vision Mamba modules arranged in series, each of which includes a first normalization layer arranged in sequence, two parallel Mamba modules connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer, and a projection layer, and each Mamba module is provided with a first adaptive adjustment operation layer; The PKAGN encoder is implemented using two parallel multi-channel parallel KAGtention network models. Each multi-channel parallel KAGtention network model includes multiple multi-channel parallel KAGtention modules arranged in series. Each multi-channel parallel KAGtention module includes a third normalization layer arranged in sequence, four parallel KAGN modules connected to the output channel of the third normalization layer, a residual fast connection layer, a fourth normalization layer and a projection layer. Each KAGN module is provided with a second adaptive adjustment operation layer.

Images