Method and system for realizing infrared and visible light image fusion

By using the multi-channel parallel cross-visual Mamba network model and the multi-channel parallel KAGtention network model for global and local feature extraction, the problems of blurred details and texture loss in infrared and visible light images are solved, and efficient and low-parameter image fusion is achieved Effect.

CN119515696AActive Publication Date: 2025-02-25GUANGZHOU NANSHA DISTRICT BEIKE PHOTON SENSING TECH RES INST +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411572909.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-06
Publication Date
2025-02-25
Estimated Expiration
2044-11-06

AI Technical Summary

Technical Problem

The prior art has problems of blurred details and texture loss in the fusion process of infrared and visible light images, and the model parameters are large and the fusion effect is poor.

Method used

The multi-channel parallel cross-visual Mamba network model and the multi-channel parallel KAGtention network model are respectively used to extract global and local features, merge features and map them to the resolution of the original image to generate a fusion image.

Benefits of technology

It significantly improves the detail clarity and texture performance in the fusion image, reduces the amount of parameters of the model, and reduces the demand for computing resources while maintaining high accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119515696A_ABST
    Figure CN119515696A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of digital image processing, and provides a method and system for realizing infrared and visible light image fusion, and the method comprises the steps: extracting an initial infrared feature and an initial visible light feature; respectively carrying out global feature extraction on the initial infrared features and the initial visible light features by adopting a multi-channel parallel cross vision Mama network model to obtain infrared global features and visible light global features; performing local feature extraction on the initial infrared features and the initial visible light features by adopting a multi-channel parallel KAGtention network model to obtain infrared local features and visible light local features; performing combination and mapping reconstruction on the infrared global features and the visible light global features to obtain global features; the infrared local features and the visible light local features are combined and mapped and reconstructed to obtain local features; and generating a fused image according to the global features and the local features. According to the method, the quality of the fused image can be remarkably improved, and required computing resources are reduced while high precision is kept.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a method and system for realizing infrared and visible light image fusion. Background Art

[0002] As an advanced image processing method, multimodal image fusion technology aims to integrate image data from different sensing technologies and imaging modes into a single output image with huge information content. The core goal of this technology is to combine the advantages of each image source and overcome the limitations of a single image, thereby providing images with higher quality and information density. The application of multimodal image fusion technology is extremely wide, involving many important fields such as medical imaging, remote sensing, computer vision, and military monitoring.

[0003] At present, one of the typical multimodal image fusion tasks is the fusion of infrared and visible light images. The goal of this task is to make full use of the data of two different wavelength images to produce richer and more comprehensive information than a single image. Infrared images are favored for their unique ability to observe at night and penetrate smoke, while visible light images are known for their high-resolution detail display capabilities. However, in the existing technology, there are still problems such as detail blurring and texture loss in the fusion process of infrared and visible light images. Not only is the parameter volume of the model large, but the fusion effect is not good. Therefore, the main challenge facing the field of multimodal image fusion is: how to effectively protect and enhance the subtle texture structure in the original image while deeply fusing the information of multiple modal images, while suppressing noise interference, reducing unnecessary artifacts, and reducing the number of parameters of the fusion model. This is the key to promoting the development of this field. Specifically, the focus of the task is how to achieve complementary advantages of infrared and visible light images by fusing them - that is, not only retaining the thermal radiation characteristics in the infrared image, but also maintaining the detail clarity of the visible light image. Going further, the process aims to enhance key features in the source image, such as the shadows of leaves in low light conditions, the fine structures on the surface of buildings, and the texture of zebra crossings on the road, so as to improve the richness and recognition of the overall visual information. Summary of the invention

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present invention provides a method and system for realizing infrared and visible light image fusion.

[0005] One aspect of the present invention provides a method for realizing infrared and visible light image fusion, the method comprising:

[0006] Extracting features from the original infrared image and the original visible light image respectively to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image;

[0007] A two-way preset multi-channel parallel cross-vision Mamba network model is used to extract global features from the initial infrared features and the initial visible light features respectively, so as to obtain infrared global features that highlight the global information of the original infrared image and visible light global features that highlight the global information of the original visible light image.

[0008] Two preset multi-channel parallel KAGattention network models are used in parallel to extract local features of the initial infrared features and the initial visible light features, respectively, to obtain the infrared local features of the original infrared image and the visible light local features of the original visible light image.

[0009] Merging the infrared global feature and the visible light global feature, mapping the merged global fusion feature to the resolution of the original image, and obtaining a reconstructed global feature;

[0010] Merging the infrared local features and the visible light local features, mapping the merged local fusion features to the resolution of the original image, and obtaining reconstructed local features;

[0011] The image is reconstructed according to the reconstructed global features and local features to generate a fused image.

[0012] Furthermore, the multi-channel parallel cross-vision Mamba network model includes a plurality of multi-channel parallel cross-vision Mamba modules arranged in series, each multi-channel parallel cross-vision Mamba module includes a first normalization layer arranged in sequence, two Mamba modules arranged in parallel connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with a first adaptive adjustment operation layer.

[0013] Furthermore, the implementation process of the multi-channel parallel cross-vision Mamba module for global feature extraction includes:

[0014] The first normalization layer is used to preprocess the initial infrared features or initial visible light features and split the output features into two feature vectors and Each feature vector includes 1 / 2 features of the original channel C, and the calculation process is as follows:

[0015]

[0016] The two feature vectors are respectively input into two parallel Mamba modules for enhanced feature extraction, and the enhanced feature extraction results are adaptively adjusted according to the preset adjustment factor through the first adaptive adjustment operation layer, and the feature vectors processed by the two Mamba modules are cross-merged to generate two merged feature vectors. The operation process is as follows:

[0017]

[0018]

[0019] The two merged feature vectors are connected by a residual fast connection layer through a residual connection mechanism and normalized by a second normalization layer, and the normalized connection features are projected to obtain the infrared global feature or the visible light global feature. The operation process is as follows:

[0020]

[0021] Among them, LN(·) represents layer normalization, Sp(·) represents Split operation, Mamba(·) represents Mamba operation, θ represents the preset adjustment factor, Concat(·) represents concatenation operation, Pro(·) represents projection operation, i represents the number of layers of Mamba block, and R represents the input feature.

[0022] Furthermore, the multi-channel parallel KAGtention network model includes a plurality of multi-channel parallel KAGtention modules arranged in series, each multi-channel parallel KAGtention module includes a third normalization layer arranged in sequence, four KAGN modules arranged in parallel connected to the output channel of the third normalization layer, a residual fast connection layer, a fourth normalization layer and a projection layer, and each KAGN module is provided with a second adaptive adjustment operation layer.

[0023] Furthermore, the implementation process of the multi-channel parallel KAGattention module for local feature extraction includes:

[0024] The third normalization layer is used to preprocess the initial infrared features or initial visible light features, and the output features are divided into four feature vectors and Each feature vector includes 1 / 4 of the features of the original channel C, and the calculation process is as follows:

[0025]

[0026] The four feature vectors are respectively input into four KAGN modules set in parallel to extract local detail information features, and the local detail information feature extraction results are adaptively adjusted according to the preset adjustment factor through the second adaptive adjustment operation layer. The operation process is as follows:

[0027]

[0028] The four adaptively adjusted output features are connected by a residual fast connection layer through a residual connection mechanism and normalized by a fourth normalization layer, and the normalized connection features are projected to obtain the infrared local feature or the visible light local feature. The operation process is as follows:

[0029]

[0030]

[0031] Among them, LN(·) represents layer normalization, Sp(·) represents Split operation, KAGN(·) represents KAGN operation, θ represents the preset adjustment factor, Concat(·) represents concatenation operation, Pro(·) represents projection operation, j represents the number of layers of KAGN module, and R represents input features.

[0032] Furthermore, the KAGN module is composed of a set of univariate nonlinear functions:

[0033] The convolution operation model of the KAGN module is:

[0034]

[0035] Among them, I vi ,I ir are visible light images and infrared images respectively, C represents the number of channels, H and W represent the height and width of image features respectively, k is the kernel size of the KAGN module, Each is a univariate nonlinear learnable function with its own trainable parameters. a, b, and d represent the a, b, and d layers in KAGN, respectively. Represents the univariate nonlinear learnable functions of the a, b, and d layers, respectively, and the polynomial It is expressed as follows:

[0036]

[0037]

[0038] b(x)=SiLU(x)=x / (1+e -x )

[0039] Among them, ω b represents weight, b(x) represents nonlinear activation function, polynomial Gram is a polynomial of degree n about x, for n = 0, 1, 2, ..., N-1, which is constructed based on two polynomials of different orders about The correlation of the weight function of , where δ(·) represents the Dirac function.

[0040] Further, mapping the merged global fusion features to the resolution of the original image to obtain the reconstructed global features includes: using a plurality of multi-channel parallel cross-vision Mamba modules arranged in series to iterate the merged global fusion features multiple times to map the global fusion features to the resolution of the original image to obtain the global features;

[0041] The method of mapping the merged local fusion features to the resolution of the original image to obtain the reconstructed local features includes: using multiple multi-channel parallel KAGattention modules arranged in series to iterate the merged local fusion features multiple times to map the local fusion features to the resolution of the original image to obtain the local features.

[0042] Furthermore, the method further comprises:

[0043] Construct the loss function, the loss function is L:

[0044] L=λ1·L int +λ2·L grad +λ3·L dec

[0045] Among them, L int Indicates strength loss, L grad represents the gradient loss, L dec represents the multi-branch feature decomposition loss, λ1λ2 and λ3 represent the weights of the corresponding loss functions respectively;

[0046] L int The function form is as follows:

[0047]

[0048] Among them, H and W represent the height and width of the image feature respectively;

[0049] L grad The functional form is as follows:

[0050]

[0051] Where, ▽ represents the Sobel gradient operator;

[0052] L dec The functional form is as follows:

[0053]

[0054] Among them, CC(·) represents the correlation coefficient operation, Represent the visible light image I vi and infrared image I irVisible light global features and infrared global features extracted by multi-channel parallel cross-vision Mamba network model, and I vi and I ir Visible light local features and infrared local features extracted by the multi-channel parallel KAGattention network model.

[0055] Another aspect of the present invention provides a system for implementing infrared and visible light image fusion, the system comprising a first convolution model, a PCVM encoder, a PKAGN encoder, a first channel addition module, a second channel addition module, a PCVM decoder, a PKAGN decoder and a second convolution model;

[0056] The first convolution model is used to extract features from the original infrared image and the original visible light image respectively, so as to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image;

[0057] A PCVM encoder is used to extract global features from initial infrared features and initial visible light features respectively using two preset multi-channel parallel cross-vision Mamba network models in parallel, so as to obtain infrared global features that highlight global information of the original infrared image and visible light global features that highlight global information of the original visible light image;

[0058] The PKAGN encoder is used to extract local features of the initial infrared features and the initial visible light features respectively by using two preset multi-channel parallel KAGattention network models set in parallel, so as to obtain the infrared local features of the original infrared image and the visible light local features of the original visible light image;

[0059] A first channel addition module is used to merge the infrared global feature and the visible light global feature to obtain a global fusion feature;

[0060] PCVM decoder, used to map the global fusion features to the resolution of the original image to obtain the reconstructed global features;

[0061] A second channel adding module is used to combine the infrared local feature and the visible light local feature to obtain a local fusion feature;

[0062] PKAGN decoder, used to map the local fusion features to the resolution of the original image to obtain the reconstructed local features;

[0063] The second convolution model is used to reconstruct the image according to the reconstructed global features and local features to generate a fused image.

[0064] Furthermore, the PCVM encoder is implemented by two parallel multi-channel parallel cross-vision Mamba network models, each of which includes a plurality of multi-channel parallel cross-vision Mamba modules arranged in series, each of which includes a first normalization layer arranged in sequence, two parallel Mamba modules connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with a first adaptive adjustment operation layer;

[0065] The PKAGN encoder is implemented using two parallel multi-channel parallel KAGtention network models. Each multi-channel parallel KAGtention network model includes multiple multi-channel parallel KAGtention modules arranged in series. Each multi-channel parallel KAGtention module includes a third normalization layer arranged in sequence, four parallel KAGN modules connected to the output channel of the third normalization layer, a residual fast connection layer, a fourth normalization layer and a projection layer. Each KAGN module is provided with a second adaptive adjustment operation layer.

[0066] The method and system for realizing infrared and visible light image fusion provided by the present invention, by introducing a multi-channel cross-vision Mamba module and a multi-channel parallel KAGtention module, respectively perform global and local feature extraction, thereby enhancing the effect of image fusion and significantly improving the detail clarity and texture expression in the fused image. The present invention can reduce the required computing resources while maintaining high precision, reflecting its high efficiency and practicality in practical applications. Furthermore, the present invention adopts a multi-branch heterogeneous framework to process infrared and visible light images separately to make full use of their respective characteristics, which has a significant effect in improving the quality of fused images. In the task of infrared and visible light image fusion, the present invention not only performs well in subjective visual evaluation, but also surpasses the existing technology in the objective quantitative indicators of infrared and visible light image fusion. It has broad application potential and is of great significance to further promote the development of the image processing field. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0068] Figure 1 A network architecture diagram of a system for implementing infrared and visible light image fusion provided by an embodiment of the present invention;

[0069] Figure 2 A flowchart of a method for implementing infrared and visible light image fusion provided in an embodiment of the present invention;

[0070] FIG3( a ) is a network structure diagram of a multi-channel parallel cross-vision Mamba module provided in an embodiment of the present invention;

[0071] FIG3( b) is a network structure diagram of a Mamba module provided in an embodiment of the present invention;

[0072] FIG4( a ) is a network structure diagram of multiple multi-channel parallel KAGattention modules provided in an embodiment of the present invention;

[0073] FIG4( b ) is a network structure diagram of a KAGtention module provided in an embodiment of the present invention;

[0074] Figure 5 It is a schematic diagram comparing the fusion results of the existing image fusion method and the method for realizing infrared and visible light image fusion provided by the present invention on the LLVIP data set;

[0075] Figure 6 A bubble chart is provided to compare and analyze the efficiency and effect of the method for implementing infrared and visible light image fusion of the present invention and existing image fusion methods on the LLVIP dataset. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] In order to reduce the number of model parameters while ensuring the high performance of the infrared and visible light image fusion model, the present invention proposes an innovative method and system for realizing infrared and visible light image fusion. The present invention can significantly reduce the number of model parameters. At the same time, by using advanced image processing technology and optimized fusion strategies, it not only retains the key features of the original image, but also significantly improves the overall visual effect of the fused image. The present invention is mainly composed of three key modules: a multi-branch heterogeneous encoder, responsible for feature extraction and decomposition; a multi-branch heterogeneous decoder, used for image reconstruction and generation; and a set of carefully designed loss functions, used to optimize the performance of the model training process.

[0078] Figure 1 FIG. 1 shows a network architecture diagram of a system for realizing infrared and visible light image fusion according to the present invention. Figure 1 As shown, the system includes a first convolution model, a PCVM encoder, a PKAGN encoder, a first channel addition module, a second channel addition module, a PCVM decoder, a PKAGN decoder and a second convolution model.

[0079] The first convolution model includes two convolution blocks, and the outputs of the two convolution blocks are respectively connected to the inputs of the PCVM encoder and the PKAGN encoder, and are used to extract features of the original infrared image and the original visible light image, respectively, to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image;

[0080] A PCVM encoder is used to extract global features from initial infrared features and initial visible light features respectively using two preset multi-channel parallel cross-vision Mamba network models in parallel, so as to obtain infrared global features that highlight global information of the original infrared image and visible light global features that highlight global information of the original visible light image;

[0081] The PKAGN encoder is used to extract local features of the initial infrared features and the initial visible light features respectively by using two preset multi-channel parallel KAGattention network models set in parallel, so as to obtain the infrared local features of the original infrared image and the visible light local features of the original visible light image;

[0082] A first channel addition module is used to merge the infrared global feature and the visible light global feature to obtain a global fusion feature;

[0083] PCVM decoder, used to map the global fusion features to the resolution of the original image to obtain the reconstructed global features;

[0084] A second channel adding module is used to combine the infrared local feature and the visible light local feature to obtain a local fusion feature;

[0085] PKAGN decoder, used to map the local fusion features to the resolution of the original image to obtain the reconstructed local features;

[0086] The second convolution model is used to reconstruct the image according to the reconstructed global features and local features to generate a fused image.

[0087] Furthermore, the second convolution model includes two convolution blocks and an element multiplication operation module connected to the outputs of the two convolution blocks; one of the convolution blocks is used to extract the global features output by the PCVM decoder to obtain a first fused feature, and the other convolution block is used to extract the local features output by the PKAGN decoder to obtain a second fused feature, and the element multiplication operation module is used to perform element-by-element multiplication on the first fused feature and the second fused feature to generate a final fused image.

[0088] exist Figure 1 In a specific embodiment shown, the PCVM encoder is implemented using two multi-channel parallel cross-vision Mamba network models arranged in parallel, and each multi-channel parallel cross-vision Mamba network model includes a plurality of multi-channel parallel cross-vision Mamba modules arranged in series. The output features of the multi-channel parallel cross-vision Mamba module of the upper layer in the multi-channel parallel cross-vision Mamba network model are used as the input of the multi-channel parallel cross-vision Mamba module of the next layer to achieve multi-level iterative feature extraction. In this embodiment, three multi-channel parallel cross-vision Mamba modules are arranged in series as an example for explanation. Furthermore, each multi-channel parallel cross-vision Mamba module includes a first normalization layer arranged in sequence, two Mamba modules arranged in parallel connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with a first adaptive adjustment operation layer.

[0089] The Mamba module is implemented using the Mamba model, the improved S4, which is an advanced sequence modeling method that can maintain high efficiency when processing complex data. It dynamically adapts to different scenarios by combining input-related state space adjustments and linear time complexity design, effectively capturing long-term dependencies in sequences. Mamba shows higher efficiency and competitiveness when processing long sequences, providing a new, high-performance solution for sequence data analysis. The Mamba series of models have shown extraordinary performance in the field of computer vision.

[0090] exist Figure 1In a specific embodiment shown, the PKAGN encoder is implemented using two multi-channel parallel KAGtention network models arranged in parallel, and each multi-channel parallel KAGtention network model includes multiple multi-channel parallel KAGtention modules arranged in series. The output features of the multi-channel parallel KAGtention module of the previous layer in the multi-channel parallel KAGtention network model are used as the input of the multi-channel parallel KAGtention module of the next layer to achieve multi-level iterative feature extraction. In this embodiment, three multi-channel parallel KAGtention modules are arranged in series as an example for explanation. Furthermore, each multi-channel parallel KAGtention module includes a third normalization layer arranged in sequence, four KAGN modules arranged in parallel connected to the output channel of the third normalization layer, a residual fast connection layer, a fourth normalization layer and a projection layer, and each KAGN module is provided with a second adaptive adjustment operation layer.

[0091] The KAGN module is implemented using the Kolmogorov-Arnold network (KAN), a new architecture based on the Kolmogorov-Arnold (KN) representation theorem that can efficiently and flexibly simulate complex functions with fewer parameters. Each weight of the network is represented by a learnable univariate function, which not only reduces the number of parameters but also enhances the interpretability of the model. With a solid mathematical theoretical foundation, KAN shows great potential in image processing tasks.

[0092] In this embodiment, the visible light image and the infrared image are represented as I vi and I ir , I F Represents the fused image. The PCVM encoder and PKAGN encoder network models each have three layers, and their input and output channel configurations are (4, 8), (8, 16), and (16, 32), respectively.

[0093]

[0094]

[0095] in, I vi and I ir The features extracted by the PCVM encoder are Represent the features of two types of images obtained by PKAGN encoder. PCVM(·), PKAGN(·) represent the multi-channel parallel cross-vision Mamba module processing and multi-channel parallel KAGtention module operation processing, respectively, and Conv(·) represents the convolution operation of the convolution block. The features from these two different encoders are further merged by the channel addition module for subsequent processing.

[0096]

[0097] Wherein, Concat(·) represents a concatenation operation.

[0098] In this embodiment, since the multi-channel parallel cross vision Mamba module (Multi-Channel ParallelCross Vision Mamba, PCVM) has parallel processing capabilities, it can process multiple feature maps simultaneously during the fusion process. The module can effectively integrate information from different modalities, and contains a mechanism to identify complementary information and how to merge this information without compromising image quality. The Mamba module, due to its selective state space characteristics, can dynamically select valuable features or information during the fusion process, which is essential for maintaining key features of the image (such as edges and textures).

[0099] The application of the Multi-Channel Parallel KAGtention (PKAGN) module in infrared and visible light image fusion is mainly to improve the quality and efficiency of the fused image. The Kolmogorov-Arnold model is a mathematical model that shows that any multivariable function can be approximated by a two-parameter function. In image processing, the KAN model can be used to describe the complex relationship between the pixel values ​​in the image and other related factors. In the process of infrared and visible light image fusion, the use of the PKAGN module can help achieve more efficient, more stable and higher quality image fusion effects.

[0100] The application of the PCVM module and the PKAGN module shows significant complementarity. The PCVM module focuses on capturing the global information of the image, which not only deepens the understanding of the overall structure of the image, but also enhances the understanding of the relationship between the various objects in the image. In contrast, the PKAGN module pays special attention to extracting local features of the image, and can efficiently identify subtle differences in the image, such as textures, edges, and other local patterns. When the PCVM module is combined with the PKAGN module, this combination not only gives the model a strong global perception ability, enabling it to better understand and analyze the overall layout of the image, but also ensures that the model maintains a high sensitivity to the subtleties of the image and can accurately capture important local information.

[0101] Figure 2 The method for realizing infrared and visible light image fusion provided by the embodiment of the present invention is applicable to the following embodiments: Figure 1 The system for realizing infrared and visible light image fusion is shown in FIG. Figure 2 As shown, the method for realizing infrared and visible light image fusion proposed in the embodiment of the present invention specifically includes the following steps:

[0102] S11 . Extract features from the original infrared image and the original visible light image respectively to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image.

[0103] In the embodiment of the present invention, a collection device capable of simultaneously capturing visible light images and infrared images may be used to simultaneously acquire visible light images and infrared images, or different collection devices may be used to respectively acquire visible light images and infrared images. The embodiment of the present invention does not specifically limit this.

[0104] Before entering the PCVM encoder or PKAGN encoder, a convolution block is set before both types of encoders, where the convolution kernel size is 3×3, the number of input channels is 1, and the number of output channels is 4. Next, the output features of the convolution block enter the PCVM encoder or PKAGN encoder for feature extraction.

[0105] S12, using two parallel preset multi-channel parallel cross-vision Mamba network models to perform global feature extraction on the initial infrared features and the initial visible light features, respectively, to obtain the infrared global features of the original infrared image and the visible light global features of the original visible light image. Specifically, the initial infrared features and the initial visible light features output by the convolution block set before the PCVM encoder are respectively input into the two parallel multi-channel parallel cross-vision Mamba network models set in parallel in the PCVM encoder, and the corresponding multi-channel parallel cross-vision Mamba network models are used to perform global feature extraction on the initial infrared features and the initial visible light features.

[0106] S13, using two preset multi-channel parallel KAGtention network models set in parallel to perform local feature extraction on the initial infrared features and the initial visible light features, respectively, to obtain the infrared local features of the original infrared image and the visible light local features of the original visible light image. Specifically, the initial infrared features and the initial visible light features output by the convolution block set before the PKAGN encoder are respectively input into the two preset multi-channel parallel KAGtention network models set in parallel in the PKAGN encoder, and the corresponding multi-channel parallel KAGtention network models are used to perform local feature extraction on the initial infrared features and the initial visible light features.

[0107] S14, merging the infrared global feature and the visible light global feature, mapping the merged global fusion feature to the resolution of the original image, and obtaining a reconstructed global feature.

[0108] S15. Merge the infrared local features and the visible light local features, and map the merged local fusion features to the resolution of the original image to obtain reconstructed local features.

[0109] S16. Reconstruct the image according to the reconstructed global features and local features to generate a fused image. Specifically, a convolution block may be used to extract the global features to obtain a first fused feature, another convolution block may be used to extract the local features to obtain a second fused feature, and then the first fused feature and the second fused feature may be multiplied element by element to generate a final fused image.

[0110] The method for realizing infrared and visible light image fusion provided by the present invention introduces a multi-channel cross-vision Mamba module and a multi-channel parallel KAGtention module to perform global and local feature extraction respectively, thereby enhancing the effect of image fusion, so that the present invention can reduce the required computing resources while maintaining high precision, reflecting its high efficiency and practicality in practical applications. Furthermore, the present invention adopts a multi-branch heterogeneous framework to process infrared and visible light images separately to make full use of their respective characteristics, which has a significant effect in improving the quality of fused images. In the task of infrared and visible light image fusion, the present invention not only performs well in subjective visual evaluation, but also surpasses the existing technology in the objective quantitative indicators of infrared and visible light image fusion. It has broad application potential and is of great significance to further promote the development of the image processing field.

[0111] In one embodiment of the present invention, the multi-channel parallel cross-vision Mamba network model includes a plurality of multi-channel parallel cross-vision Mamba modules arranged in series. As shown in FIG3(a), each multi-channel parallel cross-vision Mamba module includes a first normalization layer arranged in sequence, two Mamba modules arranged in parallel connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with a first adaptive adjustment operation layer. Among them, the specific network structure of the Mamba module is shown in FIG3(b). The Mamba module includes a first linear layer arranged in parallel, a second linear layer, a first activation function operation module connected to the output channel of the first linear layer, a convolution block connected to the output channel of the second linear layer in sequence, a second activation function operation module and a state space equation module, an element multiplication operation module connected to the output end of the first activation function operation module and the output end of the state space equation module, and a third linear layer.

[0112] In the PCVM encoder, first, the input features are preprocessed by the LN normalization layer and then split into two feature vectors and The number of channels of each feature vector is C / 2. Then, and Each is sent to the corresponding multi-channel parallel cross-vision Mamba module for independent processing. Inside the multi-channel parallel cross-vision Mamba module, the processed output is integrated using residual connections and adjustment factors. This design is designed to enhance the model's ability to capture long-range spatial information. Subsequently, the two processed feature vectors are crossed and finally merged into a single feature vector through the Concat operation. Subsequently, the feature vector is double-transformed by the LN layer and the Projection layer to generate the output feature

[0113] Specifically, the implementation process of the multi-channel parallel cross-vision Mamba module for global feature extraction includes:

[0114] The first normalization layer is used to preprocess the initial infrared features or initial visible light features and split the output features into two feature vectors and Each feature vector includes 1 / 2 features of the original channel C, and the calculation process is as follows:

[0115]

[0116] The two feature vectors are respectively input into two parallel Mamba modules for enhanced feature extraction, and the enhanced feature extraction results are adaptively adjusted according to the preset adjustment factor through the first adaptive adjustment operation layer, and the feature vectors processed by the two Mamba modules are cross-merged to generate two merged feature vectors. The operation process is as follows:

[0117]

[0118]

[0119] The two merged feature vectors are connected by a residual fast connection layer through a residual connection mechanism and normalized by a second normalization layer, and the normalized connection features are projected to obtain the infrared global feature or the visible light global feature. The operation process is as follows:

[0120]

[0121] Among them, LN(·) represents layer normalization, Sp(·) represents Split operation, Mamba(·) represents Mamba operation, θ represents the preset adjustment factor, Concat(·) represents concatenation operation, Pro(·) represents projection operation, i represents the number of layers of Mamba block, and R represents the input feature.

[0122] In one embodiment of the present invention, the multi-channel parallel KAGtention network model includes a plurality of multi-channel parallel KAGtention modules arranged in series. As shown in FIG4(a), each multi-channel parallel KAGtention module includes a third normalization layer arranged in sequence (not shown in the figure), four KAGN modules arranged in parallel connected to the output channel of the third normalization layer, a residual fast connection layer, a fourth normalization layer and a projection layer (not shown in the figure), and each KAGN module is provided with a second adaptive adjustment operation layer. Among them, the specific network structure of the KAGtention module is shown in FIG4(b). Referring to FIG4, the KAGtention module is composed of a set of univariate nonlinear functions.

[0123] In the PKAGN encoder, the input features pass through the LN layer and are divided into four features: and The number of channels for each feature is C / 4. These independent features are fed into the multi-channel parallel KAGattention network model for processing, and their outputs are combined with adjustment factors through residual connections to enhance the model's ability to capture long-range spatial information. After this step, the four processed features are concat-operated and finally merged into a feature vector. The vector is then double-transformed by the LN layer and the Projection layer to generate the final output feature.

[0124] Specifically, the implementation process of the multi-channel parallel KAGattention module for local feature extraction includes:

[0125] The third normalization layer is used to preprocess the initial infrared features or initial visible light features, and the output features are divided into four feature vectors and Each feature vector includes 1 / 4 of the features of the original channel C, and the calculation process is as follows:

[0126]

[0127] The four feature vectors are respectively input into four KAGN modules set in parallel to extract local detail information features, and the local detail information feature extraction results are adaptively adjusted according to the preset adjustment factor through the second adaptive adjustment operation layer. The operation process is as follows:

[0128]

[0129] The four adaptively adjusted output features are connected by a residual fast connection layer through a residual connection mechanism and normalized by a fourth normalization layer, and the normalized connection features are projected to obtain the infrared local feature or the visible light local feature. The operation process is as follows:

[0130]

[0131]

[0132] Among them, LN(·) represents layer normalization, Sp(·) represents Split operation, KAGN(·) represents KAGN operation, θ represents the preset adjustment factor, Concat(·) represents concatenation operation, Pro(·) represents projection operation, j represents the number of layers of KAGN module, and R represents input features.

[0133] Furthermore, in the multi-channel parallel KAGattention network model, the KAGN module consists of a set of univariate nonlinear functions and is applied to infrared and visible light images. vi ,I ir ∈R c×m×n , where C represents the number of channels, H and W represent the height and width of the image respectively. Therefore, for a kernel of size k, the convolution operation model of the KAGN module in the KAN framework is:

[0134]

[0135] Among them, I vi ,I ir are visible light images and infrared images respectively, C represents the number of channels, H and W represent the height and width of image features respectively, k is the kernel size of the KAGN module, Each is a univariate nonlinear learnable function with its own trainable parameters. a, b, and d represent the a, b, and d layers in KAGN, respectively. Represents the univariate nonlinear learnable functions of the a, b, and d layers, respectively, and the polynomial It is expressed as follows:

[0136]

[0137]

[0138] b(x)=SiLU(x)=x / (1+e -x )

[0139] Among them, ω b represents weight, b(x) represents nonlinear activation function, polynomial Gram is a polynomial of degree n about x, for n = 0, 1, 2, ..., N-1, which is constructed based on two polynomials of different orders about , where δ(·) represents the Dirac function. KAGN not only allows efficient parameter fine-tuning of the pre-trained model, but also reduces the number of parameters that need to be trained.

[0140] In KAGN, applying regularization techniques to Kolmogorov-Arnold convolutions involves the operation of polynomial Grams. Specifically, KAGN first computes the polynomial basis on the variable x and then performs a weighted sum on it. Therefore, KAGN provides three different locations for applying dropout: before layer processing (Full), before computing the polynomial basis (Poly), and before applying weights to the polynomial (Degree).

[0141] Further, mapping the merged global fusion features to the resolution of the original image to obtain the reconstructed global features includes: using a plurality of multi-channel parallel cross-vision Mamba modules arranged in series to iterate the merged global fusion features multiple times to map the global fusion features to the resolution of the original image to obtain the global features. Mapping the merged local fusion features to the resolution of the original image to obtain the reconstructed local features includes: using a plurality of multi-channel parallel KAGattention modules arranged in series to iterate the merged local fusion features multiple times to map the local fusion features to the resolution of the original image to obtain the local features.

[0142] In an embodiment of the present invention, the multi-channel parallel cross-visual Mamba module and the multi-channel parallel KAGtention module in the PCVM decoder and the PKAGN decoder have the same network structure as the corresponding encoder, and the only difference is the number of input and output channels, so as to realize channel merging. Specifically, first, the obtained global fusion features and local fusion features are decomposed and assigned to the corresponding PCVM decoder and PKAGN decoder respectively. Then, these decomposed feature maps are spliced ​​along the channel dimension, while ensuring that the spatial size and pixel value range of the feature map are consistent; finally, the spliced ​​feature map is passed as input to the decoder, and the number of channels is reduced layer by layer until the final image representation is output, wherein the number of input and output channels of each layer of the decoder is configured as (64, 32), (32, 16), and (16, 8), respectively. This process not only helps to enhance the efficiency of the decoder in utilizing feature information, but also promotes the generation of more accurate and detailed output images.

[0143] In the decoder of the present invention, the decomposed features are concatenated along the channel dimension as input, and two types of decoder outputs (PCVM decoder and PKAGN decoder) and the final fused image are generated. The decoder has the same number of layers as the encoder, that is, both have 3 layers, and the input and output channel number configurations of each layer are (64, 32), (32, 16) and (16, 8) respectively. The feature representation of the decoder is as follows:

[0144]

[0145] in, and Respectively represent the features processed by the PCVM decoder and the PKAGN decoder. This design ensures that the decoder can gradually restore the spatial details of the original image while maintaining sufficient channel depth to capture complex feature information. Therefore, the fused image finally obtained by the present invention is:

[0146]

[0147] The number of input and output channels of Conv(·) is (8,1).

[0148] The loss function is crucial in the infrared and visible light image fusion model. It measures the difference between the model output and the real target, and guides the model to optimize parameters by minimizing this difference, thereby effectively fusing image features and improving the quality of the output image. In this embodiment of the present invention, the constructed loss function L is defined as

[0149] L=λ1·L int +λ2·L grad +λ3·L dec

[0150] Among them, L int Indicates strength loss, L grad represents the gradient loss, L dec represents the multi-branch feature decomposition loss, λ1λ2 and λ3 represent the weights of the corresponding loss functions respectively;

[0151] L int The main function of image fusion is to maintain and optimize the brightness information and contrast of the image, ensuring that the fused image can not only reflect the thermal radiation characteristics of the infrared image, but also retain the good visual effects and details of the visible light image. int The function form is as follows:

[0152]

[0153] Among them, H and W represent the height and width of the image feature respectively;

[0154] Lgrad The role of the present invention is to encourage the model to maintain the edge and detail information of the image during the training process. It helps the model learn more accurate boundary and texture features by penalizing the consistency difference between the gradient information in the fused image and the source image (infrared image and visible light image), thereby generating a clearer and more detailed fused image. grad The functional form is as follows:

[0155]

[0156] Where, ▽ represents the Sobel gradient operator;

[0157] In order to improve the performance and effect of multimodal image fusion tasks, L dec By decomposing the feature space, the PCVM encoder and PKAGN encoder can more efficiently capture the correlation and information exchange between different modalities. dec The functional form is as follows:

[0158]

[0159] Among them, CC(·) represents the correlation coefficient operation, Represent the visible light image I vi and infrared image I ir Visible light global features and infrared global features extracted by multi-channel parallel cross-vision Mamba network model, and I vi and I ir Visible light local features and infrared local features extracted by the multi-channel parallel KAGattention network model.

[0160] A specific example is given below to verify and illustrate the beneficial effects of the method for realizing infrared and visible light image fusion proposed by the present invention.

[0161] In this embodiment, the MSRS dataset containing 1083 images, 300 M3FD images and 300 LLVIP images are used to train the framework. The pixel size of these images is set to 64×64 as training samples. During the training process, 30 epochs are set, of which the first 5 epochs maintain a fixed learning rate, and the exponential decay strategy is used to adjust the learning rate in the next 25 epochs. The initial learning rate is 0.001, the momentum is set to 0.9, and the weight decay is 10-4. Through a series of enumeration experiments, the 10th epoch is determined to be the optimal training cycle for the model. Optionally, for the weight parameters of the corresponding loss function, λ1, λ2 and λ3 are set to 1, 100 and 100 respectively, which is based on preliminary parameter tuning experiments.

[0162] Figure 5 Table 1 and Table 2 systematically summarize and show the quantitative evaluation results of ten compared algorithms on six common evaluation indicators. The evaluation criteria adopt information entropy (EN), spatial frequency (SF), average gradient (AG), sum of correlation differences (SCD), correlation coefficient (CC) and multi-scale structural similarity index measure (MS-SSIM). For each evaluation indicator, the average score is calculated based on 50 pairs of test images in the LLVIP dataset, and the best result is marked in bold, the second best result is marked with an underline, and the third best result is displayed in italics to enhance the visual contrast effect. The analysis results show that the method for realizing infrared and visible light image fusion proposed in the present invention has achieved the best level in all six evaluation indicators. This detailed data analysis not only highlights the excellent performance of the present invention in objective performance evaluation, but also provides strong evidence for the effectiveness and necessity of the introduced PCVM and PKAGN module design.

[0163] Table 1. Average index values ​​obtained by existing fusion methods and the present invention on the LLVIP dataset

[0164] EN↑ SF↑ AG↑ SD↑ SCD↑ MS_SSIM↑ <![CDATA[CrossFuse Li ]]> 6.324 9.316 2.474 24.683 0.882 0.908 <![CDATA[CrossFuse Wang ]]> 6.367 13.833 3.645 37.227 1.177 0.900 DSFusion

[52] 6.517 14.380 4.172 35.272 1.126 0.947 LRRNet

[53] 6.348 10.565 2.865 28.931 0.983 0.800 MBHFuse 7.255 14.596 4.118 46.574 1.669 1.045 MFEIF 6.783 8.583 2.451 39.635 1.543 1.034 PIAFusion 7.182 13.883 4.087 44.890 1.539 1.029 SemLA 6.849 9.631 2.730 36.571 1.345 0.890 YDTR 6.527 8.260 2.099 29.940 1.064 0.929 Ours 7.315 15.466 4.476 47.982 1.821 1.058

[0165] Figure 5 A schematic diagram showing the comparison of fusion results on the LLVIP dataset between the existing image fusion method and the method for realizing infrared and visible light image fusion provided by the present invention.

[0166] Figure 6A bubble chart comparing the efficiency and effectiveness of the present invention and the current SOTA method on the LLVIP dataset is shown. The horizontal axis of the chart represents the average inference time of different frameworks on the LLVIP dataset, which reflects the computational efficiency of the model; the vertical axis represents the normalized sum of the six evaluation indicators, which is used to measure the overall performance of the model. In addition, the size of the circle in the figure and the number marked in the circle represent the number of parameters of each model, which reflects the complexity of the model and the demand for computing resources. Figure 6 It can be clearly seen that the present invention not only leads in comprehensive performance, but also has the smallest number of model parameters. This outstanding performance is mainly attributed to the PCVM and PKAGN proposed in the present invention. The former enhances the feature extraction capability through an efficient cross-channel information exchange mechanism, thereby improving the model performance without adding too many parameters; the latter uses KAN to effectively capture important features while maintaining low computational complexity. The design of these two modules effectively balances the performance and efficiency of the model, allowing the present invention to reduce the required computing resources while maintaining high accuracy, reflecting its efficiency and practicality in practical applications.

[0167] The method and system for realizing infrared and visible light image fusion provided by the present invention, by introducing a multi-channel cross-vision Mamba module and a multi-channel parallel KAGtention module, respectively perform global and local feature extraction, thereby enhancing the effect of image fusion and significantly improving the detail clarity and texture expression in the fused image. The present invention can reduce the required computing resources while maintaining high precision, reflecting its high efficiency and practicality in practical applications. Furthermore, the present invention adopts a multi-branch heterogeneous framework to process infrared and visible light images separately to make full use of their respective characteristics, which has a significant effect in improving the quality of fused images. In the task of infrared and visible light image fusion, the present invention not only performs well in subjective visual evaluation, but also surpasses the existing technology in the objective quantitative indicators of infrared and visible light image fusion. It has broad application potential and is of great significance to further promote the development of the image processing field.

[0168] Those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for realizing infrared and visible light image fusion, characterized in that: The method comprises: Extracting features from the original infrared image and the original visible light image respectively to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image; Two preset multi-channel parallel cross-vision Mamba network models are used in parallel to extract global features from the initial infrared features and the initial visible light features, respectively, to obtain the infrared global features of the original infrared image and the visible light global features of the original visible light image; Two preset multi-channel parallel KAGattention network models are used in parallel to extract local features of the initial infrared features and the initial visible light features, respectively, to obtain the infrared local features of the original infrared image and the visible light local features of the original visible light image. Merging the infrared global feature and the visible light global feature, mapping the merged global fusion feature to the resolution of the original image, and obtaining a reconstructed global feature; Merging the infrared local features and the visible light local features, mapping the merged local fusion features to the resolution of the original image, and obtaining reconstructed local features; The image is reconstructed according to the reconstructed global features and local features to generate a fused image.

2. The method according to claim 1, characterized in that The multi-channel parallel cross-vision Mamba network model includes a plurality of multi-channel parallel cross-vision Mamba modules arranged in series, each multi-channel parallel cross-vision Mamba module includes a first normalization layer arranged in sequence, two Mamba modules arranged in parallel and connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with a first adaptive adjustment operation layer.

3. The method according to claim 2, characterized in that The implementation process of the multi-channel parallel cross-vision Mamba module for global feature extraction includes: The first normalization layer is used to preprocess the initial infrared features or initial visible light features and split the output features into two feature vectors and Each feature vector includes 1 / 2 features of the original channel C, and the calculation process is as follows: The two feature vectors are respectively input into two parallel Mamba modules for enhanced feature extraction, and the enhanced feature extraction results are adaptively adjusted according to the preset adjustment factor through the first adaptive adjustment operation layer, and the feature vectors processed by the two Mamba modules are cross-merged to generate two merged feature vectors. The operation process is as follows: The two merged feature vectors are connected by a residual fast connection layer through a residual connection mechanism and normalized by a second normalization layer, and the normalized connection features are projected to obtain the infrared global feature or the visible light global feature. The operation process is as follows: Among them, LN(·) represents layer normalization, Sp(·) represents Split operation, Mamba(·) represents Mamba operation, θ represents the preset adjustment factor, Concat(·) represents concatenation operation, Pro(·) represents projection operation, i represents the number of layers of Mamba block, and R represents the input feature.

4. The method according to any one of claims 1 to 3, characterized in that: The multi-channel parallel KAGtention network model includes a plurality of multi-channel parallel KAGtention modules arranged in series, each multi-channel parallel KAGtention module includes a third normalization layer arranged in sequence, four KAGN modules arranged in parallel and connected to the output channel of the third normalization layer, a residual fast connection layer, a fourth normalization layer and a projection layer, and each KAGN module is provided with a second adaptive adjustment operation layer.

5. The method according to claim 4, characterized in that The implementation process of the multi-channel parallel KAGattention module for local feature extraction includes: The third normalization layer is used to preprocess the initial infrared features or initial visible light features, and the output features are divided into four feature vectors and Each feature vector includes 1 / 4 of the features of the original channel C, and the calculation process is as follows: The four feature vectors are respectively input into four KAGN modules set in parallel to extract local detail information features, and the local detail information feature extraction results are adaptively adjusted according to the preset adjustment factor through the second adaptive adjustment operation layer. The operation process is as follows: The four adaptively adjusted output features are connected by a residual fast connection layer through a residual connection mechanism and normalized by a fourth normalization layer, and the normalized connection features are projected to obtain the infrared local feature or the visible light local feature. The operation process is as follows: Among them, LN(·) represents layer normalization, Sp(·) represents Split operation, KAGN(·) represents KAGN operation, θ represents the preset adjustment factor, Concat(·) represents concatenation operation, Pro(·) represents projection operation, j represents the number of layers of KAGN module, and R represents input features.

6. The method according to claim 5, characterized in that The KAGN module consists of a set of univariate nonlinear functions: The convolution operation model of the KAGN module is: Among them, Iv i ,Ii r are visible light images and infrared images respectively, C represents the number of channels, H and W represent the height and width of image features respectively, k is the kernel size of the KAGN module, is a univariate nonlinear learnable function with trainable parameters. a, b, and d represent the a, b, and d layers in KAGN, respectively. Represents the univariate nonlinear learnable functions of the a, b, and d layers, respectively, and the polynomial It is expressed as follows: b(x)=SiLU(x)=x / (1+e -x ) Among them, ω b represents weight, b(x) represents nonlinear activation function, polynomial Gram is a polynomial of degree n about x, for n = 0, 1, 2, ..., N-1, which is constructed based on two polynomials of different orders about The correlation of the weight function of , where δ(·) represents the Dirac function.

7. The method according to claim 4, characterized in that The step of mapping the merged global fusion features to the resolution of the original image to obtain the reconstructed global features comprises: using a plurality of multi-channel parallel cross-vision Mamba modules arranged in series to iterate the merged global fusion features multiple times to map the global fusion features to the resolution of the original image to obtain the global features; The method of mapping the merged local fusion features to the resolution of the original image to obtain the reconstructed local features includes: using multiple multi-channel parallel KAGattention modules arranged in series to iterate the merged local fusion features multiple times to map the local fusion features to the resolution of the original image to obtain the local features.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Construct the loss function, the loss function is L: L=λ1·L int +λ2·L grad +λ3·L dec Among them, L int Indicates strength loss, L grad represents the gradient loss, L dec represents the multi-branch feature decomposition loss, λ1λ2 and λ3 represent the weights of the corresponding loss functions respectively; L int The function form is as follows: Among them, H and W represent the height and width of the image feature respectively; L grad The function form is as follows: in, Represents Sobel gradient operator; L dec The function form is as follows: Among them, CC(·) represents the correlation coefficient operation, Represent the visible light image I vi and infrared image I ir Visible light global features and infrared global features extracted by multi-channel parallel cross-vision Mamba network model, and I vi and I ir Visible light local features and infrared local features extracted by the multi-channel parallel KAGattention network model.

9. A system for realizing infrared and visible light image fusion, characterized in that: The system includes a first convolution model, a PCVM encoder, a PKAGN encoder, a first channel addition module, a second channel addition module, a PCVM decoder, a PKAGN decoder and a second convolution model; The first convolution model is used to extract features from the original infrared image and the original visible light image respectively, so as to obtain initial infrared features of the original infrared image and initial visible light features of the original visible light image; A PCVM encoder is used to extract global features from initial infrared features and initial visible light features respectively using two preset multi-channel parallel cross-vision Mamba network models in parallel, so as to obtain infrared global features that highlight global information of the original infrared image and visible light global features that highlight global information of the original visible light image; The PKAGN encoder is used to extract local features of the initial infrared features and the initial visible light features respectively by using two preset multi-channel parallel KAGattention network models set in parallel, so as to obtain the infrared local features of the original infrared image and the visible light local features of the original visible light image; A first channel addition module is used to merge the infrared global feature and the visible light global feature to obtain a global fusion feature; PCVM decoder, used to map the global fusion features to the resolution of the original image to obtain the reconstructed global features; A second channel adding module is used to combine the infrared local feature and the visible light local feature to obtain a local fusion feature; PKAGN decoder, used to map the local fusion features to the resolution of the original image to obtain the reconstructed local features; The second convolution model is used to reconstruct the image according to the reconstructed global features and local features to generate a fused image.

10. The system according to claim 9, characterized in that The PCVM encoder is implemented by two parallel multi-channel parallel cross-vision Mamba network models, each of which includes a plurality of multi-channel parallel cross-vision Mamba modules arranged in series, each of which includes a first normalization layer arranged in sequence, two parallel Mamba modules connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with a first adaptive adjustment operation layer; The PKAGN encoder is implemented using two parallel multi-channel parallel KAGtention network models. Each multi-channel parallel KAGtention network model includes multiple multi-channel parallel KAGtention modules arranged in series. Each multi-channel parallel KAGtention module includes a third normalization layer arranged in sequence, four parallel KAGN modules connected to the output channel of the third normalization layer, a residual fast connection layer, a fourth normalization layer and a projection layer. Each KAGN module is provided with a second adaptive adjustment operation layer.

Citation Information

Patent Citations

  • Multi-scale infrared and visible light image fusion method based on bipartial differential equation

    CN117745555A

  • Rapid infrared and visible light image fusion method based on local and global information interaction, electronic equipment and storage medium

    CN118172256A

  • Double-branch infrared and visible light image fusion method and system

    CN118411298A

  • Infrared and visible light image fusion method for improving image details and textures

    CN119090739A

  • Infrared and visible light fusion method

    US20220044374A1