A method for fusion of infrared and visible light images to enhance image details and texture

Abstract

The present invention relates to the field of digital image processing technology, and provides an infrared and visible light image fusion method for improving image details and textures, the method comprising: splicing feature vectors of an original infrared image and an original visible light image; using a serial multiple residual parallel visual Mamba network model to extract initial feature maps of different spatial scales from the obtained source image features; using the initial feature map obtained by the last iteration as a basic feature map, and using a multi-channel parallel visual Mamba network model to identify fusion feature information of the basic feature map; extending the fusion feature information to the initial feature maps of different spatial scales to obtain initial fusion feature maps on different spatial scales; upsampling the initial fusion feature maps on each spatial scale to the resolution of the original infrared image and generating a fusion image; and iteratively optimizing according to the feature loss of the fusion image and the original image. The present invention can significantly improve the detail clarity and texture expression in the fusion image.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a method for fusing infrared and visible light images for improving image details and textures. Background Art

[0002] As an advanced image processing method, multimodal image fusion technology is committed to integrating image data from different sensing technologies and imaging mechanisms into a single and information-rich image output. The fundamental purpose of this technology is to bring together the advantages of each image source, transcend the limitations of a single image, and provide a higher level of information density and quality. It has a wide range of applications, covering multiple key areas such as medical imaging, remote sensing technology, and computer vision.

[0003] At present, typical multimodal image fusion tasks include infrared and visible light image fusion. The fusion of infrared and visible light images aims to comprehensively utilize the information of two different band images to obtain richer and more comprehensive information than a single image. Infrared images have unique advantages in night vision and penetrating smoke, while visible light images can provide high-resolution detail information. However, in the existing technology, there are still problems such as detail blurring and texture loss in the fusion process of infrared and visible light images, and the fusion effect is not good. Therefore, the field of multimodal image fusion faces a core challenge: how to preserve and enhance the fine texture structure of the source image while deeply fusing multimodal image information, and effectively suppress noise pollution and reduce the generation of unwanted artifacts. This is a key technical problem that needs to be overcome to promote progress in this field. Summary of the invention

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present invention provides an infrared and visible light image fusion method for improving image details and texture.

[0005] In one aspect of the present invention, a method for fusing infrared and visible light images for improving image details and texture is provided, the method comprising:

[0006] The feature vectors extracted from the original infrared image and the original visible light image are directly concatenated to obtain the source image features;

[0007] Using a plurality of preset residual parallel visual Mamba network models arranged in series to perform initial feature extraction on the source image features for multiple iterations, to obtain a plurality of initial feature maps of the source image features at different spatial scales;

[0008] The initial feature map obtained by the last iteration in the initial feature extraction process is used as the basic feature map, and the preset multi-channel parallel visual Mamba network model is used to identify the fusion feature information between the global feature information and the local feature information in the basic feature map;

[0009] Expand the fused feature information to the initial feature maps at different spatial scales to obtain the initial fused feature maps at different spatial scales;

[0010] The initial fused feature map is upsampled at each spatial scale to match the resolution of the original infrared image, and the upsampled feature maps are integrated by weighted summation to generate a fused image.

[0011] The feature losses of the fused image and the original infrared image, and the feature losses of the fused image and the original visible light image are calculated, and the residual parallel visual Mamba network model and / or the multi-channel parallel visual Mamba network model are iteratively optimized according to the feature losses.

[0012] Furthermore, the residual parallel visual Mamba network model includes a convolutional layer, a residual block group and a four-channel parallel visual Mamba module, and the residual block group includes at least 2 residual blocks;

[0013] The implementation process of the residual parallel visual Mamba network model for initial feature extraction includes:

[0014] Inputting the source image features or the initial feature map obtained by the previous initial feature extraction iterative process into the convolution layer to extract the original feature map;

[0015] The original feature map is input into the residual block group to perform feature transformation through convolution, batch normalization and activation function operations inside the residual block group, and the transformed feature map is directly added to the element value of the corresponding position of the original feature, and the added feature map is input into the ReLU activation function;

[0016] The feature data output by the ReLU activation function is input into the four-channel parallel vision Mamba module for normalization. The normalized features are evenly divided into four branches, each of which includes 1 / 4 of the features of the original channel C. The calculation model is as follows:

[0017] ;

[0018] The local feature extraction of each branch is enhanced by the preset Mamba operation, and the enhanced local feature extraction result is adaptively adjusted by the preset adjustment factor. The operation process is as follows:

[0019] ;

[0020] The output features of each feature branch after adaptive adjustment are merged and normalized through the residual connection mechanism, and the normalized merged features are projected to obtain the initial feature map of the current initial feature extraction iterative process. The operation process is as follows:

[0021] ;

[0022] in, Respectively represent the features after 4 channels, represents the initial feature map of the output, The representative layer is normalized. Represents the Split operation, Represents Mamba operation, represents the preset adjustment factor, represents the concatenation operation, represents the projection operation, and R represents the input feature.

[0023] Furthermore, the four-channel parallel visual Mamba module includes: a first normalization layer arranged in sequence, four Mamba modules arranged in parallel and connected to the output channel of the first normalization layer, a residual fast connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with an adaptive adjustment operation layer in a one-to-one correspondence;

[0024] Further, the multi-channel parallel vision Mamba network model includes a four-channel parallel vision Mamba module, an eight-channel parallel vision Mamba module and a sixteen-channel parallel vision Mamba module arranged in series;

[0025] The four-channel parallel visual Mamba module is used to perform feature recognition on the basic feature map to obtain local feature information for highlighting fine features and local information of the original image;

[0026] The eight-channel parallel visual Mamba module is used to perform feature recognition on the local feature information to obtain image structural feature information for highlighting the structural pattern and regional relationship of the original image;

[0027] The sixteen-channel parallel visual Mamba module is used to perform feature recognition on the image structure feature information to obtain fusion feature information between the global feature information and the local feature information.

[0028] Furthermore, the step of extending the fused feature information to initial feature maps of different spatial scales includes:

[0029] The fused feature information is upsampled to the same resolution as the initial feature map of each spatial scale, and each fused feature information obtained by upsampling is expanded to the initial feature map of the corresponding spatial scale.

[0030] Furthermore, the step of expanding each fusion feature information obtained by upsampling to an initial feature map of a corresponding spatial scale includes:

[0031] Each fused feature information obtained by upsampling is multiplied element-by-element with the initial feature map of the corresponding scale.

[0032] Furthermore, the calculating of the feature loss between the fused image and the original infrared image, and between the fused image and the original visible light image, comprises:

[0033] The pre-trained VGG-16 network is used to perform multi-scale feature extraction on the original infrared image and the original visible light image to obtain visible light image features and infrared image features;

[0034] The L1 loss function is used to calculate the L1 loss between the fusion image features and the visible light image features of the fusion image. , and the L1 loss between the fused image features and the infrared image features of the fused image , the L1 loss function is expressed as:

[0035]

[0036] Among them, F vi 、F ir and F Fused Respectively represent visible light image features, infrared image features and fused image features, Represents the L1 norm.

[0037] Furthermore, the method further comprises:

[0038] The total loss function of the construction method is:

[0039]

[0040] in, Represents the loss function of each network model used in the fusion image generation process, , Indicates the strength loss, represents the gradient loss, Indicates that the structural similarity is lost. , and They represent the weights of the three corresponding losses.

[0041] Another aspect of the present invention provides an infrared and visible light image fusion system for improving image details and texture, the system comprising:

[0042] A generator is used to generate a fused image, specifically used to directly splice feature vectors extracted from an original infrared image and an original visible light image to obtain source image features; multiple preset residual parallel visual Mamba network models set in series are used to perform initial feature extraction on the source image features for multiple iterations to obtain multiple initial feature maps of the source image features at different spatial scales; the initial feature map obtained in the last iteration during the initial feature extraction process is used as a basic feature map, and a preset multi-channel parallel visual Mamba network model is used to identify fused feature information between global feature information and local feature information in the basic feature map; the fused feature information is extended to initial feature maps of different spatial scales to obtain initial fused feature maps at different spatial scales; the initial fused feature map is upsampled at each spatial scale to match the resolution of the original infrared image, and the upsampled feature maps are integrated by weighted summation to generate a fused image;

[0043] The discriminator is used to optimize the visual perception quality of the fused image, specifically to calculate the feature loss between the fused image and the original infrared image, and between the fused image and the original visible light image, and iteratively optimize the residual parallel visual Mamba network model and / or the multi-channel parallel visual Mamba network model according to the feature loss.

[0044] Furthermore, the generator includes a plurality of residual parallel visual Mamba network models, a multi-channel parallel visual Mamba network model, an upsampling network and a feature superposition network. The plurality of residual parallel visual Mamba network models are connected in series to the multi-channel parallel visual Mamba network model. The output end of the multi-channel parallel visual Mamba network model is connected to the upsampling network. The upsampling network is connected to the output end of each residual parallel visual Mamba network model through the feature superposition network to expand the fused feature information identified by the multi-channel parallel visual Mamba network model to the initial feature maps of different spatial scales, thereby obtaining the initial fused feature maps at different spatial scales.

[0045] The discriminator includes a four-channel parallel vision Mamba module, an eight-channel parallel vision Mamba module and a sixteen-channel parallel vision Mamba module arranged in series, the four-channel parallel vision Mamba module is used to perform feature recognition on the basic feature map to obtain local feature information for highlighting the fine features and local information of the original image; the eight-channel parallel vision Mamba module is used to perform feature recognition on the local feature information to obtain image structure feature information for highlighting the structural pattern and regional relationship of the original image; the sixteen-channel parallel vision Mamba module is used to perform feature recognition on the image structure feature information to obtain fusion feature information between global feature information and local feature information.

[0046] The infrared and visible light image fusion method for improving image details and texture provided by the present invention extracts multiple initial feature maps of source image features at different spatial scales through a series-arranged residual parallel visual Mamba network model, which can significantly improve the realism of image details and texture in the initial feature map, and then extracts the fusion feature information between the global feature information and the local feature information in the feature map through a multi-channel parallel visual Mamba network model, thereby enhancing the ability to capture the global structure and local details of the image, and finally uses adversarial training to promote continuous optimization of the system, further optimizing the visual performance. The infrared and visible light image fusion method for improving image details and texture provided by the present invention can significantly improve the detail clarity and texture performance in the fused image in the infrared and visible light image fusion task, not only retaining the key features of the original image, but also enhancing the overall visual effect of the fused image. Moreover, the present invention not only obtains excellent subjective visual effects, but also has outstanding performance in objective quantitative indicators, and has significant effects in improving fusion quality and in a wide range of practical applications, making important contributions to the field of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1 A network architecture diagram of an infrared and visible light image fusion system for improving image details and texture provided by an embodiment of the present invention;

[0049] Figure 2 A flowchart of the infrared and visible light image fusion method for enhancing image details and texture provided by an embodiment of the present invention;

[0050] Figure 3 A network structure diagram of a residual parallel visual Mamba network model provided by an embodiment of the present invention;

[0051] Figure 4 A network structure diagram of the Mamba module provided in an embodiment of the present invention;

[0052] Figure 5 A network structure diagram of a multi-channel parallel visual Mamba network model provided by an embodiment of the present invention;

[0053] Figure 6 It is a schematic diagram comparing the fusion results of the existing image fusion method and the infrared and visible light image fusion method for enhancing image details and texture provided by the present invention on the LLVIP dataset. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In order to solve the problems of detail blur and texture loss in the infrared and visible light image fusion process in the prior art, the present invention proposes an infrared and visible light image fusion method for improving image details and texture. The method can significantly improve the detail clarity and texture expression in the fused image. By adopting advanced image processing technology and optimized fusion strategy, not only the key features of the original image are retained, but also the overall visual effect of the fused image is enhanced.

[0056] Figure 1 The network architecture diagram of the infrared and visible light image fusion system for improving image details and texture of the present invention is shown. Figure 1 In the embodiment shown, four residual parallel visual Mamba network models are set serially as an example for explanation. Figure 1 As shown, the system comprises:

[0057] Generator, used to generate a fused image, specifically used to convert the original infrared image I ir and the original visible light image I vi The feature vectors extracted from the source image are directly spliced ​​to obtain the source image features; multiple preset residual parallel visual Mamba network models set in series are used to perform initial feature extraction on the source image features after feature splicing for multiple iterations to obtain multiple initial feature maps of the source image features at different spatial scales; the initial feature map obtained by the last iteration in the initial feature extraction process is used as the basic feature map, and the preset multi-channel parallel visual Mamba network model is used to identify the fusion feature information between the global feature information and the local feature information in the basic feature map; the fusion feature information is extended to the initial feature maps of different spatial scales to obtain the initial fusion feature maps at different spatial scales; the initial fusion feature map is upsampled at each spatial scale to match the resolution of the original infrared image, and the upsampled feature maps are integrated by weighted summation to generate a fusion image I F ;

[0058] The discriminator is used to optimize the visual perception quality of the fused image, specifically to calculate the feature loss between the fused image and the original infrared image, and between the fused image and the original visible light image, and iteratively optimize the residual parallel visual Mamba network model and / or the multi-channel parallel visual Mamba network model according to the feature loss.

[0059] In a specific embodiment, the generator includes a plurality of residual parallel visual Mamba network models, a multi-channel parallel visual Mamba network model, an upsampling network, and a feature superposition network. After the plurality of residual parallel visual Mamba network models are connected in series, the multi-channel parallel visual Mamba network model output end is connected to the upsampling network, and the upsampling network is connected to the output end of each residual parallel visual Mamba network model through the feature superposition network, so as to extend the fusion feature information identified by the multi-channel parallel visual Mamba network model to the initial feature map of different spatial scales, and obtain the initial fusion feature map at different spatial scales. Specifically, the present application connects 4 residual parallel visual Mamba network models in series and then connects the multi-channel parallel visual Mamba network model. First, the original infrared image and the original visible light image are spliced ​​and then the initial feature extraction is performed by each residual parallel visual Mamba network model in turn, and multiple initial feature maps at different spatial scales are obtained. Then, the output features of the last layer of residual parallel visual Mamba network model are passed to the multi-channel parallel visual Mamba network model to deeply explore the association between global and local features and obtain fusion feature information. In this module, after capturing the key cross-source interaction information, namely the fused feature information, the upsampling network also expands these fused feature information to different spatial scales through the upsampling step. At each scale, the feature stacking network performs element-by-element multiplication of the fused feature information with the initial feature map of the corresponding scale to produce a preliminary fused representation. These feature maps are then upsampled to the original image resolution and aggregated through the stacking strategy to generate the final fused image.

[0060] In a specific embodiment, the discriminator includes a four-channel parallel visual Mamba module, an eight-channel parallel visual Mamba module, and a sixteen-channel parallel visual Mamba module arranged in series, the four-channel parallel visual Mamba module is used to perform feature recognition on the basic feature map to obtain local feature information for highlighting the fine features and local information of the original image; the eight-channel parallel visual Mamba module is used to perform feature recognition on the local feature information to obtain image structural feature information for highlighting the structural pattern and regional relationship of the original image; the sixteen-channel parallel visual Mamba module is used to perform feature recognition on the image structural feature information to obtain fusion feature information between global feature information and local feature information. Specifically, by using a pre-trained VGG-16 network as a basis, and fine-tuning during the training process. Specifically, the original infrared image and the original visible light image are sent to the VGG-16 network for feature extraction. By calculating the L1 loss between the features of the fused image and the source image (the original infrared image and the original visible light image), it is ensured that the fused image is close to the source image at the content level. During training, the source image generates a preliminary fused image through the generator, which is then evaluated by the discriminator, and the quality of the fused image is quantified by a carefully designed loss function. This generation and evaluation cycle builds a dynamic adversarial framework between the generator and the discriminator, driving continuous optimization of the system. After iterative training, a generator with excellent performance is obtained that can efficiently generate high-quality fused images.

[0061] Furthermore, the residual parallel visual Mamba network model includes a convolutional layer, a residual block group and a four-channel parallel visual Mamba module. Specifically, the residual block group includes at least 2 residual blocks, and the four-channel parallel visual Mamba module includes a first normalization layer arranged in sequence, four Mamba modules arranged in parallel connected to the output channel of the first normalization layer, an adaptive adjustment operation layer corresponding to each Mamba module, a residual fast connection layer, a second normalization layer and a projection layer.

[0062] Among them, the Mamba module, the improved S4, is an advanced sequence modeling method that can maintain high efficiency when processing complex data. It dynamically adapts to different scenarios by combining input-related state space adjustments and linear time complexity design, effectively capturing long-term dependencies in sequences. Mamba shows higher efficiency and competitiveness when processing long sequences, providing a new, high-performance solution for sequence data analysis.

[0063] The present invention proposes a multimodal image fusion method combining a Mamba model with a GAN network. GAN has a significant advantage in high-quality image fusion due to its unique adversarial training mechanism. Through the competition between the generator and the discriminator, GAN can learn more complex and realistic image features, and because it does not require a predefined loss function, it has outstanding effects in unsupervised learning and image fusion.

[0064] Figure 2 The infrared and visible light image fusion method for improving image details and texture according to an embodiment of the present invention is shown in the flowchart of an implementation of the infrared and visible light image fusion method for improving image details and texture according to an embodiment of the present invention. Figure 1 The infrared and visible light image fusion system shown in Figure 2 As shown, the infrared and visible light image fusion method for improving image details and textures proposed in the embodiment of the present invention specifically includes the following steps:

[0065] S11. Directly concatenate the feature vectors extracted from the original infrared image and the original visible light image to obtain source image features.

[0066] In the embodiment of the present invention, a collection device capable of simultaneously capturing visible light images and infrared images may be used to simultaneously acquire visible light images and infrared images, or different collection devices may be used to respectively acquire visible light images and infrared images. The embodiment of the present invention does not specifically limit this.

[0067] S12. Using multiple preset residual parallel visual Mamba network models set in series, perform initial feature extraction on the source image features after feature merging for multiple iterations to obtain multiple initial feature maps of the source image features at different spatial scales.

[0068] S13, taking the initial feature map obtained by the last iteration in the initial feature extraction process as the basic feature map, and using a preset multi-channel parallel visual Mamba network model to identify the fusion feature information between the global feature information and the local feature information in the basic feature map.

[0069] S14, expanding the fused feature information to initial feature maps of different spatial scales to obtain initial fused feature maps at different spatial scales.

[0070] S15. Upsample the initial fused feature map at each spatial scale to match the resolution of the original infrared image, and integrate the upsampled feature map by weighted summation to generate a fused image.

[0071] S16, calculating feature losses between the fused image and the original infrared image, and between the fused image and the original visible light image, and iteratively optimizing the residual parallel visual Mamba network model and / or the multi-channel parallel visual Mamba network model according to the feature losses.

[0072] The infrared and visible light image fusion method for improving image details and texture provided by the present invention can significantly improve the realism of image details and texture in the initial feature map by extracting multiple initial feature maps of source image features at different spatial scales through a residual parallel visual Mamba network model set in series, and then extract the fusion feature information between the global feature information and the local feature information in the feature map through a multi-channel parallel visual Mamba network model, thereby enhancing the ability to capture the global structure and local details of the image, and finally using adversarial training to promote continuous optimization of the system, further optimizing the visual performance. The infrared and visible light image fusion method for improving image details and texture provided by the present invention can significantly improve the detail clarity and texture performance in the fused image in the infrared and visible light image fusion task, not only retaining the key features of the original image, but also enhancing the overall visual effect of the fused image.

[0073] In the embodiment of the present invention, Figure 3 As shown, the residual parallel visual Mamba network model includes a convolutional layer, a residual block group and a four-channel parallel visual Mamba module. Specifically, the residual block group includes at least 2 residual blocks, and the four-channel parallel visual Mamba module includes a first normalization layer arranged in sequence, four Mamba modules arranged in parallel connected to the output channel of the first normalization layer, an adaptive adjustment operation layer corresponding to each Mamba module, a residual fast connection layer, a second normalization layer and a projection layer.

[0074] See also Figure 3, the residual parallel visual Mamba network model provided by the embodiment of the present invention is composed of a convolution layer, a group of residual blocks and a four-channel parallel visual Mamba module. The source image features after the source image is merged are first converted by a 1×1 convolution layer, and then added with the features processed by the two residual blocks. The processed features are then input into the visual Mamba module set in parallel. The present invention realizes the initial feature map recognition and extraction through the residual parallel visual Mamba network model, effectively maintains the integrity of the multimodal image features, and introduces deep feature expression through the residual learning mechanism, thereby realizing richer feature representation and information capture in high-dimensional space, and using parallel visual Mamba as the core component, and then using its unique parallel structure to deeply integrate and extract image features. After completing the feature extraction of the residual block, the features obtained by the addition operation are then standardized by the normalization layer to ensure that the channels have the same scale and distribution. Subsequently, the features are divided into four branches, each of which specializes in processing 1 / 4 of the original channel number C, so as to deepen the focused analysis of different feature subsets.

[0075] These independent feature branches are then imported into the Mamba module. The specific network structure of the Mamba module is as follows: Figure 4 As shown. The Mamba module includes a first linear layer, a second linear layer, a first activation function operation module connected to the output channel of the first linear layer, a convolution block, a second activation function operation module and a state space equation module connected to the output channel of the second linear layer in sequence, an element multiplication operation module connected to the output end of the first activation function operation module and the output end of the state space equation module, and a third linear layer. The present invention uses the Mamba module for fine tuning to mine deep patterns, including using pre-trained weight initialization, implementing data enhancement technology, adjusting the loss function to adapt to specific task requirements, adopting strategic learning rate scheduling, applying batch normalization and gradient clipping. The output of the Mamba model is combined with a preset adjustment factor through a residual connection mechanism, which not only strengthens the grasp of local features, but also significantly enhances the model's ability to capture long-distance spatial dependencies, which is crucial for understanding the global context. Specifically, the design principle of the adjustment factor is to enhance the adaptability of the model, achieve the combination of local and global information, and avoid overfitting by balancing the contributions of different feature branches.

[0076] Furthermore, the implementation process of the residual parallel visual Mamba network model for initial feature extraction includes the following steps not shown in the accompanying drawings:

[0077] S21, inputting the source image features or the initial feature map obtained in the previous initial feature extraction iterative process into the convolution layer to extract the original feature map;

[0078] S22, inputting the original feature map into the residual block group, performing feature transformation through convolution, batch normalization and activation function operations within the residual block group, and directly adding the transformed feature map to the element value of the corresponding position of the original feature, and inputting the added feature map into the ReLU activation function;

[0079] S23, input the feature data output by the ReLU activation function into the four-channel parallel vision Mamba module for normalization processing, and evenly divide the normalized features into four branches, each branch includes 1 / 4 features of the original channel C, and the calculation model is as follows:

[0080] ;

[0081] S24, respectively performing enhanced local feature extraction on each branch through a preset Mamba operation, and adaptively adjusting the enhanced local feature extraction result through a preset adjustment factor, the operation process is as follows:

[0082]

[0083] S25, the output features of each feature branch after adaptive adjustment are merged and normalized through the residual connection mechanism, and the normalized merged features are projected to obtain the initial feature map of the current initial feature extraction iterative process. The operation process is as follows:

[0084] ;

[0085] in, Respectively represent the features after 4 channels, represents the initial feature map of the output, The representative layer is normalized. Represents the Split operation, Represents Mamba operation, represents the preset adjustment factor, represents the concatenation operation, represents the projection operation, and R represents the input feature.

[0086] In the embodiment of the present invention, Figure 5As shown, the multi-channel parallel vision Mamba network model includes a four-channel parallel vision Mamba module, an eight-channel parallel vision Mamba module and a sixteen-channel parallel vision Mamba module arranged in series. Among them, the four-channel parallel vision Mamba module is used to perform feature recognition on the basic feature map to obtain local feature information for highlighting the fine features and local information of the original image; the eight-channel parallel vision Mamba module is used to perform feature recognition on the local feature information to obtain image structure feature information for highlighting the structural pattern and regional relationship of the original image; the sixteen-channel parallel vision Mamba module is used to perform feature recognition on the image structure feature information to obtain fusion feature information between global feature information and local feature information.

[0087] See also Figure 5 , a multi-channel parallel vision Mamba network model provided by a specific embodiment of the present invention is composed of a 4-channel parallel vision Mamba module, an 8-channel parallel vision Mamba module and a 16-channel parallel vision Mamba module in series. Specifically, the 4-channel parallel vision Mamba module focuses on capturing fine features and local information in the image. The 8-channel parallel vision Mamba module can effectively identify and parse the structural patterns and regional relationships in the image, thereby playing the role of a bridge connecting local information and global information. The 16-channel parallel vision Mamba focuses on information processing at the global level to ensure accurate grasp of the global information of the image. The series connection and parallel work of these three sub-modules not only greatly enrich the level and diversity of image features, but also significantly enhance the model's ability to understand complex image scenes.

[0088] In the architecture design of the multi-channel parallel vision Mamba module, each module shares a similar basic structure. The core difference between them lies in the different configurations of the number of channels, namely 4, 8 and 16 channels, in order to meet the needs of different levels of information extraction. Among them, the specific operation process of the 8-channel and 16-channel modules is as follows:

[0089] ;

[0090] ;

[0091] ;

[0092] Where N represents the number of channels, which can be 8 or 16.

[0093] In an embodiment of the present invention, the specific implementation method of expanding the fused feature information to the initial feature map of different spatial scales in step S14 is as follows: up-sampling the fused feature information to the same resolution as the initial feature map of each spatial scale, and expanding each fused feature information obtained by up-sampling to the initial feature map of the corresponding spatial scale. Further, the expansion of each fused feature information obtained by up-sampling to the initial feature map of the corresponding spatial scale includes: performing element-by-element multiplication operation on each fused feature information obtained by up-sampling and the initial feature map of the corresponding scale. In this embodiment, after obtaining the fused feature information through the multi-channel parallel visual Mamba module, an up-sampling step is applied to the fused feature map, i.e., the fused feature information, and the fused feature information is expanded to different spatial scales using bilinear interpolation technology. This process not only helps to restore the spatial details that may be lost in the early processing stage, but also enhances the model's understanding of multi-scale features, so that objects and details of different sizes in the image can be more effectively processed.

[0094] In this technical solution, bilinear interpolation technology is used to upsample the fused feature information obtained through the multi-channel parallel visual Mamba module to the same resolution as the initial feature map of each spatial scale, and then perform element-by-element multiplication operation on the upsampled fused feature information and the initial feature map of the corresponding scale, so as to enhance the understanding and processing capabilities of multi-scale features and restore the spatial details that may be lost in the early processing stage.

[0095] In the embodiment of the present invention, the calculation of the feature loss between the fused image and the original infrared image, and between the fused image and the original visible light image in step S16 specifically includes:

[0096] The pre-trained VGG-16 network is used to perform multi-scale feature extraction on the original infrared image and the original visible light image to obtain visible light image features and infrared image features;

[0097] The L1 loss function is used to calculate the L1 loss between the fusion image features and the visible light image features of the fusion image. , and the L1 loss between the fused image features and the infrared image features of the fused image , the L1 loss function is expressed as:

[0098]

[0099] Among them, F vi 、F ir and F Fused Respectively represent visible light image features, infrared image features and fused image features, Represents the L1 norm.

[0100] In this embodiment, for the multimodal image fusion task, the discriminator includes a bimodal image discriminator, which uses a pre-trained VGG-16 network to extract multi-scale features from the original infrared image and the original visible light image, and uses deep features to emphasize feature significance. The method framework uses a feature-level L1 loss function to promote feature matching between the fused image and the original image.

[0101] Furthermore, the total loss function of the generator and the discriminator in the infrared and visible light image fusion method for enhancing image details and texture provided by the present invention is:

[0102]

[0103] in, Represents the loss function of the generator, that is, the loss function of each network model used in the fusion image generation process, which is expressed as follows:

[0104] ,

[0105] in, Indicates the strength loss, represents the gradient loss, Indicates that the structural similarity is lost. , and They represent the weights of the three corresponding losses.

[0106] A specific example is given below to verify and illustrate the beneficial effects of the infrared and visible light image fusion method for enhancing image details and texture proposed in the present invention.

[0107] In this embodiment, the MSRS dataset containing 1083 images is used to train the system framework. The pixel size of these images is set to 256×256 as training samples. During the training process, 50 epochs are set, the Adam optimizer is used, the batch size is set to 16, and the learning rate is set to 10-4. For the key weight parameters involved in the loss function formula of the generator, , and The values ​​are set to 10, 100 and 1, which are obtained based on preliminary parameter tuning experiments. In order to fully verify the accuracy and performance of the system framework of the present invention, model inference and performance evaluation are implemented on the LLVIP dataset. The model is implemented on an NVIDIA™ GPU (RTX 4090) using PyTorch as the programming environment.

[0108] Table 1 systematically summarizes and shows the quantitative evaluation conclusions of the ten compared algorithms and the proposed method on six general evaluation indicators. The evaluation criteria adopt information entropy (EN), spatial frequency (SF), average gradient (AG), sum of correlation differences (SCD), correlation coefficient (CC) and multi-scale structural similarity index measure (MS-SSIM), where the higher the value of all indicators, the better the performance. For each evaluation indicator, the average score is calculated separately using 50 test instances in the LLVIP dataset, where the best result is highlighted in bold, the second best result is underlined, and the third best performance is italicized to enhance the visual contrast effect. The analysis results show that the infrared and visible light image fusion method for improving image details and texture proposed in the present invention has reached the highest level in terms of the five evaluation indicators.

[0109] Table 1. Average index values ​​obtained by existing fusion methods and the present invention on the LLVIP dataset

[0110] EN↑ SF↑ AG↑ CC↑ SCD↑ MS_SSIM↑ CDDFuse <![CDATA[ 7.184 ]]> 13.282 3.536 0.681 1.552 1.019 <![CDATA[CrossFuse Li ]]> 6.324 9.316 2.474 <![CDATA[ 0.699 ]]> 0.882 0.908 <![CDATA[CrossFuse Wang ]]> 6.367 13.833 3.645 0.653 1.177 0.900 DATFuse 6.987 10.152 2.705 0.684 1.304 0.979 DSFusion 6.517 14.380 4.172 0.652 1.126 0.947 LRRNet 6.348 10.565 2.865 0.668 0.983 0.800 SemLA 6.849 9.631 2.730 0.690 1.345 0.890 SwinFusion 7.180 <![CDATA[ 15.264 ]]> <![CDATA[ 4.374 ]]> 0.672 <![CDATA[ 1.585 ]]> <![CDATA[ 1.006 ]]> YDTR 6.527 8.260 2.099 0.693 1.064 0.929 Ours 7.210 17.174 5.248 0.712 1.807 0.973

[0111] Figure 6 The figure shows the comparison of the fusion results of the existing image fusion method and the infrared and visible light image fusion method for enhancing image details and texture provided by the present invention on the LLVIP dataset. Figure 6 , showing the fusion effects of ten models on the LLVIP dataset, among which CDDFuse and DATFuse are insufficient in edge details; CrossFuseLi has weak infrared texture extraction; CrossFuseWang has low brightness; DSFusion and LRRNet also lack brightness. SeMLA and SwinFusion partially solve the brightness problem, but the clarity of some scenes is still insufficient. The YDTR image is blurred. In contrast, the framework of the present invention performs well in low light, such as clear building textures and leaf shadow details. It not only retains the texture of visible images, but also clearly expresses the thermal radiation of infrared images.

[0112] The infrared and visible light image fusion method for improving image details and texture provided by the present invention extracts multiple initial feature maps of source image features at different spatial scales through a series-arranged residual parallel visual Mamba network model, which can significantly improve the realism of image details and texture in the initial feature map, and then extracts the fusion feature information between the global feature information and the local feature information in the feature map through a multi-channel parallel visual Mamba network model, thereby enhancing the ability to capture the global structure and local details of the image, and finally uses a bimodal image discriminator for adversarial training to further optimize the visual performance. Experimental results show that the infrared and visible light image fusion method for improving image details and texture provided by the present invention can significantly improve the detail clarity and texture performance in the fused image in the infrared and visible light image fusion task, not only retaining the key features of the original image, but also enhancing the overall visual effect of the fused image. Moreover, the present invention not only obtains excellent subjective visual effects, but also has outstanding performance in objective quantitative indicators, and has significant effects in improving fusion quality and in a wide range of practical applications, making important contributions to the field of image processing.

[0113] Those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusion of infrared and visible light images to enhance image details and texture, characterized in that: The method comprises: The feature vectors extracted from the original infrared image and the original visible light image are directly concatenated to obtain the source image features; Using a plurality of preset residual parallel visual Mamba network models arranged in series to perform initial feature extraction on the source image features for multiple iterations, to obtain a plurality of initial feature maps of the source image features at different spatial scales; The initial feature map obtained by the last iteration in the initial feature extraction process is used as the basic feature map, and the preset multi-channel parallel visual Mamba network model is used to identify the fusion feature information between the global feature information and the local feature information in the basic feature map; Expand the fused feature information to the initial feature maps at different spatial scales to obtain the initial fused feature maps at different spatial scales; The initial fused feature map is upsampled at each spatial scale to match the resolution of the original infrared image, and the upsampled feature maps are integrated by weighted summation to generate a fused image. Calculating feature losses between the fused image and the original infrared image, and between the fused image and the original visible light image, and iteratively optimizing the residual parallel visual Mamba network model and / or the multi-channel parallel visual Mamba network model according to the feature losses; The residual parallel visual Mamba network model includes a convolutional layer, a residual block group and a four-channel parallel visual Mamba module, and the residual block group includes at least 2 residual blocks; The implementation process of the residual parallel visual Mamba network model for initial feature extraction includes: Inputting the source image features or the initial feature map obtained by the previous initial feature extraction iterative process into the convolution layer to extract the original feature map; The original feature map is input into the residual block group to perform feature transformation through convolution, batch normalization and activation function operations inside the residual block group, and the transformed feature map is directly added to the element value of the corresponding position of the original feature map, and the added feature map is input into the ReLU activation function; The feature data output by the ReLU activation function is input into the four-channel parallel vision Mamba module for normalization. The normalized features are evenly divided into four branches, each of which includes 1 / 4 of the features of the original channel C. The calculation model is as follows: ； The local feature extraction of each branch is enhanced by the preset Mamba operation, and the enhanced local feature extraction result is adaptively adjusted by the preset adjustment factor. The operation process is as follows: ； The output features of each feature branch after adaptive adjustment are merged and normalized through the residual connection mechanism, and the normalized merged features are projected to obtain the initial feature map of the current initial feature extraction iterative process. The operation process is as follows: ； in, Respectively represent the features after 4 channels, represents the initial feature map of the output, The representative layer is normalized. Represents the Split operation, Represents Mamba operation, represents the preset adjustment factor, represents the concatenation operation, represents the projection operation, and R represents the input feature; The multi-channel parallel vision Mamba network model includes a four-channel parallel vision Mamba module, an eight-channel parallel vision Mamba module and a sixteen-channel parallel vision Mamba module arranged in series; The four-channel parallel visual Mamba module is used to perform feature recognition on the basic feature map to obtain local feature information for highlighting fine features and local information of the original image; The eight-channel parallel visual Mamba module is used to perform feature recognition on the local feature information to obtain image structural feature information for highlighting the structural pattern and regional relationship of the original image; The sixteen-channel parallel visual Mamba module is used to perform feature recognition on the image structure feature information to obtain fusion feature information between the global feature information and the local feature information.

2. The method according to claim 1, characterized in that The four-channel parallel visual Mamba module includes: a first normalization layer arranged in sequence, four Mamba modules arranged in parallel and connected to the output channel of the first normalization layer, a residual block connection layer, a second normalization layer and a projection layer, and each Mamba module is provided with an adaptive adjustment operation layer in a one-to-one correspondence.

3. The method according to claim 1, characterized in that The step of extending the fused feature information to initial feature maps of different spatial scales includes: The fused feature information is upsampled to the same resolution as the initial feature map of each spatial scale, and each fused feature information obtained by upsampling is expanded to the initial feature map of the corresponding spatial scale.

4. The method according to claim 3, characterized in that The step of expanding each fusion feature information obtained by upsampling to an initial feature map of a corresponding spatial scale includes: Each fused feature information obtained by upsampling is multiplied element-by-element with the initial feature map of the corresponding scale.

5. The method according to claim 1, characterized in that The calculating the feature loss between the fused image and the original infrared image, and between the fused image and the original visible light image, comprises: The pre-trained VGG-16 network is used to perform multi-scale feature extraction on the original infrared image and the original visible light image to obtain visible light image features and infrared image features; The L1 loss function is used to calculate the L1 loss between the fusion image features and the visible light image features of the fusion image. , and the L1 loss between the fused image features and the infrared image features of the fused image , the L1 loss function is expressed as: Among them, F vi 、F ir and F Fused Respectively represent visible light image features, infrared image features and fused image features, Represents the L1 norm.

6. The method according to claim 5, characterized in that The method further comprises: The total loss function of the construction method is: in, Represents the loss function of each network model used in the fusion image generation process, , Indicates the strength loss, represents the gradient loss, Indicates that the structural similarity is lost. , and They represent the weights of the three corresponding losses.

7. An infrared and visible light image fusion system for enhancing image details and texture, characterized in that: The system comprises: A generator is used to generate a fused image, specifically used to directly splice feature vectors extracted from an original infrared image and an original visible light image to obtain source image features; multiple preset residual parallel visual Mamba network models set in series are used to perform initial feature extraction on the source image features for multiple iterations to obtain multiple initial feature maps of the source image features at different spatial scales; the initial feature map obtained in the last iteration during the initial feature extraction process is used as a basic feature map, and a preset multi-channel parallel visual Mamba network model is used to identify fused feature information between global feature information and local feature information in the basic feature map; the fused feature information is extended to initial feature maps of different spatial scales to obtain initial fused feature maps at different spatial scales; the initial fused feature map is upsampled at each spatial scale to match the resolution of the original infrared image, and the upsampled feature maps are integrated by weighted summation to generate a fused image; A discriminator, used for optimizing the visual perception quality of the fused image, specifically for calculating the feature loss between the fused image and the original infrared image, and between the fused image and the original visible light image, and iteratively optimizing the residual parallel visual Mamba network model and / or the multi-channel parallel visual Mamba network model according to the feature loss; The residual parallel visual Mamba network model includes a convolutional layer, a residual block group and a four-channel parallel visual Mamba module, and the residual block group includes at least 2 residual blocks; The implementation process of the residual parallel visual Mamba network model for initial feature extraction includes: Inputting the source image features or the initial feature map obtained by the previous initial feature extraction iterative process into the convolution layer to extract the original feature map; The original feature map is input into the residual block group to perform feature transformation through convolution, batch normalization and activation function operations inside the residual block group, and the transformed feature map is directly added to the element value of the corresponding position of the original feature map, and the added feature map is input into the ReLU activation function; The feature data output by the ReLU activation function is input into the four-channel parallel vision Mamba module for normalization. The normalized features are evenly divided into four branches, each of which includes 1 / 4 of the features of the original channel C. The calculation model is as follows: ； The local feature extraction of each branch is enhanced by the preset Mamba operation, and the enhanced local feature extraction result is adaptively adjusted by the preset adjustment factor. The operation process is as follows: ； The output features of each feature branch after adaptive adjustment are merged and normalized through the residual connection mechanism, and the normalized merged features are projected to obtain the initial feature map of the current initial feature extraction iterative process. The operation process is as follows: ； in, Respectively represent the features after 4 channels, represents the initial feature map of the output, The representative layer is normalized. Represents the Split operation, Represents Mamba operation, represents the preset adjustment factor, represents the concatenation operation, represents the projection operation, and R represents the input feature; The multi-channel parallel vision Mamba network model includes a four-channel parallel vision Mamba module, an eight-channel parallel vision Mamba module and a sixteen-channel parallel vision Mamba module arranged in series; The four-channel parallel visual Mamba module is used to perform feature recognition on the basic feature map to obtain local feature information for highlighting fine features and local information of the original image; The eight-channel parallel visual Mamba module is used to perform feature recognition on the local feature information to obtain image structural feature information for highlighting the structural pattern and regional relationship of the original image; The sixteen-channel parallel visual Mamba module is used to perform feature recognition on the image structure feature information to obtain fusion feature information between the global feature information and the local feature information.

Images