Infrared and visible light image fusion method based on feature enhancement

Abstract

The invention discloses a feature enhancement-based infrared and visible light image fusion method. The method comprises the following steps of: preprocessing data; performing shallow feature extraction on the two types of modal images by using a double-branch encoder, and performing decomposition to obtain low-frequency and high-frequency features; constraining the low-frequency and high-frequency features; the low-frequency features are superposed and recombined, and cross-modal difference mining is carried out after the high-frequency features are superposed; refining the fusion features, performing Hadamard product operation on the fusion features and corresponding modal fusion features to obtain self-refining features, and performing cross-modal feature combination reconstruction to generate complementary refining features; and reconstructing to obtain a fused image. According to the embodiment of the invention, preliminary integration of cross-modal information is realized through feature superposition and difference mining, and core features of each modal are reserved; through refining and complementary enhancement processing, collaborative expression of cross-modal structure features and detail information can be effectively enhanced, it is ensured that a fused image has stable performance in the aspects of texture fidelity and saliency target presentation, network conciseness is kept, and the quality of a fusion result is high.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for fusing infrared and visible light images based on feature enhancement. Background Technology

[0002] Infrared and visible light images have complementary characteristics. Infrared imaging can highlight thermal targets hidden in the background, effectively highlighting salient targets even at night, in extreme weather, and under partial obstruction, unaffected by lighting conditions or weather. However, infrared imaging has poor detail, blurred edges, and unclear object outlines, failing to provide sufficient environmental information. Visible light images contain rich texture and structural information, target morphology, and other visual details, better matching human visual perception. However, visible light images are susceptible to extreme environments and obstruction, resulting in target loss, and their imaging performance is poor in inclement weather.

[0003] Traditional image fusion algorithms typically measure pixel activity levels in the spatial domain or transform domain (by transforming the image to the transform domain using some mathematical transformation) and manually design fusion rules to achieve image fusion. While traditional algorithms can produce satisfactory results in most cases, several challenges hinder their further development. This is because these methods often use the same transformations or representations to extract features from the source images, failing to account for the inherent differences between different source images. Secondly, manually designed activity level measurements and fusion rules are difficult to adapt to complex fusion scenarios, and the design of activity level measurements and fusion rules becomes increasingly complex in pursuit of better fusion performance.

[0004] To address this issue, researchers have proposed deep learning-based image fusion algorithms. These algorithms utilize neural networks to automatically learn multimodal features through an end-to-end learning approach and are trained on large datasets to adapt to various complex scenarios. Existing deep learning-based image fusion algorithms primarily focus on solving three key problems in image fusion: feature extraction, feature fusion, and image reconstruction. Based on the network architecture employed, deep learning-based image fusion algorithms can be categorized into three types: image fusion frameworks based on autoencoders (AEs), image fusion frameworks based on convolutional neural networks (CNNs), and image fusion frameworks based on generative adversarial networks (GANs).

[0005] Autoencoder-based methods first train an autoencoder on a large-scale natural image dataset. Then, pre-trained encoding and decoding networks are used for feature extraction and image reconstruction, respectively. Finally, a hand-designed fusion strategy is typically used to fuse the deep features extracted by the encoding network to achieve image fusion. End-to-end image fusion frameworks based on convolutional neural networks (CNNs) offer another approach that avoids the drawbacks of manually designing fusion rules. These methods typically rely on the designed network structure and loss function to implicitly achieve feature extraction, feature fusion, and image reconstruction. Generative adversarial networks (GANs) can effectively model data distributions even without supervised information, a characteristic well-suited to the needs of infrared and visible light image fusion. In GAN-based models, GANs are often used to simultaneously make the fused image distributionally similar to the input and perceptually satisfactory.

[0006] Considering the combination of image fusion and downstream pattern recognition tasks, Liu et al. pioneered the exploration of combining image fusion and detection, using the gradient of the loss function for segmentation and detection to guide the generation of fused images. Liang et al. proposed a self-supervised learning framework to complete the fusion task without paired images. Furthermore, adding a preprocessing registration module before the fusion module effectively solves the registration problem of the source images. Jiang et al. first developed a stitching method based on multi-view and multimodal fusion for comprehensive scene perception. The Transformer was first proposed by Vaswani et al. for the field of Natural Language Processing (NLP). Later, many Transformer-based models were widely used in computer vision, achieving satisfactory results in image classification, object detection, image segmentation, and multimodal learning. For low-level vision tasks, compared with convolutional neural network-based methods, the combination of Transformer with multi-task learning and the Swing Transformer module has achieved state-of-the-art results. Considering the significant computational overhead of spatial self-attention mechanisms, Wu et al. proposed a lightweight Transformer (LT) architecture for mobile NLP tasks. This module, through long-short-range attention and a flat feedforward network, significantly reduces the number of parameters while maintaining model performance. Zhao et al. proposed a correlation-driven feature decomposition and fusion algorithm based on a dual-branch LT and CNN, simultaneously processing high-frequency and low-frequency features. This addresses the challenges of cross-modal feature modeling and decomposing ideal modality-specific and modality-shared features, achieving promising results.

[0007] Liu et al. proposed a lightweight unified image fusion network for computationally constrained scenarios such as embedded systems and mobile devices. They designed a pixel-level unified framework that, while compressing model size, still competes with large models in both subjective and objective evaluations. However, its convolutional structure has limited ability to model global information and exhibits weak stability in complex scenarios. Li et al. proposed a representation learning-guided image fusion framework. Addressing the issues of network design relying on experience and inconsistencies between feature learning and fusion objectives, they constructed a guidance mechanism centered on representation learning. However, the representation learning module increases training complexity and exhibits limited robustness in cross-domain or noisy environments. Zhang et al. proposed an end-to-end synchronous multimodal fusion and detection framework. Addressing the feature fragmentation and suboptimal multi-stage training problems in traditional processes, they designed a unified network with shared parameters, enabling simultaneous multimodal fusion and object detection. However, this method is sensitive to training stability and hyperparameters, and its performance is poor under conditions of large differences in modal quality or extreme noise.

[0008] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0009] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0010] The purpose of this invention is to provide a feature-enhanced infrared and visible light image fusion method, thereby overcoming, at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0011] This invention first provides a method for fusing infrared and visible light images based on feature enhancement, comprising: S1, preprocess the original infrared and visible light images; S2 utilizes a dual-branch encoder to extract shallow features from infrared and visible light modal images, and decomposes them to obtain low-frequency global features and high-frequency detail features; S3 uses a loss function to constrain low-frequency global features and high-frequency detail features, and optimizes the network structure training. S4: Low-frequency global features are superimposed and then recombined, and high-frequency detail features are superimposed and then cross-modal difference mining is performed to obtain the fused features of the two modalities. S5. The fused features are refined and then Hadamard product operation is performed with the corresponding modal fused features to obtain self-refined features. Then, the self-refined features are combined with another modal fused features to reconstruct complementary refined features across modalities. S6 uses a decoder to perform feature splicing, dimensionality reduction transformation, and residual connection processing on complementary refined features to reconstruct the fused image.

[0012] In this invention, S1 includes the following steps: S101: Infrared images are acquired by an infrared imaging device, and visible light images are acquired by a regular camera or mobile device. Image registration is performed on the two types of images to form the original dataset. S102, convert the images in the original dataset to grayscale images; S103, a sliding window method is used to divide the grayscale image into image blocks of fixed size; S104, perform contrast evaluation on the generated image blocks and filter out low-contrast image blocks; S105 stores the valid image blocks uniformly as HDF5 format files and divides the dataset into multiple batches.

[0013] In this invention, S2 includes the following steps: S201 performs shallow feature extraction on the input infrared and visible light images, while capturing local details and global structural information. S202 decomposes shallow features into two branches: low-frequency global features and high-frequency detail features.

[0014] In this invention, step S3 includes the following steps: S301, calculate the correlation between low-frequency features and high-frequency features of infrared and visible light images respectively; S302, construct a correlation loss function based on the correlation between low-frequency features and high-frequency features; S303, construct a fusion loss function based on the correlation loss function, gradient loss function, and pixel feature loss function; S304 optimizes the network weights through backpropagation algorithm, thereby training and optimizing the network structure.

[0015] In this invention, S4 includes the following steps: S401, superimpose the low-frequency global features of the infrared image and the visible light image to obtain the preliminary fused low-frequency global features, and superimpose the high-frequency detail features of the infrared image and the visible light image to obtain the preliminary fused high-frequency detail features. S402 uses a multi-head self-attention mechanism to dynamically allocate attention weights to enhance the initially fused low-frequency global features, resulting in deeply fused low-frequency global features. It also uses a convolutional structure to mine cross-modal differences from the initially fused high-frequency detail features, resulting in deeply fused high-frequency detail features.

[0016] In this invention, step S5 includes the following steps: S501, performs preliminary refinement on the deep fusion features of infrared and visible light images; the preliminary refinement includes: convolutional layer, batch normalization layer, LeakyReLU activation function and Sigmoid activation function processing; S502, calculate the Hadamard product of the initially refined features and the deeply fused features to form self-refined features; S503 combines and reconstructs the self-refined features of one modality with the deep fusion features of another modality to generate complementary refined features.

[0017] In this invention, step S6 includes the following steps: S601 splices high- and low-frequency features of complementary refined features in the channel dimension; S602 performs dimensionality reduction transformation on the stitched features, adjusting the number of feature channels through convolution operations to restore the feature dimensions of the original image. S603 utilizes the residual connection mechanism to combine the dimensionality-reduced features with shallow features, resulting in enhanced features that integrate deep fusion information and shallow initial features; S604 uses the Restormer module to perform multi-layer processing on the enhanced features, converts the enhanced features into an image format, and obtains the final fused image.

[0018] The present invention further provides a feature-enhanced infrared and visible light image fusion device, comprising: The preprocessing module is used to preprocess the raw infrared and visible light images; The feature extraction module is used to perform shallow feature extraction on infrared and visible light modal images using a dual-branch encoder, and decompose them to obtain low-frequency global features and high-frequency detail features. The loss function constraint module is used to constrain low-frequency global features and high-frequency detail features using a loss function to optimize network structure training; The feature fusion module is used to superimpose low-frequency global features and then reorganize them, and superimpose high-frequency detail features and then perform cross-modal difference mining to obtain fused features of the two modalities. The complementary information enhancement module is used to refine the fused features, and then perform a Hadamard product operation with the corresponding modal fused features to obtain self-refined features. Then, the self-refined features are combined with the fused features of another modality across modalities to reconstruct complementary refined features. The image reconstruction module is used to reconstruct a fused image by performing feature stitching, dimensionality reduction transformation and residual connection processing on complementary refined features through a decoder.

[0019] In this invention, the device employs a two-stage overlapping training strategy: the first stage trains the feature extraction module and the image reconstruction module; the second stage trains the feature fusion module and the complementary information enhancement module based on the training parameters of the first stage.

[0020] The technical solution provided by this invention may include the following beneficial effects: This invention presents a feature-enhanced infrared and visible light image fusion method. It employs a dual-branch coding approach that balances global and local feature extraction, achieving precise separation of low-frequency and high-frequency features to provide a high-quality feature source for fusion. The method optimizes network parameters using a correlation loss function to ensure feature representation meets fusion requirements, thereby improving model stability and generalization ability. Preliminary integration of cross-modal information is achieved through feature overlay and difference mining, preserving the core characteristics of each modality. Refinement and complementary enhancement processes effectively strengthen the collaborative expression of cross-modal structural features and detailed information, ensuring stable performance of the fused image in terms of texture fidelity and salient object presentation. This allows for the continuous output of high-quality fusion results while maintaining network simplicity. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A flowchart illustrating a feature-enhanced infrared and visible light image fusion method in an exemplary embodiment of this disclosure is shown. Figure 2 This diagram illustrates the system architecture of the first stage of training in an exemplary embodiment of this disclosure. Figure 3 This diagram illustrates the system architecture of the second-stage training in an exemplary embodiment of this disclosure. Figure 4 This diagram illustrates the network structure of the wavelet transform convolution module in an exemplary embodiment of this disclosure. Figure 5 This diagram illustrates the network structure of a multi-level aggregation convolutional module in an exemplary embodiment of this disclosure. Figure 6 This diagram illustrates the network structure of the complementary information enhancement module in an exemplary embodiment of this disclosure. Figure 7 This diagram illustrates an overlapping training strategy in an exemplary embodiment of the present disclosure. Figure 8 The diagram illustrates the visualization of six commonly used datasets in an exemplary embodiment of this disclosure. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0024] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0025] This example implementation first provides a feature-enhanced infrared and visible light image fusion method, please refer to... Figure 1 This method may include: S1-S6, as follows: S1, preprocess the original infrared and visible light images; S2 utilizes a dual-branch encoder to extract shallow features from infrared and visible light modal images, and decomposes them to obtain low-frequency global features and high-frequency detail features; S3 uses a loss function to constrain low-frequency global features and high-frequency detail features, and optimizes the network structure training. S4: Low-frequency global features are superimposed and then recombined, and high-frequency detail features are superimposed and then cross-modal difference mining is performed to obtain the fused features of the two modalities. S5. The fused features are refined and then Hadamard product operation is performed with the corresponding modal fused features to obtain self-refined features. Then, the self-refined features are combined with another modal fused features to reconstruct complementary refined features across modalities. S6 uses a decoder to perform feature splicing, dimensionality reduction transformation, and residual connection processing on complementary refined features to reconstruct the fused image.

[0026] In this embodiment, a dual-branch encoding approach is adopted to balance global and local feature extraction, achieving accurate separation of low-frequency and high-frequency features and providing a high-quality feature source for fusion. The network parameters are optimized using a correlation loss function to ensure that the feature representation meets the fusion requirements, thereby improving model stability and generalization ability. Preliminary integration of cross-modal information is achieved through feature overlay and difference mining, preserving the core characteristics of each modality. Refinement and complementary enhancement processes effectively strengthen the collaborative expression of cross-modal structural features and detailed information, ensuring stable performance of the fused image in terms of texture fidelity and salient object presentation, thus continuously outputting high-quality fusion results while maintaining network simplicity.

[0027] The specific process of each step in the above embodiments will be described below.

[0028] S1 is the data preprocessing process, which specifically includes the following steps: S101: Infrared images are acquired by an infrared imaging device, and visible light images are captured by a regular camera or mobile device. Image registration is performed between the two types of images to form the original dataset. The original data is a W×H two-dimensional image, where W represents the image width and H represents the image height. The actual image size is 640×480, representing the width and height of the image, respectively.

[0029] S102, convert the images in the original dataset to grayscale images. Specifically, the images in the dataset are all color images, i.e., RGB three-channel format. In the data processing stage, they are first converted to grayscale images, at which point the number of channels is 1, representing brightness information.

[0030] S103, the grayscale image is divided into fixed-size image patches using a sliding window method. Specifically, the grayscale image is divided into fixed-size image patches using a sliding window method, which in practice is divided into a set of 128×128 image patches.

[0031] S104: Perform contrast evaluation on the generated image blocks and filter out low-contrast image blocks.

[0032] S105, the valid image blocks are uniformly stored as HDF5 format files, and the dataset is divided into multiple batches. Specifically, HDF5 format files are easy to read quickly; the batch size is usually set according to the graphics card's video memory size. For example, in this application, it is divided into 8 batches in the experiment.

[0033] S2 is the feature extraction process, which specifically includes the following steps: S201 performs shallow feature extraction on the input infrared and visible light images, while capturing local details and global structural information.

[0034] Specifically, feature extraction is performed using a dual-branch encoder, which includes a multi-layer Transformer and a Restormer module for convolutional operations, a BTE module for processing low-frequency features, and a DCE module for processing high-frequency features.

[0035] The Restormer module performs shallow feature extraction on the input infrared and visible light images, capturing both local details and global structural information. Let the input features be: The formula processed by this module is as follows:

[0036]

[0037] in, It is a multi-head, deep, separable attention mechanism. It is a gated feedforward network. It is layer normalization.

[0038] The process for capturing long-range spatial dependencies is as follows: First, the key attention parameters are calculated using depthwise separable convolution, as shown in the following formula:

[0039] Q , K , V Attention mechanisms The query, key, and value.

[0040] After vector normalization to the interval of convergence, the formula is as follows:

[0041] Then the attention weights are calculated using the following formula:

[0042] in, It is a learnable temperature parameter. Finally, after weighted summation, multi-head fusion, and layer normalization, it is compared with the source features. Adding them together gives The formula is as follows:

[0043]

[0044] in, Representative features are pieced together.

[0045] To enhance the representation of local texture and nonlinearity, feature decomposition is first performed, as shown in the following formula:

[0046] Then, feature aggregation is performed using activation functions and Hada code product, and then... Summation yields The formula is as follows:

[0047] in, It is a probability-based smooth activation function.

[0048] S202 decomposes shallow features into two branches: low-frequency global features and high-frequency detail features. Specifically, the BTE and DCE modules further decompose shallow features into these two branches. The BTE module mainly consists of a lightweight Transformer, and its core principle is similar to the Restormer module, used to enhance the global correlation between features to obtain deep low-frequency global features. The DCE module is based on a convolutional neural network structure, used to improve the model's ability to express local texture and edge information, obtaining deep high-frequency detail features. For a detailed overview of the feature processing flow, please refer to [link to documentation]. Figure 2 and Figure 3 .

[0049] S3 is the process of constructing the loss function and applying constraints, which includes the following steps: S301, calculate the correlation between low-frequency features and high-frequency features of infrared and visible light images respectively, using the following formulas:

[0050] in, and It is a feature of the source image. and It is the mean of the corresponding feature. It is a very small constant used to prevent division by zero.

[0051] S302 constructs a correlation loss function based on the correlation between low-frequency and high-frequency features. This correlation loss function consists of two parts: the correlation loss of low-frequency features and the correlation loss of high-frequency features. It guides the network to improve the correlation between high-frequency features and maintain the relative independence of low-frequency features. The formula is as follows:

[0052] in, and These represent the high-frequency detail features of visible light and infrared images, respectively. and These represent the low-frequency global features of visible light and infrared images, respectively.

[0053] S303 constructs a fusion loss function based on the correlation loss function, gradient loss function, and pixel feature loss function, and uses the loss function to optimize the network structure.

[0054] The formula for the total loss in the first stage is as follows:

[0055] in, and These represent the reconstruction losses for the two modalities, primarily calculated using SSIM and MSE. It is correlation loss; It is the gradient consistency loss, calculated by the Sobel operator; and These are weighting factors, set to 2 and 8 respectively.

[0056] The total loss formula for the second stage is as follows:

[0057] in, The fusion loss is a combined loss that represents pixel intensity information and gradient detail information, and the formula is as follows:

[0058] In the formula, The heat information of a person is obtained by calculating the L1 loss between the maximum pixel intensity and the fused image. The L1 loss, which measures detailed texture information, is calculated from the maximum gradient and the gradient of the fused image. To balance the network's attention to the two types of information, it is set to 10.

[0059] S304 optimizes network weights through backpropagation to train and optimize the network structure. Different Adam optimizers are set for different network modules, and corresponding learning rate schedulers are configured for modular training of the network structure.

[0060] S4 is the process of fusing low-frequency and high-frequency features during model training. It mainly utilizes wavelet transform convolution modules and multi-level aggregation convolution modules. The structures of the two modules are as follows: Figure 4 and Figure 5 As shown. S4 specifically includes the following steps: S401, superimpose the low-frequency global features of the infrared image and the visible light image to obtain the preliminary fused low-frequency global features, and superimpose the high-frequency detail features of the infrared image and the visible light image to obtain the preliminary fused high-frequency detail features.

[0061] S402 uses a multi-head self-attention mechanism to dynamically allocate attention weights to enhance the initially fused low-frequency global features, resulting in deeply fused low-frequency global features. It also uses a convolutional structure to mine cross-modal differences from the initially fused high-frequency detail features, resulting in deeply fused high-frequency detail features.

[0062] Specifically, the superimposed low-frequency global features are input into the BTE module, and attention weights are dynamically allocated through a multi-head self-attention mechanism to focus on regions that play a key role in the global structure, suppress redundant information, and enhance the model's global representation ability of low-frequency basic features. For high-frequency detail feature differential modeling, the superimposed high-frequency detail features are input into the DCE module, and cross-modal differences are mined through convolutional structures, such as texture information in visible light images and thermal target edge features in infrared images. The unique and representative detail features of each modality are selected and retained, presenting clear structural and texture-rich detail information in the fusion result.

[0063] S5 utilizes the CCE module (Complementary Information Cross-Enhancement) to achieve complementary information enhancement. The CCE module structure is as follows: Figure 6 As shown. S5 includes the following steps: S501, perform preliminary refinement on the deep fusion features of the infrared image and the visible light image; the preliminary refinement includes: convolutional layer, batch normalization layer, LeakyReLU activation function and Sigmoid activation function processing.

[0064] First, the depth fusion features from the infrared and visible light images are input into the CCE module along two separate branch paths. Then, preliminary refinement is performed on each path, as shown in the following formula:

[0065] in, It is the LeakyReLU activation function. It is the Sigmoid activation function. Representatives criticized normalization. The original features that represent a certain modality.

[0066] S502 calculates the Hadamard product of the initially refined features and the deep fusion features to form self-refined features. The core of self-refinement is the Softmax operation, which aims to probabilistically model the channel dimensions, thereby effectively suppressing some redundant or irrelevant scene information. The formula for the two-branch Softmax is as follows:

[0067]

[0068] in, and The channel saliency weights, representing the two modalities respectively, are obtained from preliminary feature refinement; each Indicates spatial location Place, No. The importance of each channel to the current mode; the superscript of the summation symbol. This is the total number of channels; It is an exponential operation.

[0069] S503 combines self-refined features from one modality with deeply fused features from another modality to reconstruct complementary refined features. Complementary refinement enhances the information interaction between the two modalities, improving the model's ability to perceive and express cross-modal complementary features.

[0070] The mutual refinement formula for the visible light image path is as follows:

[0071] in, It is a self-refining feature of the visible light image pathway. These are the original features of visible light images. These are the original features of infrared images.

[0072] Similarly, the mutual refinement formula for the infrared image path is as follows:

[0073] After refining, the output features, which have undergone self-refining and complementary refining, are used for subsequent image reconstruction tasks.

[0074] S6 is the image reconstruction process, which specifically includes the following steps: S601 concatenates high- and low-frequency features of complementary refined features along the channel dimension. First, the complementary refined features after feature fusion are used as input to the decoder, and then concatenated to form a global feature representation.

[0075] S602 performs dimensionality reduction transformation on the stitched features, adjusting the number of feature channels through convolution operations to restore the feature dimensions of the original image.

[0076] S603 utilizes a residual connection mechanism to combine the dimensionality-reduced features with shallow features, gradually restoring the spatial resolution and detail information of the image, resulting in enhanced features that integrate deep fusion information and shallow initial features.

[0077] S604 utilizes the Restormer module to perform multi-layer processing on the enhanced features, converting them into an image format to obtain the final fused image. This image maintains the consistency of the global structure while possessing rich and clear local detail information.

[0078] The beneficial effects of this invention can be further illustrated by the following simulation experiments: 1. Simulation conditions This invention was implemented in the following hardware and software environment: the central processing unit was an Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, the system architecture was x86_64, and the operating system was Ubuntu 20.04.1 Linux distribution. The computational acceleration environment of this invention was built on the CUDA 12.2 platform, using multiple NVIDIA GeForce RTX 3090 graphics cards for parallel computing, with each card having approximately 24GB (24576 MiB) of video memory. The simulation program was written in Python, edited in the VSCODE integrated development environment, and executed through the system command-line interface.

[0079] 2. Simulation Content (1) Simulation 1 The performance of the method proposed in this invention was verified on existing publicly available infrared and visible light image fusion datasets. Experiments were conducted on six commonly used datasets: RoadScene, TNO, MSRS, M3FD, M2VD, and LLVIP. The corresponding objective evaluation index results are shown in Tables 1 and 2.

[0080] This invention employs eight objective evaluation metrics to quantitatively assess the fusion results, specifically including: metrics based on image statistical features (SD, SF), metrics based on correlation analysis (SCD), metrics based on information theory (EN, MI, and Qabf), metrics based on image structure (SSIM), and metrics based on the human visual perception model (VIF). The specific meanings of the eight metrics are as follows: EN (Information Entropy) measures the amount of information contained in an image; a higher value indicates a richer information content. SD (Standard Deviation) reflects the dispersion of the image's gray-level distribution; a higher value indicates a more dispersed gray-level distribution and carries more information. SF (Spatial Frequency) describes the drastic changes in image gray levels; a higher value indicates a clearer image. MI (Mutual Information) measures the degree of information sharing between the fused image and the source images; a higher value indicates that the fused image retains more information from the source images and is of better quality. SCD (Sum of Differences and Correlation) evaluates the relationship between the fused image and two classes of data. The balance of correlation strength between source images is considered; a higher value indicates better fidelity and information balance in the fusion result. Visual fidelity (VIF) measures the subjective quality of an image based on a human visual perception model; a higher value indicates better quality of the fused image. Qabf (edge ​​information-based index) measures the ability of the fused image to retain edge and detail information; a higher value indicates richer detail information in the fused image. SSIM (structural similarity index) evaluates image quality by comprehensively comparing brightness, contrast, and structural information; its value ranges from -1 to 1. A value closer to 1 indicates that the fused image is closer to the source image in overall structure, meaning it retains more complete information.

[0081] Table 1 Performance metrics of the present invention on two classic datasets

[0082] Table 2 Performance metrics of the present invention on other datasets

[0083] As can be seen from Tables 1 and 2, the present invention performs well on multiple key evaluation indicators of various fusion datasets, which fully verifies that the proposed method has good fusion performance in multiple dimensions such as the richness of source image information, image clarity, preservation of image structure and visual perception.

[0084] (2) Simulation 2 Visualization experiments were conducted using the method proposed in this invention on six existing publicly available infrared and visible light image fusion datasets. The corresponding fusion results are as follows: Figure 8 As shown. Figure 8 In the image, the vertical columns represent images of different modalities, from left to right: visible light image, infrared image, and fused image; the horizontal rows correspond to different datasets, from top to bottom: RoadScene dataset, TNO dataset, MSRS dataset, M3FD dataset, M2VD dataset, and LLVIP dataset. The fusion result is a grayscale image directly generated by the image fusion model of this invention.

[0085] Depend on Figure 8 It can be seen that the method proposed in this invention can effectively integrate complementary information in visible light images and infrared images, and exhibits excellent performance in improving the overall image quality.

[0086] This disclosure also provides an infrared and visible light image fusion device based on feature enhancement, including: a preprocessing module, a feature extraction module, a loss function constraint module, a feature fusion module, a complementary information enhancement module, and an image reconstruction module.

[0087] Specifically, the preprocessing module is used to preprocess the original infrared and visible light images.

[0088] The feature extraction module is used to perform shallow feature extraction on infrared and visible light modal images using a dual-branch encoder, and decompose them into low-frequency global features and high-frequency detail features.

[0089] The loss function constraint module is used to constrain low-frequency global features and high-frequency detail features using a loss function to optimize network structure training.

[0090] The feature fusion module is used to superimpose low-frequency global features and then reorganize them, and superimpose high-frequency detail features and then perform cross-modal difference mining to obtain fused features of the two modalities. The feature fusion module includes: wavelet transform convolution module and multi-level aggregation convolution module.

[0091] The wavelet transform convolution module addresses the performance saturation issue caused by simply increasing the number of convolution kernels, which can lead to parameter redundancy. Larger kernels tend to extract low-frequency features, while conventional convolution operations are better suited for high-frequency detail extraction. The attention mechanism excels at low-frequency global modeling, thus enabling the Transformer to effectively handle long-distance dependencies. Two-dimensional wavelet transform preserves spatial resolution while modeling in the frequency domain, capturing both global and local features. Using small-scale convolution kernels on the wavelet sub-band frequency domain feature map aligns with the characteristics of spatial convolution and effectively expands the receptive field in multi-level cascading, with low parameter overhead. This invention employs a three-level wavelet decomposition, with each level acting on the upper-level low-frequency components. Starting from the third layer, detailed features are extracted through convolution, and semantic information is reconstructed using inverse wavelet transform, optimizing progressively. The sub-band convolution after wavelet decomposition has a larger receptive field than the original image convolution, and high-frequency information is effectively utilized during propagation, enhancing detail representation.

[0092] Multi-level aggregation convolution module: adopts an encoder-decoder architecture. The encoder consists of four levels of three-dimensional convolution downsampling to extract contextual semantics; the decoder combines upsampling and deep feature fusion modules to gradually restore the feature dimension.

[0093] The feature fusion module can aggregate and enhance features at all levels, combining the feature representations of the encoder and decoder and expanding the receptive field. In the encoding stage, features are extracted using dynamic large-kernel convolution, halving the spatial size and doubling the number of channels; the receptive field increases after concatenation. In the decoding stage, the features output by the deep feature fusion encoder and decoder are restored to their spatial dimensions using dynamic large-kernel convolution and fused with the initial input to obtain the final output. This module generates feature weights through pooling, convolution, and activation functions, possessing adaptive feature selection capabilities. It employs multi-scale deep convolutional kernels to model contextual information, sequentially aggregating and expanding the receptive field, and can adaptively select discriminative features based on the global context.

[0094] The complementary information enhancement module is used to refine the fused features, and then perform a Hadamard product operation with the corresponding modal fused features to obtain self-refined features. Then, the self-refined features are combined with the fused features of another modality across modalities to reconstruct complementary refined features; the complementary information enhancement module is also the complementary information interaction module.

[0095] Among them, such as Figure 6 As shown, the CCE module includes two feature refinement sub-modules. Features from the infrared and visible light modalities are input along two branch paths, respectively. First, they pass through the feature refinement sub-module, which consists of convolution, batch normalization, LeakyReLU activation function, and Sigmoid activation function. The processed features are then multiplied by Hadamard (element-wise multiplication) to form self-refining features, effectively suppressing some redundant or irrelevant scene information. Subsequently, these self-refined features are fused and reconstructed with the original modal features to generate mutual-refining features, achieving information interaction and complementary enhancement between the two modalities, further improving the model's ability to express cross-modal features. The CCE module focuses on feature interaction between different modalities, achieving cross-modal feature refinement by strengthening complementary information, thereby improving the model's ability to perceive complementary features.

[0096] The image reconstruction module uses a decoder to perform feature concatenation, dimensionality reduction, and residual connection processing on complementary refined features to reconstruct a fused image. The decoder module is responsible for reconstructing the fused image. The decoder mainly consists of a Restormer module, used to combine the two-branch fused features. Basic feature fusion and detail feature fusion are completed in parallel in the previous stage, and the two types of fused features output from each stage serve as input to the decoder. The decoder effectively integrates the two types of fused features through feature concatenation, dimensionality reduction, and residual connection operations, gradually restoring the spatial structure and ultimately generating the fused image. This design ensures that the output image maintains global structural consistency while possessing rich and clear local detail information.

[0097] It should be noted that, as Figure 7 As shown, the device employs a two-stage overlapping training strategy: the first stage trains the feature extraction module and the image reconstruction module; the second stage trains the feature fusion module and the complementary information enhancement module based on the training parameters of the first stage.

[0098] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0099] Furthermore, this invention employs an autoencoder module for feature extraction. The encoder and decoder consist of a CNN and a Transformer, respectively. The Transformer can be replaced with a novel Mamba architecture or other Transformer variants, such as the Swin Transformer or Mobile ViT. The autoencoder structure used in this invention can be entirely replaced with other network structures, such as Generative Adversarial Networks (GANs) or Diffusion models. By introducing the characteristics of novel network architectures, this invention explores new methods for network structure design and implementation.

[0100] In summary, this application has the following beneficial effects: (1) To address the problem that existing lightweight pixel-level fusion methods rely excessively on low-level information such as local gradients and brightness, making it difficult to effectively model the global semantic structure, this invention introduces wavelet transform convolution and multi-level aggregation convolution into the convolution module to enhance the network's ability to capture the global context while maintaining the advantages of convolutional neural networks in modeling local details. By improving the network's ability to perceive multi-scale semantics, this invention overcomes the limitation of traditional pixel-level unified fusion methods that neglect high-level semantics, enabling the fusion process to balance detail preservation and semantic structure expression, thereby obtaining fusion results with greater semantic consistency and structural integrity.

[0101] (2) To address the shortcomings of low-rank representation-based fusion methods, which impose strong structural assumptions on the source image and result in insufficient fusion flexibility, this invention adopts a mainstream autoencoder architecture to uniformly represent cross-modal features and introduces L1 constraints based on the difference between the maximum gradient and the gradient of the fused image to optimize model parameters. Compared to the strict priors of low-rank modeling, the design of this invention can preserve visible light texture details while highlighting infrared salient structures, achieving a dynamic balance between detail layer and semantic layer features, thereby effectively improving the generalization performance of the algorithm under different scenarios and multimodal input conditions.

[0102] (3) To address the problems of complex structure, unstable training process, and difficulty in balancing global and local feature modeling in current end-to-end multimodal fusion detection frameworks, this invention constructs a lightweight encoder by combining wavelet transform convolution and multi-level aggregated convolution, and adopts a two-stage overlapping training strategy to achieve a good balance between global information capture and local detail preservation. At the same time, the complementary information enhancement (CCE) module proposed in this invention can effectively strengthen the collaborative expression of cross-modal structural features and detail information, ensuring stable performance of fused images in terms of texture fidelity and salient target presentation, thereby continuously outputting high-quality fusion results while maintaining network simplicity.

[0103] It should be noted that although several modules of the system for executing actions are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment. Components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0104] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

[0105] It should be noted that the installation of image acquisition and personal identification equipment in public places involved in this application is necessary for maintaining public safety, complies with relevant national regulations, and is accompanied by prominent warning signs. The collected personal images and identification information can only be used for the purpose of maintaining public safety and will not be used for other purposes; or the images, personal identification data, etc. in this application are all legally and compliantly obtained or collected with the individual's separate consent.

Claims

1. A method for fusing infrared and visible light images based on feature enhancement, characterized in that, include: S1, preprocess the original infrared and visible light images; S2 utilizes a dual-branch encoder to extract shallow features from infrared and visible light modal images, and decomposes them to obtain low-frequency global features and high-frequency detail features; S3 uses a loss function to constrain low-frequency global features and high-frequency detail features, and optimizes the network structure training. S4: Low-frequency global features are superimposed and then recombined, and high-frequency detail features are superimposed and then cross-modal difference mining is performed to obtain the fused features of the two modalities. S5. The fused features are refined and then Hadamard product operation is performed with the corresponding modal fused features to obtain self-refined features. Then, the self-refined features are combined with another modal fused features to reconstruct complementary refined features across modalities. S6 uses a decoder to perform feature splicing, dimensionality reduction transformation, and residual connection processing on complementary refined features to reconstruct the fused image.

2. The infrared and visible light image fusion method based on feature enhancement according to claim 1, characterized in that, S1 includes the following steps: S101: Infrared images are acquired by an infrared imaging device, and visible light images are acquired by a regular camera or mobile device. Image registration is performed on the two types of images to form the original dataset. S102, convert the images in the original dataset to grayscale images; S103, a sliding window method is used to divide the grayscale image into image blocks of fixed size; S104, perform contrast evaluation on the generated image blocks and filter out low-contrast image blocks; S105 stores the valid image blocks uniformly as HDF5 format files and divides the dataset into multiple batches.

3. The infrared and visible light image fusion method based on feature enhancement according to claim 1, characterized in that, S2 includes the following steps: S201 performs shallow feature extraction on the input infrared and visible light images, while capturing local details and global structural information. S202 decomposes shallow features into two branches: low-frequency global features and high-frequency detail features.

4. The infrared and visible light image fusion method based on feature enhancement according to claim 3, characterized in that, S3 includes the following steps: S301, calculate the correlation between low-frequency features and high-frequency features of infrared and visible light images respectively; S302, construct a correlation loss function based on the correlation between low-frequency features and high-frequency features; S303, construct a fusion loss function based on the correlation loss function, gradient loss function, and pixel feature loss function; S304 optimizes the network weights through backpropagation algorithm, thereby training and optimizing the network structure.

5. The infrared and visible light image fusion method based on feature enhancement according to claim 4, characterized in that, S4 includes the following steps: S401, superimpose the low-frequency global features of the infrared image and the visible light image to obtain the preliminary fused low-frequency global features, and superimpose the high-frequency detail features of the infrared image and the visible light image to obtain the preliminary fused high-frequency detail features. S402 uses a multi-head self-attention mechanism to dynamically allocate attention weights to enhance the initially fused low-frequency global features, resulting in deeply fused low-frequency global features. It also uses a convolutional structure to mine cross-modal differences from the initially fused high-frequency detail features, resulting in deeply fused high-frequency detail features.

6. The infrared and visible light image fusion method based on feature enhancement according to claim 5, characterized in that, S5 includes the following steps: S501, performs preliminary refinement on the deep fusion features of infrared and visible light images; the preliminary refinement includes: convolutional layer, batch normalization layer, LeakyReLU activation function and Sigmoid activation function processing; S502, calculate the Hadamard product of the initially refined features and the deeply fused features to form self-refined features; S503 combines and reconstructs the self-refined features of one modality with the deep fusion features of another modality to generate complementary refined features.

7. The infrared and visible light image fusion method based on feature enhancement according to claim 6, characterized in that, S6 includes the following steps: S601 splices high- and low-frequency features of complementary refined features in the channel dimension; S602 performs dimensionality reduction transformation on the stitched features, adjusting the number of feature channels through convolution operations to restore the feature dimensions of the original image. S603 utilizes the residual connection mechanism to combine the dimensionality-reduced features with shallow features, resulting in enhanced features that integrate deep fusion information and shallow initial features; S604 uses the Restormer module to perform multi-layer processing on the enhanced features, converting the enhanced features into an image format to obtain the final fused image.

8. A feature-enhanced infrared and visible light image fusion device, characterized in that, include: The preprocessing module is used to preprocess the raw infrared and visible light images; The feature extraction module is used to perform shallow feature extraction on infrared and visible light modal images using a dual-branch encoder, and decompose them to obtain low-frequency global features and high-frequency detail features. The loss function constraint module is used to constrain low-frequency global features and high-frequency detail features using a loss function to optimize network structure training. The feature fusion module is used to superimpose low-frequency global features and then reorganize them, and superimpose high-frequency detail features and then perform cross-modal difference mining to obtain fused features of the two modalities. The complementary information enhancement module is used to refine the fused features, and then perform a Hadamard product operation with the corresponding modal fused features to obtain self-refined features. Then, the self-refined features are combined with the fused features of another modality across modalities to reconstruct complementary refined features. The image reconstruction module is used to reconstruct a fused image by performing feature stitching, dimensionality reduction transformation and residual connection processing on complementary refined features through a decoder.

9. The infrared and visible light image fusion device based on feature enhancement according to claim 8, characterized in that, The device employs a two-stage overlapping training strategy: the first stage trains the feature extraction module and the image reconstruction module; The second stage trains the feature fusion module and the complementary information enhancement module based on the training parameters from the first stage.

Images