Infrared and visible light image fusion method fusing adaptive decomposition and multi-scale transformation

Abstract

The invention discloses an infrared and visible light image fusion method fusing adaptive decomposition and multi-scale transformation, and relates to the technical field of multi-modal image fusion. In order to solve the problems of detail loss, noise interference, structural distortion and the like in infrared and visible light image fusion, the method comprises the steps of firstly performing normalization and registration preprocessing on an input image, then extracting low-frequency and high-frequency features by adopting adaptive Fourier decomposition, and performing sparse representation and orthogonal matching pursuit fusion on a low-frequency part to obtain a low-frequency part and a high-frequency part; and the high-frequency part guides dynamic weighted fusion according to local energy and information entropy, and finally reconstruction is performed and cross-scale consistency verification is introduced to improve fusion robustness. The method can be widely applied to fusion of infrared and visible light images, and the quality of the fused image is improved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of image processing and computer vision, specifically a method for fusing infrared and visible light images based on a combination of Adaptive Fourier Decomposition (AFD) and Multi-scale Transform (MST). This method significantly improves the fusion quality of infrared and visible light images by introducing an adaptive learning mechanism, and can be widely applied in edge device scenarios such as UAV navigation and target detection, providing more comprehensive and accurate image support for visual perception in complex environments. Therefore, considering hardware limitations, a traditional method incorporating an adaptive learning mechanism is proposed. Background Technology

[0002] With the rapid development of infrared and visible light imaging technologies, infrared and visible light images are widely used in various scenarios. Infrared images can capture thermal radiation information, making them suitable for low-light or night vision scenarios, while visible light images provide rich texture and color details. However, due to their different imaging mechanisms, single-modal images are insufficient to meet the perception requirements for comprehensive information in complex scenes. For example, infrared images have strong thermal feature representation capabilities in target detection, but their detail resolution is low; visible light images are rich in detail under sufficient lighting conditions, but they are prone to failure under low-light or occlusion conditions. Therefore, infrared and visible light image fusion technology has become an important research direction in the field of computer vision.

[0003] Current mainstream image fusion methods include spatial domain-based methods, transform domain-based methods, and deep learning-based methods. Transform domain-based methods, due to their advantages in processing frequency domain features and detail information, exhibit good performance in infrared and visible light image fusion. Multi-scale transforms, such as Discrete Wavelet Transform (DWT) and Non-Subsampled Prototype Transform (NSCT), can effectively decompose the structural hierarchy of images; however, their reliance on fixed basis functions makes it difficult to adapt to the modal differences between infrared and visible light images, potentially leading to problems such as loss of detail, blurred texture, or information redundancy in the fusion results. Furthermore, traditional high-frequency fusion strategies (such as maximum selection or weighted averaging) have weak noise resistance and are prone to introducing artifacts; low-frequency fusion strategies also have limitations in preserving structural features.

[0004] Given the current situation, how to preserve high-quality detail information and suppress noise interference during the fusion process, while also achieving good adaptability and computational efficiency, has become an urgent problem to be solved. This invention proposes a method for fusing infrared and visible light images using adaptive decomposition and multi-scale transformation. By combining an adaptive atom selection mechanism and dynamic fusion rules with traditional transform domain methods and adaptive learning strategies, it aims to improve image detail preservation, enhance fusion robustness, and optimize computational efficiency. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies in terms of image feature extraction adaptability, fusion rule robustness, and computational efficiency, this invention proposes a method for fusing infrared and visible light images by incorporating adaptive decomposition and multi-scale transformation. This method improves the fusion quality of infrared and visible light images by introducing an adaptive learning mechanism, combined with sparse representation and dynamic weight allocation strategies. It is suitable for edge device applications such as UAV navigation and target detection. Therefore, considering hardware limitations, a traditional method incorporating an adaptive learning mechanism is proposed. This method consists of the following steps:

[0006] Step 1: Image preprocessing stage. Normalize and grayscale standardize the input infrared and visible light images to eliminate brightness and contrast differences between different imaging modalities. Improve image texture stability through edge-preserving filtering to provide a foundation for subsequent fusion.

[0007] Step 2: Adaptive Fourier Decomposition (AFD) stage, using AFD to perform frequency domain transformation on the image. This step constructs a redundant atom dictionary and employs an adaptive atom selection mechanism based on image content to extract low-frequency and high-frequency features layer by layer. Low-frequency components mainly reflect the overall structure and grayscale trends of the image, while high-frequency components reflect detailed edges and texture features, possessing higher information density.

[0008] Step 3: In the low-frequency fusion stage, the low-frequency images obtained from the decomposition of the two images are processed using a sliding window block method. Each image block is sparsely encoded using the Orthogonal Matching Pursuit (OMP) algorithm, while L1 regularization is introduced to enhance compression and robustness. The fusion strategy selects the most expressive sparse coefficients based on the maximum L1 norm, thereby obtaining a fused low-frequency feature map with stronger structural consistency.

[0009] Step 4: In the high-frequency fusion stage, a dynamic fusion rule based on local energy and information entropy is proposed. By analyzing the energy distribution of high-frequency atoms in AFD, higher weights are assigned to high-frequency atoms in texture-rich regions, while sparse constraints are introduced in low-texture regions to control the high-frequency response and suppress noise diffusion. This method, combined with saliency detection and a region-adaptive weighting mechanism, further enhances the image detail restoration effect.

[0010] Step 5: In the image reconstruction stage, the fused low-frequency and high-frequency information undergoes inverse AFD transformation to reconstruct the final fused image. To ensure the consistency of feature representation at different scales, this invention introduces a cross-scale consistency check mechanism based on orthogonality verification. When multi-scale response anomalies are detected, the fusion weights are dynamically adjusted or a local compensation strategy is executed, effectively avoiding structural distortion. The main technical features of this invention are as follows:

[0011] Furthermore, the AFD in step 2 includes:

[0012] A redundant dictionary is constructed, and the decomposition process is dynamically adjusted through an adaptive atomic selection mechanism.

[0013] A hierarchical decomposition mechanism is adopted to extract local features at different scales, thereby enhancing the adaptability to images of different modalities.

[0014] Furthermore, the low-frequency fusion in step 3 includes:

[0015] The low-frequency sub-band is divided into image blocks using the sliding window technique;

[0016] Perform mean normalization on each image patch;

[0017] A sparse representation optimization model is adopted, and the sparse coefficients are solved by the orthogonal matching pursuit (OMP) algorithm;

[0018] L1 regularization constraints are introduced to optimize the fusion quality of low-frequency features;

[0019] The formula for calculating the sparsity coefficient is:

[0020]

[0021] in, and Let D be the normalized image patch vector, D be the dictionary, and K be the sparsity constraint.

[0022] The merging and summing are performed using the "maximum L1 norm" rule to obtain the fused sparse vector:

[0023]

[0024] Furthermore, the dynamic high-frequency fusion in step 4 includes: calculating the energy distribution of each atom after AFD decomposition, adopting a local energy maximization strategy in the high-entropy region to assign higher weights to high-energy atoms; and introducing sparsity constraints in the smooth region to reduce noise interference and improve the clarity of the fused image.

[0025] Furthermore, step 5 further includes a cross-scale consistency verification algorithm, which uses the orthogonality feature of AFD atoms to check the consistency of decomposition results at different scales; when distortion is detected, the consistency of the fused image is ensured by adjusting the fusion weights or performing feature alignment, so as to avoid fusion distortion caused by modal differences.

[0026] Furthermore, the method employs GPU parallelization to accelerate the AFD decomposition process and optimizes computational efficiency through adaptive step size control to meet real-time requirements.

[0027] Furthermore, the adaptive step size control mechanism dynamically adjusts the sliding window step size according to the image resolution, using a larger step size in high-resolution images to improve efficiency, and a smaller step size in low signal-to-noise ratio images to ensure fusion accuracy.

[0028] The advantages of this method include:

[0029] (1) The method integrates the frequency domain adaptability of AFD with the structural expression capability of MST, and has a stronger feature adaptability between different modal images. It can flexibly handle image pairs with complex structures and obvious contrast differences.

[0030] (2) By using sparse representation and orthogonal matching pursuit method, information redundancy is effectively compressed and the stability of low-frequency structural feature extraction is enhanced. The introduction of L1 norm further improves the energy aggregation and detail integrity of fused image.

[0031] (3) The high-frequency dynamic fusion mechanism not only considers local energy and texture information, but also integrates regional saliency and information entropy estimation, which can effectively enhance image edge, texture and fine-grained information, while having strong noise resistance.

[0032] (4) Introduce a cross-scale consistency verification mechanism to ensure that the fusion content between different frequency bands and different modalities is consistent, effectively avoiding common fusion distortion problems such as structural mismatch and image blurring;

[0033] (5) The fusion algorithm supports GPU parallel optimization and adaptive step size strategy, which can dynamically adjust the computation intensity according to the image size and complexity, significantly improve the overall fusion speed, and meet the high requirements of terminal devices such as UAVs for real-time processing. Attached Figure Description

[0034] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the invention in any way. In the drawings:

[0035] Figure 1 A flowchart of an infrared and visible light image fusion method that integrates adaptive decomposition and multi-scale transformation is provided in an embodiment of the present invention;

[0036] Figure 2 A flowchart of the dynamic high-frequency fusion rules and AFD sparse constraint design provided for embodiments of the present invention;

[0037] Figure 3 This is a comparison image of the visual effects before and after image fusion provided in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] Figure 1 A flowchart illustrating an infrared and visible light image fusion method incorporating adaptive decomposition and multi-scale transformation, as provided in an embodiment of the present invention, is shown below. Figure 1 As shown, this invention provides a method for fusing infrared and visible light images by integrating adaptive decomposition and multi-scale transformation. The method includes the following steps:

[0040] Step 1: Image preprocessing stage. The input infrared and visible light images are normalized and grayscale standardized to eliminate the differences in brightness and contrast between different imaging modalities, providing a basis for subsequent fusion.

[0041] This invention uses the M3FD dataset as the source of experimental data. M3FD (Multi-Modal Fusion Dataset) is a publicly available dataset specifically designed for infrared and visible light image fusion, containing over 6000 pairs of infrared and visible light images, covering various scenarios (such as day and night surveillance, drone aerial photography, target detection, etc.), with resolutions ranging from 320×240 to 1280×720. The dataset provides registered image pairs with a registration error of less than 1 pixel, supporting various fusion tasks (such as target detection, semantic segmentation). M3FD includes rich environmental conditions (lighting changes, weather changes, occlusion, etc.) and target types (pedestrians, vehicles, buildings, etc.), and provides pixel-level annotations, suitable for quantitative evaluation (such as PSNR, SSIM, MI, etc.). Its advantages lie in its large data volume, diverse scenes, and high registration accuracy, making it suitable for developing and validating infrared and visible light fusion algorithms; its limitation is that some low-resolution images may affect detail extraction, requiring preprocessing techniques to enhance the fusion effect. This dataset is obtained through a public platform and must adhere to academic citation guidelines; it is an important resource for research on infrared and visible light image fusion.

[0042] In the preprocessing stage, all input images are first normalized. The normalization formula is as follows:

[0043]

[0044] This step maps the pixel values ​​of different modal images to a uniform range of [0,1], eliminating the problem of inconsistent grayscale distribution caused by differences in imaging devices.

[0045] Step 2: Adaptive Fourier Decomposition Stage, using AFD to perform frequency domain transformation on the image.

[0046] This step employs an innovative Adaptive Fourier Decomposition (AFD) method to perform multi-scale feature decomposition on the preprocessed image, extracting local features at different scales. Specifically, the infrared and visible light images are first converted to specific transform domains, and then the redundant dictionary of AFD is used to adaptively select atoms to extract local features of anatomical structures. By constructing a redundant dictionary and adaptively selecting atoms, important information in the image can be captured more effectively. During training, model parameters are adjusted to improve model performance. Through continuous iteration and optimization, the model can learn an effective representation of image features, thereby improving the quality of the fused image. The specific implementation process is as follows:

[0047] A redundant dictionary is constructed, and the decomposition process is dynamically adjusted through an adaptive atomic selection mechanism.

[0048] A hierarchical decomposition mechanism is adopted to extract local features at different scales, thereby enhancing the adaptability to images of different modalities.

[0049] Step 3: Low-frequency fusion stage

[0050] For the low-frequency images obtained from the decomposition of two images, a sliding window block processing is adopted. Each image block is sparsely encoded by the Orthogonal Matching Pursuit (OMP) algorithm. At the same time, L1 regularization is introduced to enhance compression and robustness. The fusion strategy selects the most expressive sparse coefficients based on the maximum L1 norm, thereby obtaining a fused low-frequency feature map with stronger structural consistency.

[0051] Low-frequency fusion is achieved through the following fusion rules:

[0052] (1) Apply sliding window technology to L A and L B Divided into sizes The image patch is divided into two parts, from the top left corner to the bottom right corner, with a step size of s pixels. Assume that in L... A and L B There are T blocks in each, represented as and

[0053] (2) For each position, Rearrange into column vectors Then, the mean of each vector is normalized to zero using the following formula, resulting in...

[0054]

[0055] Where 1 represents an n×1 vector consisting entirely of 1s. and They are and The mean of all elements in the dataset.

[0056] (3) Calculate using the Orthogonal Matching Pursuit (OMP) algorithm sparse coefficient vector

[0057]

[0058] Where D is the learned dictionary.

[0059] (4) Merge using the "maximum L1 norm" rule and To obtain the fused sparse vector:

[0060]

[0061] and The fusion result is calculated using the following formula:

[0062]

[0063] The combined mean is obtained using the following formula:

[0064]

[0065] (5) and Repeat the above process for all source image patches to obtain all fused vectors. Let L F This represents the result of low-pass fusion. For each Reshape it into a block Then Insert it in L F The original position in the LF. Because the blocks overlap, the value of each pixel in the LF is the average of its accumulated counts.

[0066] Step 4: High-frequency fusion stage

[0067] Figure 2 This invention provides a flowchart of the dynamic high-frequency fusion rule and AFD sparse constraint design for embodiments of the present invention. In the process of infrared and visible light image fusion, the processing of high-frequency information is crucial because it carries the details and edge features of the image. However, traditional high-frequency fusion methods rely on fixed rules (such as the "maximum absolute value" rule), which have significant limitations when processing images of different modalities. To effectively address these issues, this invention proposes a dynamic high-frequency fusion rule based on AFD guidance and combines it with sparse constraints to optimize the fusion process.

[0068] The dynamic high-frequency fusion rule: To overcome the limitations of traditional methods, this invention introduces a dynamic fusion strategy. This strategy first uses AFD decomposition technology to process infrared and visible light images, adaptively determining the fusion weights by calculating the energy distribution of each atom. For high-entropy regions in the image (such as detailed areas like blood vessel branches), a local energy maximization strategy is adopted, assigning greater weights to high-energy atoms to highlight these key details. For smooth regions, sparsity constraints are introduced, and regularization techniques are used to reduce the impact of noise, improving the smoothness and clarity of the image.

[0069] like Figure 2 As shown, the specific implementation of the present invention is as follows: AFD decomposition is performed on the input infrared and visible light images to obtain atomic coefficients at different scales; the energy of each atom is calculated, and the fusion weight is determined according to the energy distribution; the fusion weight is dynamically adjusted according to the atomic energy distribution to highlight the detailed information in the high-entropy region, while sparse constraints are applied in the smooth region to reduce noise.

[0070] Step 5: Image Reconstruction Stage

[0071] The fused low-frequency and high-frequency information is subjected to inverse AFD transformation to reconstruct the final fused image. In order to ensure the consistency of feature representation at different scales, this invention introduces a cross-scale consistency check mechanism based on orthogonality verification. When multi-scale response anomalies are detected, the fusion weights are dynamically adjusted or a local compensation strategy is executed to effectively avoid structural distortion.

[0072] Cross-scale consistency verification: To ensure the consistency of the fused image across different scales, this invention proposes a cross-scale consistency verification algorithm based on the atomic orthogonality of AFD. During the fusion process, by checking the consistency of AFD decomposition results at different scales, fusion distortion caused by modal differences can be detected and corrected in a timely manner. If distortion is detected, the problem can be solved by adjusting the fusion weights or performing feature alignment, thereby ensuring that the fused image maintains structural and feature consistency across different scales.

[0073] Figure 3 This is a comparison image of the visual effects before and after image fusion provided in an embodiment of the present invention.

[0074] like Figure 3As shown, the fusion results of the M3FD dataset are displayed. After introducing multi-scale feature decomposition and adaptive weighting strategies, the trained image fusion network can generate fused images with rich details, balanced contrast, and complete structural information. Specifically, in the fused image, target edges, texture details, and thermal feature representation all exhibit high clarity and high consistency, further verifying the effectiveness of the infrared and visible light image fusion method based on adaptive decomposition and multi-scale transformation proposed in this invention.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of the invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusing infrared and visible light images by integrating adaptive decomposition and multi-scale transformation, characterized in that, Includes the following steps: Step 1: Image preprocessing stage. Normalize and grayscale standardize the input infrared and visible light images to eliminate brightness and contrast differences between different imaging modalities. This stage may also include image alignment and registration operations to ensure that the geometric structure of the source images is consistent, providing a basis for subsequent fusion. Step 2: Adaptive Fourier Decomposition (AFD) stage. AFD is used to transform the image in the frequency domain. This step constructs a redundant atom dictionary and adopts an adaptive atom selection mechanism based on image content to extract low-frequency and high-frequency features layer by layer. The low-frequency components mainly reflect the large structure and gray-level trend of the image, while the high-frequency components reflect the details, edges and texture features, and have high information density. Step 3: Low-frequency fusion stage. For the low-frequency images obtained from the decomposition of the two images, a sliding window block processing is adopted. Each image block is sparsely encoded by the Orthogonal Matching Pursuit (OMP) algorithm. At the same time, L1 regularization is introduced to enhance compression and robustness. The fusion strategy selects the most expressive sparse coefficients based on the maximum L1 norm, thereby obtaining a fused low-frequency feature map with stronger structural consistency. Step 4: In the high-frequency fusion stage, a dynamic fusion rule based on local energy and information entropy is proposed. By analyzing the energy distribution of high-frequency atoms in AFD, higher weights are assigned to high-frequency atoms in texture-rich regions, while sparse constraints are introduced in low-texture regions to control the high-frequency response and suppress noise diffusion. This method, combined with saliency detection and a region-adaptive weighting mechanism, further enhances the image detail restoration effect. Step 5: In the image reconstruction stage, the fused low-frequency and high-frequency information is subjected to inverse AFD transformation to reconstruct the final fused image. In order to ensure the consistency of feature representation at different scales, this invention introduces a cross-scale consistency check mechanism based on orthogonality verification. When multi-scale response anomalies are found, the fusion weights are dynamically adjusted or a local compensation strategy is executed to effectively avoid structural distortion.

2. The infrared and visible light image fusion method based on adaptive decomposition and multi-scale transformation according to claim 1, characterized in that, The AFD in step 2 includes: A redundant dictionary is constructed, and the decomposition process is dynamically adjusted through an adaptive atomic selection mechanism. A hierarchical decomposition mechanism is adopted to extract local features at different scales, thereby enhancing the adaptability to images of different modalities.

3. The infrared and visible light image fusion method based on adaptive decomposition and multi-scale transformation according to claim 1, characterized in that, The low-frequency fusion in step 3 includes: The low-frequency sub-band is divided into image blocks using the sliding window technique; Perform mean normalization on each image patch; A sparse representation optimization model is adopted, and the sparse coefficients are solved by the orthogonal matching pursuit (OMP) algorithm; L1 regularization constraints are introduced to optimize the fusion quality of low-frequency features.

4. The infrared and visible light image fusion method based on adaptive decomposition and multi-scale transformation according to claim 1, characterized in that, The dynamic high-frequency fusion in step 4 includes: calculating the energy distribution of each atom after AFD decomposition, adopting a local energy maximization strategy in high-entropy regions to assign higher weights to high-energy atoms, and introducing sparsity constraints in smooth regions to reduce noise interference and improve the clarity of the fused image.

5. The infrared and visible light image fusion method based on adaptive decomposition and multi-scale transformation according to claim 1, characterized in that, Step 5 further includes a cross-scale consistency verification algorithm, which uses the orthogonality feature of AFD atoms to check the consistency of decomposition results at different scales; when distortion is detected, the consistency of the fused image is ensured by adjusting the fusion weights or performing feature alignment, so as to avoid fusion distortion caused by modal differences.

6. The infrared and visible light image fusion method based on adaptive decomposition and multi-scale transformation according to claim 1, characterized in that, The method uses GPU parallelization to accelerate the AFD decomposition process and optimizes computational efficiency through adaptive step size control to meet real-time requirements.

7. The infrared and visible light image fusion method based on adaptive decomposition and multi-scale transformation according to claim 6, characterized in that, The adaptive step size control mechanism dynamically adjusts the sliding window step size according to the image resolution. A larger step size is used in high-resolution images to improve efficiency, while a smaller step size is used in low signal-to-noise ratio images to ensure fusion accuracy.

Images