Image fusion model modeling method based on visible light and infrared light and program product
By using an image fusion model optimized through adaptive multi-level feature fusion and joint loss function, the shortcomings of infrared and visible light image fusion technology are addressed, generating highly adaptable and high-quality fused images and improving the performance of downstream tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-19
- Publication Date
- 2026-03-13
AI Technical Summary
Existing infrared and visible light image fusion technologies have shortcomings in terms of feature fusion precision, illumination adaptability, and high-level vision task synergy, resulting in blurred textures, edge distortion, and loss of important semantic information in the fused images.
We employ an image fusion model based on visible light and infrared, and improve the quality of fused images and the performance of downstream tasks by adaptive multi-level feature fusion and joint loss function optimization, combined with semantic segmentation tasks.
It generates fused images with clear textures that highlight thermal targets, adapts to complex lighting conditions, and improves the segmentation accuracy and visual understanding capabilities of downstream tasks.
Smart Images
Figure CN121661454A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and image processing technology, and in particular relates to a modeling method and program product for image fusion model based on visible light and infrared. Background Technology
[0002] With the widespread application of intelligent perception systems in fields such as autonomous driving, security monitoring, and nighttime navigation, effectively fusing sensor information from different modalities to enhance environmental perception capabilities has become a key technological challenge. Among these challenges, the fusion of infrared and visible light images has attracted significant attention due to their strong complementarity: infrared images are sensitive to thermal radiation and can highlight targets such as pedestrians and vehicles in low light and adverse weather conditions; visible light images provide rich texture, color, and spatial details, conforming to human visual perception. Therefore, generating a high-quality fused image that highlights thermal targets while preserving clear textures is crucial for improving the performance of advanced visual tasks such as object detection and semantic segmentation.
[0003] In recent years, models based on convolutional neural networks, generative adversarial networks, and autoencoders have significantly improved image fusion results. However, existing technologies still have some key shortcomings: Many methods employ simple element-wise addition or maximum value operations for feature fusion, failing to effectively model the complementarity between infrared and visible light modes, resulting in blurred textures or distorted edges in the fused image. Existing fusion networks typically do not consider the actual lighting conditions of the scene, and cannot dynamically adjust the fusion weights in low-light environments, leading to a decline in the quality of the fusion results. Most methods only focus on the visual quality or traditional image metrics of the fused image, neglecting its actual utility in downstream tasks, resulting in a fused image that, while clear, may lose important semantic information.
[0004] In summary, current infrared and visible light image fusion technologies still fall short in terms of feature fusion precision, illumination adaptability, and synergy with high-level vision tasks. Summary of the Invention
[0005] In view of the problems existing in the prior art, the present invention provides a modeling method and program product for image fusion model based on visible light and infrared, which at least partially solves the problem of insufficient synergy in the prior art.
[0006] In a first aspect, embodiments of this disclosure provide a modeling method for image fusion based on visible light and infrared light, including: Acquire the registered infrared and visible light images; Multi-level features were extracted from infrared and visible light images respectively; The extracted multi-level features are adaptively fused at different levels to obtain a fused feature set; Image fusion and semantic segmentation are performed on the fusion features in the fusion feature set, and the fused image and semantic segmentation results are obtained. A joint loss function is used to simultaneously optimize the quality of the fused image and the accuracy of semantic segmentation, thereby completing the model training and obtaining the trained image fusion model; The joint loss function is: , in, For pixel-level reconstruction loss, For gradient loss, For cross-entropy loss, , and These are the weighting coefficients; , in, To merge images Edges and textures Visible light image, This is an infrared image.
[0007] Optionally, the extraction of multi-level features from infrared and visible light images respectively includes: Infrared and visible light images are input into a fusion-segmentation joint model, which includes a shared backbone feature extraction network, a multi-scale wavelet fusion module, an image fusion task head, and a semantic segmentation task head. The backbone feature extraction network is used to extract multi-level features from infrared and visible light images.
[0008] Optionally, the multi-scale wavelet fusion module is used to adaptively fuse infrared features and visible light features at different resolution levels.
[0009] Optionally, the fused features can be simultaneously input into both the image fusion task head and the semantic segmentation task head, and the fused image and semantic segmentation results can be output respectively.
[0010] Optionally, the multi-scale wavelet fusion module includes multiple wavelet transform convolutional units, each of which performs the following operations: The two input features are subjected to L-level two-dimensional discrete wavelet transform to obtain low-frequency and high-frequency components. The two features include infrared features and visible light features. In the wavelet domain, the low-frequency components of the two features are weighted and fused using a spatially gated fusion block; The high-frequency components of the two features are convolved and then combined with the fused low-frequency components to undergo inverse wavelet transform, thereby reconstructing the spatial domain fused features. The reconstructed spatial domain fusion features are added to the residuals of the original features after basic convolution to obtain the fusion features at this level.
[0011] Optionally, the spatial gating fusion block can be implemented as follows: The low-frequency components of the two characteristics and By splicing along the channel dimension, combined features are obtained; The number of channels is generated through a two-layer convolutional attention network. Gating weights graph; The gating weight map is split along the channel dimension into... and They act on the low-frequency components respectively. and The output fusion features are , where ⊙ represents element-wise multiplication.
[0012] Optionally, the backbone feature extraction network adopts a ResNet18 structure, and the extracted features include five layers from the first to the fifth layer, with 64, 64, 128, 256 and 512 channels respectively.
[0013] Optionally, the image fusion task head adopts an autoencoder structure to reconstruct the fused image from the fusion features, and the semantic segmentation task head adopts a ResNextUNet structure.
[0014] Secondly, embodiments of this disclosure also provide a computer program product, including a computer program / instruction, which, when executed, implements the image fusion modeling method based on visible light and infrared as described in any of the first aspects.
[0015] This invention provides a modeling method and program product for image fusion based on visible light and infrared light. The modeling method utilizes a wavelet domain adaptive fusion mechanism to achieve refined integration of multi-scale features and introduces semantic segmentation as a high-level supervision signal to guide the fusion process in retaining information more valuable for visual understanding. This significantly enhances the visual quality of the fused image and its usability in downstream tasks, making it suitable for intelligent perception systems under complex lighting conditions. Ultimately, it aims to improve the precision of feature fusion, its lighting adaptability, and its synergy with high-level visual tasks. Attached Figure Description
[0016] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.
[0017] Figure 1 A flowchart illustrating the image fusion modeling method based on visible light and infrared provided in this disclosure embodiment; Figure 2 A schematic diagram of the fusion-segmentation joint model provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram of the structure of the multi-scale wavelet fusion module provided in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the image fusion task head and semantic segmentation task head structure provided in the embodiments of this disclosure. Detailed Implementation
[0018] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0019] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0020] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.
[0021] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0022] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0023] For ease of understanding, such as Figure 1 As shown, this embodiment discloses a modeling method for image fusion based on visible light and infrared light, including: Acquire the registered infrared and visible light images; Multi-level features were extracted from infrared and visible light images respectively; The extracted multi-level features are adaptively fused at different levels to obtain a fused feature set; Image fusion and semantic segmentation are performed on the fusion features in the fusion feature set, and the fused image and semantic segmentation results are obtained. A joint loss function is used to simultaneously optimize the quality of the fused image and the accuracy of semantic segmentation, thereby completing the model training and obtaining the trained image fusion model.
[0024] Optionally, the extraction of multi-level features from infrared and visible light images respectively includes: Infrared and visible light images are input into a fusion-segmentation joint model, which includes a shared backbone feature extraction network, a multi-scale wavelet fusion module, an image fusion task head, and a semantic segmentation task head. The backbone feature extraction network is used to extract multi-level features from infrared and visible light images.
[0025] Optionally, the multi-scale wavelet fusion module is used to adaptively fuse infrared features and visible light features at different resolution levels.
[0026] Optionally, the fused features can be simultaneously input into both the image fusion task head and the semantic segmentation task head, and the fused image and semantic segmentation results can be output respectively.
[0027] Optionally, the multi-scale wavelet fusion module includes multiple wavelet transform convolutional units, each of which performs the following operations: The two input features are subjected to L-level two-dimensional discrete wavelet transform to obtain low-frequency and high-frequency components. The two features include infrared features and visible light features. In the wavelet domain, the low-frequency components of the two features are weighted and fused using a spatially gated fusion block; The high-frequency components of the two features are convolved and then combined with the fused low-frequency components to undergo inverse wavelet transform, thereby reconstructing the spatial domain fused features. The reconstructed spatial domain fusion features are added to the residuals of the original features after basic convolution to obtain the fusion features at this level.
[0028] Optionally, the spatial gating fusion block can be implemented as follows: The low-frequency components of the two characteristics and By splicing along the channel dimension, combined features are obtained; The number of channels is generated through a two-layer convolutional attention network. Gating weights graph; The gating weight map is split along the channel dimension into... and They act on the low-frequency components respectively. and The output fusion features are , where ⊙ represents element-wise multiplication.
[0029] Optionally, the backbone feature extraction network adopts a ResNet18 structure, and the extracted features include five layers from the first to the fifth layer, with 64, 64, 128, 256 and 512 channels respectively.
[0030] Optionally, the image fusion task head adopts an autoencoder structure to reconstruct the fused image from the fusion features, and the semantic segmentation task head adopts a ResNextUNet structure.
[0031] Optionally, the joint loss function is: , in, For pixel-level reconstruction loss, For gradient loss, For cross-entropy loss, , and These are the weighting coefficients.
[0032] Specifically: such as Figure 2 As shown, the image fusion modeling method based on visible light and infrared light includes: S1. Obtain the registered infrared image. and visible light images ; S2. Input the infrared image and the visible light image into the fusion-segmentation joint model, which includes a shared backbone feature extraction network, a multi-scale wavelet fusion module, an image fusion task head, and a semantic segmentation task head; S3. After the backbone network extracts multi-level features, it adaptively fuses infrared and visible light features at different resolution levels through the multi-scale wavelet fusion module. S4. The fused features are simultaneously input into the image fusion task head and the semantic segmentation task head, and the fused image and semantic segmentation results are output respectively. S5. During the training phase, a joint loss function is used to simultaneously optimize and fuse image quality and semantic segmentation accuracy, achieving end-to-end collaborative learning.
[0033] In step S1, the public dataset MSRS is obtained. The entire dataset is divided into a training set containing 1083 pairs of images and a test set containing 361 pairs of images.
[0034] In step S2, feature extraction begins with analyzing the visible light image. and infrared images The input is fed into a two-stream ResNet18 backbone network. The network generates five layers of multi-scale feature maps for both modalities: and Subsequently, the network immediately enters the hierarchical feature fusion stage. We adopt a differentiated fusion strategy: shallow features are fused using a multi-scale wavelet fusion module, while deep features focus on the convergence and compression of global semantics. Fusion process This occurs at each feature layer, generating the final fused feature set: .
[0035] In step S3, the backbone extraction module uses a pre-trained ResNet18 as a shared encoder to process the input infrared image. and visible light images Five layers of features were extracted respectively, denoted as and This corresponds to different resolutions and channel numbers. The multi-scale wavelet fusion module introduces wavelet fusion units at different levels. Taking the first level as an example, the input is... and First, a three-layer two-dimensional discrete wavelet transform is performed on the two feature maps to obtain low-frequency (LL) and high-frequency (LH, HL, HH) components at each layer. Specifically, the two 64-channel feature maps are first subjected to a three-layer two-dimensional discrete wavelet transform, and the image is decomposed into low-frequency approximation components (LL) and high-frequency detail components (LH, HL, HH) using wavelet basis functions. After each wavelet decomposition, the low-frequency components of the two features are input into a spatially gated fusion block. This module concatenates the two features along the channel dimension, passes them through two convolutional layers and a sigmoid activation function to generate a set of learnable gated weights, and then splits them into two weight maps corresponding to visible light and infrared light. and To achieve weighted fusion, in each layer, the low-frequency component is fed into the spatially gated fusion block, such as... Figure 3 As shown, this module generates gating weights through a two-layer convolutional attention network. and To achieve adaptive weighting: , After independent convolution processing, the high-frequency components are reconstructed together with the fused low-frequency components using inverse wavelet transform. The final output is the sum of the residuals between the reconstructed features and the original features after basic convolution. Layers two through four employ a similar structure, with the fourth layer concatenating the two feature paths along the channel dimension before fusion. This mechanism dynamically adjusts the contribution ratio of the two modalities based on local image content, enhancing texture representation in well-lit conditions and highlighting thermal target information in low-light conditions, thereby improving the adaptability of the fusion process. The high-frequency components are enhanced separately through independent depthwise separable convolutions, and then reconstructed into spatial domain features together with the fused low-frequency components using inverse wavelet transform. The final output is the sum of the residuals between the reconstructed result and the original input after basic convolution, preserving key information from the original features.
[0036] The fusion process described above is performed sequentially from the first to the fourth layer. The second and fourth layers use the same structure. The fourth layer introduces a cross-attention mechanism before fusion to enhance cross-modal feature interaction. The features in the fifth layer are concatenated through channels and then reduced in dimensionality using 1×1 convolution to form a unified fused feature representation. .like Figure 4 As shown, the fused feature is simultaneously input to two independent task heads: one is an image fusion task head, which employs an autoencoder structure and is responsible for reconstructing a fused image from the fused feature that combines infrared target saliency with visible light texture clarity. Secondly, there is the semantic segmentation task header ResNextUNet.
[0037] In step S4, the image fusion task head adopts an autoencoder structure to reconstruct the fused image from the fusion features, and the semantic segmentation task head adopts a ResNextUNet structure.
[0038] In step S5, the joint loss function is defined as:
[0039] Based on the L1 norm, used to ensure fused images Edges and textures Able to inherit visible light and infrared The clearest structural information in the image: , Constrain the overall energy fidelity of the fused image. For image fusion tasks, a high-quality fusion result should inherit the most informative pixel value from the source image at each pixel location (i,j). In infrared-visible fusion, this means that bright areas in the infrared image must be preserved. Background brightness and color information in the visible image should also be fully considered. Therefore, the constraint objective is defined as the pixel-wise maximum value map. : , .
[0040] To achieve end-to-end collaborative optimization, this embodiment uses a joint loss function to guide model training. The total loss consists of three parts: pixel-level reconstruction loss. To ensure that the fused image inherits the main intensity information of the source image, the gradient loss is... To preserve edge and texture structure, as well as semantic segmentation loss. Cross-entropy is used to measure the difference between the segmentation result and the true label. The weights of the three loss terms are set to 20, 50, and 1, respectively, to ensure that the model optimizes fusion quality while also considering performance on higher-level tasks. The entire model is implemented on an NVIDIA GPU using the PyTorch framework. The training data uses the MSRS or M3FD public dataset, with an input image size of 480×640, a batch size of 2, and a total training duration of 100 epochs. The Adam optimizer is used, with an initial learning rate of 0.0001 that linearly decays to zero in the last 50 epochs. Automatic Mixed Precision (AMP) is enabled to accelerate convergence and save GPU memory.
[0041] Experiments show that, as illustrated in Tables 1, 2, 3, and 4, the proposed method in this embodiment exhibits excellent performance on multiple test sets. The generated fused images are visually clear and have moderate contrast, effectively highlighting infrared thermal targets while preserving rich details from visible light images, demonstrating good robustness, especially in complex low-light or nighttime scenarios. Furthermore, due to the supervised guidance of the semantic segmentation task, the fused features are more conducive to high-level visual understanding, resulting in segmentation accuracy superior to traditional two-stage methods. This technology can be widely applied in fields requiring multimodal perception, such as intelligent surveillance, autonomous driving, and drone navigation, possessing high practical value and promising prospects for widespread adoption.
[0042] Table 1. Comparative experimental results on the MSRS dataset
[0043] Table 2. Comparative experimental results on the TNO dataset
[0044] Table 3. Comparative experimental results on the LLVIP dataset.
[0045] Table 4. Comparative experimental results on the ROAD dataset
[0046] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.
[0047] The processor may be a central processing unit (CPU) or other processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the image fusion modeling method based on visible light and infrared in the foregoing embodiments of this disclosure.
[0048] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.
[0049] For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the image fusion modeling method based on visible light and infrared according to embodiments of this disclosure are performed.
[0050] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0051] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the image fusion modeling method based on visible light and infrared light according to the foregoing embodiments of the present disclosure are performed.
[0052] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).
[0053] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0054] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0055] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0056] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.
[0057] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.
[0058] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.
[0059] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0060] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A modeling method for image fusion based on visible light and infrared light, characterized in that, include: Acquire the registered infrared and visible light images; Multi-level features were extracted from infrared and visible light images respectively; The extracted multi-level features are adaptively fused at different levels to obtain a fused feature set; Image fusion and semantic segmentation are performed on the fusion features in the fusion feature set, and the fused image and semantic segmentation results are obtained. A joint loss function is used to simultaneously optimize the quality of the fused image and the accuracy of semantic segmentation, thereby completing the model training and obtaining the trained image fusion model; The joint loss function is: ， in, For pixel-level reconstruction loss, For gradient loss, For cross-entropy loss, , and These are the weighting coefficients; ， in, To merge images Edges and textures Visible light image, This is an infrared image.
2. The image fusion modeling method based on visible light and infrared light according to claim 1, characterized in that, The extraction of multi-level features from infrared and visible light images, respectively, includes: Infrared and visible light images are input into a fusion-segmentation joint model, which includes a shared backbone feature extraction network, a multi-scale wavelet fusion module, an image fusion task head, and a semantic segmentation task head. The backbone feature extraction network is used to extract multi-level features from infrared and visible light images.
3. The image fusion modeling method based on visible light and infrared light according to claim 2, characterized in that, The multi-scale wavelet fusion module is used to adaptively fuse infrared and visible light features at different resolution levels.
4. The image fusion modeling method based on visible light and infrared light according to claim 3, characterized in that, The fused features are simultaneously input into the image fusion task head and the semantic segmentation task head, and the fused image and semantic segmentation results are output respectively.
5. The image fusion modeling method based on visible light and infrared light according to claim 3, characterized in that, The multi-scale wavelet fusion module includes multiple wavelet transform convolutional units, each of which performs the following operations: The two input features are subjected to L-level two-dimensional discrete wavelet transform to obtain low-frequency and high-frequency components. The two features include infrared features and visible light features. In the wavelet domain, the low-frequency components of the two features are weighted and fused using a spatially gated fusion block; The high-frequency components of the two features are convolved and then combined with the fused low-frequency components to undergo inverse wavelet transform, thereby reconstructing the spatial domain fused features. The reconstructed spatial domain fusion features are added to the residuals of the original features after basic convolution to obtain the fusion features at this level.
6. The image fusion modeling method based on visible light and infrared light according to claim 5, characterized in that, The spatial gating fusion block is implemented as follows: The low-frequency components of the two characteristics and By splicing along the channel dimension, combined features are obtained; The number of channels is generated through a two-layer convolutional attention network. Gating weights graph; The gating weight map is split along the channel dimension into... and They act on the low-frequency components respectively. and The output fusion features are , where ⊙ represents element-wise multiplication.
7. The image fusion modeling method based on visible light and infrared light according to claim 2, characterized in that, The backbone feature extraction network adopts the ResNet18 structure, and the extracted features include five layers from the first to the fifth layer, with the number of channels being 64, 64, 128, 256 and 512 respectively.
8. The image fusion modeling method based on visible light and infrared light according to claim 2, characterized in that, The image fusion task head adopts an autoencoder structure to reconstruct the fused image from the fusion features, and the semantic segmentation task head adopts a ResNextUNet structure.
9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed, it implements the image fusion modeling method based on visible light and infrared as described in any one of claims 1-8.