An image fusion method based on random channel enhancement and multi-scale fusion

Abstract

The application discloses an image fusion method based on random channel enhancement and multi-scale fusion, and relates to the technical field of image processing, which comprises the following steps: duplicating infrared and visible light images respectively, randomly selecting part of the copies, applying channel-level disturbance to the copies, splicing along the channel dimension to generate enhanced input; using a double-branch U-Net to encode the double-mode images after enhancement, extracting multi-scale features by parallel connection of different expansion rates of hollow convolution in each encoder, and adaptively weighting and fusing the features, splicing and compressing the features of the same scale double-branch; the decoder is progressively up-sampled and combined with a jump connection to reconstruct a fused image; and an intensity and gradient joint loss function is used for end-to-end training. Through the synergistic mechanism of random channel enhancement and multi-scale hollow convolution, the application effectively improves the texture detail retention capability of the fused image in a complex scene and the saliency of a thermal target, and simultaneously enhances the robustness of the model to imaging noise and degradation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image fusion method based on random channel enhancement and multi-scale fusion. Background Technology

[0002] In the context of today's rapid development of digitalization and intelligentization, computer vision technology has become a core driving force for innovation in fields such as remote sensing, security, autonomous driving, and intelligent manufacturing. Among them, multimodal image fusion, as a key link connecting perception and understanding, aims to complement and eliminate redundancy in sensor data from different imaging mechanisms (such as infrared, visible light, millimeter wave, and LiDAR) to generate a unified representation with high confidence, high robustness, and high interpretability, providing high-quality input for subsequent high-level vision tasks (such as object detection, semantic segmentation, and behavior recognition).

[0003] Infrared and Visible Image Fusion (IVIF) is one of the most representative branches of multimodal fusion. Existing fusion methods are mainly divided into traditional methods and deep learning-based methods. Traditional methods, such as weighted average and Laplacian pyramid, rely on manually designed features and are difficult to cope with complex scenes. Convolutional Neural Network (CNN)-based methods have significantly improved the fusion quality through the encoding-fusion-decoding structure, but still have the following problems: (1) Standard convolution operations are difficult to effectively model the global context, resulting in blurred edges and loss of details of large-scale hot targets. (2) Existing methods mostly use fixed convolution kernels or simple channel splicing, which cannot adaptively adjust the feature extraction range according to modal differences, and are prone to excessive texture smoothing or thermal radiation diffusion. (3) In real environments, there are often degradations such as thermal noise, gain drift, and blurring, and existing training strategies do not model these, resulting in insufficient generalization ability of the model under cross-scene and cross-sensor conditions.

[0004] Therefore, how to achieve large receptive field feature extraction, suppress noise channels, and improve robustness to various degradations within a lightweight framework is a key problem that urgently needs to be solved in the current field of fusion. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes an image fusion method based on random channel enhancement and multi-scale fusion. This method enables the fused image to retain infrared thermal radiation information while significantly enhancing the ability to restore visible light texture details, effectively suppressing the problems of thermal target blurring and texture loss.

[0006] This invention provides an image fusion method based on random channel enhancement and multi-scale fusion, comprising the following steps: The infrared and visible light images to be fused are obtained. The input infrared and visible light images are copied several times. Channel-level perturbation is applied to a portion of the copies. All copies are then stitched together along the channel dimension to generate the enhanced input tensor. A dual-branch U-Net network is used to encode the enhanced infrared and visible light images respectively. In each encoder, multiple dilated convolutions with different dilation rates are connected in parallel to extract multi-scale features and adaptively weighted and fused. The output feature maps of the two branches at the same scale are concatenated and compressed to obtain the fused features. The fused features at each scale are input into the U-Net decoder, upsampled step by step to restore resolution, and combined with skip connections to preserve details, outputting a fused image; The dual-branch U-Net network is trained end-to-end using a joint loss function that includes strength constraints and gradient constraints to optimize network parameters.

[0007] In this scheme, the input infrared image and visible light image are copied several times, a portion of the copies are randomly selected and subjected to channel-level perturbations, and then all copies are stitched together along the channel dimension to generate the enhanced input tensor, specifically including: The input single-channel infrared or visible light image is copied N times to obtain N identical image copies. All copies are then randomly divided into a hold group, a noise perturbation group, and a blur perturbation group according to a preset ratio. For each copy in the noise disturbance group, Gaussian noise is applied according to a preset noise intensity scheduling strategy, wherein different copies adopt different levels of noise variance, generating a multi-level noise degradation view from mild to severe. For each copy in the fuzzy perturbation group, a Gaussian fuzz kernel is applied according to a preset fuzz intensity scheduling strategy, wherein different copies use fuzz kernels of different sizes to generate a multi-level fuzzy degradation view that ranges from local detail preservation to global contour smoothing. Two replicas are randomly selected from the holding group, and a channel-level feature exchange operation is performed to generate a hybrid perturbation replica that includes cross-replica information exchange. After concatenating all N copies along the channel dimension, a lightweight channel attention module is introduced to adaptively weight the concatenated multi-channel tensor. The multi-channel tensor after channel attention weighting is used as the final enhancement input of the current modality image.

[0008] In this scheme, a dual-branch U-Net network is used to encode the enhanced infrared image and the visible light image separately, specifically including: An asymmetric dual-branch U-Net encoder comprising an infrared branch and a visible light branch is constructed. The infrared branch and the visible light branch learn parameters independently without weight sharing. Each encoder level includes a multi-scale dilated convolution module to extract multi-level receptive field features. The outputs of the three branches are fused through a content-aware adaptive weighting mechanism to obtain an enhanced feature map. After each encoder output, the infrared feature map and the visible light feature map are mapped to the common feature space through the projection head, and the mutual information estimator calculates the lower bound of the mutual information between the projected features. The maximum mutual information is used as a constraint to force the distribution of infrared and visible light modes to be closer in the feature space. A lightweight reconstruction head is connected to the output of each stage of the infrared encoder and the visible light encoder to reconstruct the original input image. The multi-scale reconstruction loss of the infrared branch and the visible light branch is calculated separately as an auxiliary supervision signal to guide each branch encoder to retain modality-specific information. At the same scale, the infrared feature map and the visible light feature map after feature alignment are stitched together along the channel dimension, and then compressed and fused by one-dimensional convolution to generate a fused feature map.

[0009] In this scheme, the asymmetric dual-branch coding includes a dynamic routing mechanism. After each encoder output, based on the global correlation coefficient between the infrared and visible light feature maps, it dynamically determines whether the current stage should perform cross-modal fusion and the propagation path of the fusion result. Specifically, this includes: The infrared feature map and the visible light feature map are compressed into vectors by global average pooling, and the cosine similarity between the two vectors is calculated to generate a correlation score. A first threshold and a second threshold are preset as learnable parameters of the network, which are adaptively adjusted during training. A fusion strategy is selected based on the comparison between the relevance score and the threshold. When the correlation score is greater than or equal to the first threshold, a deep fusion strategy is adopted to stitch the infrared feature map and the visible light feature map together and then fuse them through one-dimensional convolution. The fusion result is simultaneously transmitted to the decoder and the next level encoder. When the correlation score is between the first threshold and the second threshold, a selective fusion strategy is adopted, and only the fusion result is passed to the decoder, while the two branches continue to encode independently. When the correlation coefficient is less than or equal to the second threshold, a delayed fusion strategy is adopted, and the infrared and visible light branches are encoded completely independently. Each branch passes its feature map to the next level encoder and waits for the correlation of subsequent levels to improve before making a decision. Perform the corresponding feature transfer and fusion operations according to the selected fusion strategy.

[0010] In this scheme, the fused features at each scale are input into the U-Net decoder, upsampled step by step to restore resolution, and combined with skip connections to preserve details, outputting a fused image, specifically including: A three-level U-Net decoder symmetrical to the encoder is constructed, and the multi-scale fused feature maps output by the fusion module are input into each level of the decoder in order of resolution from low to high. In each level of the decoder, the input feature map is upsampled and then interacted with the feature map of the corresponding level of the encoder through bidirectional skip connections. The fused feature map is then refined in both spatial and channel dimensions through a progressive feature refinement module, and the refined feature map is further refined through a multi-scale dilated convolution module. After each level of decoder output, a lightweight reconstruction head is connected to generate an intermediate fused image of the corresponding scale. The semantic consistency loss between intermediate fused images of adjacent scales is calculated. The semantic consistency loss is based on a weighted combination of structural similarity index and L1 difference, so that the reconstruction results of different scales maintain semantic alignment. The feature map output from the first-level decoder is mapped to a single channel through a one-dimensional convolutional layer, and then the final fused image is generated through an activation function.

[0011] In this scheme, the bidirectional skip connection includes forward feature transfer and backward error feedback, specifically including: The forward feature transfer concatenates the feature map output by the encoder at the corresponding level with the feature map upsampled at the current level of the decoder along the channel dimension, and then compresses and fuses them through one-dimensional convolution to output the forward fused features. The reverse error feedback upsamples the feature map of the current level of the decoder to the resolution of the corresponding level of the encoder through the deconvolution layer, and calculates the difference map between the upsampled decoder feature map and the encoder feature map. The difference map is converted into feedback weights through the gating module. The element-wise product of the feedback weights and the difference map is then superimposed onto the encoder feature map to update the encoder feature map. The forward fusion features are used as the output feature map of the current level of the decoder and passed to the next operation unit. The updated encoder feature map is used as the current feature representation of the nth level of the encoder for subsequent forward skip connection calculations.

[0012] In this scheme, a joint loss function including strength constraints and gradient constraints is used to perform end-to-end training of the dual-branch U-Net network, specifically including: Local saliency is calculated based on the gradient magnitude in the neighborhood of each pixel location. Local saliency maps of infrared and visible light images are obtained. Based on the ratio of local saliency of infrared and visible light, adaptive weights are generated for each pixel location to construct an adaptive reference image. Calculate the L1 distance between the fused image and the adaptive reference image to obtain the adaptive weighted intensity loss. : , in To merge images at pixel locations The intensity value, To adapt the reference image at pixel position The intensity value, Total number of pixels; Calculate the gradient magnitude maps and gradient direction angles of the fused image, infrared image, and visible light image respectively, construct a gradient magnitude reference map, and calculate the gradient magnitude loss. : , in To merge images at pixel locations gradient magnitude, For gradient magnitude reference image at pixel position The gradient magnitude; Based on the ratio of local saliency of gradient directions in infrared and visible light, an adaptive reference map of gradient directions is constructed, and the gradient direction loss is calculated. ; The gradient magnitude loss and gradient direction loss are weighted and combined to obtain the direction-aware gradient loss. ,in This is the balance weighting coefficient for directional loss.

[0013] In this scheme, the joint loss function also includes auxiliary losses, specifically including: Infrared and visible light feature maps are mapped to a common feature space using a projection head. The lower bound of the mutual information between the two projected features is calculated, and negative mutual information is used as the mutual information maximization loss. ; A lightweight reconstruction head is connected to the output of each stage of the infrared encoder and the visible light encoder to reconstruct the original input image. The L1 loss between the reconstructed image and the original image is calculated as the modality-aware reconstruction loss. ; After each stage output of the decoder, an intermediate fused image of the corresponding scale is generated. The difference between intermediate fused images of adjacent scales is calculated as the multi-scale consistency loss. ; The joint loss function is expressed as: , in, for At least one of them, These are the corresponding balance weight coefficients.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides the model with a multi-perturbation view through a random channel enhancement strategy. Combined with a multi-scale dilated convolution module, it extracts multi-level receptive field features from local to global perspectives, effectively overcoming the problems of thermal target blurring and texture loss in traditional methods. This allows the fused image to simultaneously retain infrared thermal radiation information and visible light texture details. Random channel enhancement simulates various channel-level degradations such as Gaussian noise, blur kernels, and composite perturbations, exposing the model to diverse imaging interferences during training. This ensures stable fusion performance under harsh conditions such as dust and low light, significantly improving cross-scene generalization ability. This invention performs channel enhancement at the input end, eliminating the need for additional network generation or post-processing modules, keeping the parameter count the same as the baseline. Simultaneously, it employs dilated convolution and a 1×1 compression strategy to expand the receptive field while controlling computational load, and eliminates the need for paired ground truth fusion images, significantly reducing data and training costs.

[0015] By combining intensity and gradient loss, the fused image inherits the superior information of the dual-source images in both the pixel and gradient domains, providing high-quality input for subsequent high-level tasks such as object detection and semantic segmentation. This invention can be deployed in scenarios such as power line inspection, hazardous chemical monitoring, forest fire prevention, and autonomous driving. It maintains clear fusion results even in blind areas such as dense fog and nighttime, improving the efficiency of abnormal event identification, reducing the rate of missed accident reports, and providing technical support for the intelligent transformation of many high-risk industries. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or examples of the present invention, the drawings used in the embodiments or examples will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained according to these drawings without creative effort.

[0017] Figure 1 A flowchart of an image fusion method based on random channel enhancement and multi-scale fusion is shown; Figure 2 The flowchart illustrates the random channel enhancement processing performed on the input infrared and visible light images respectively; Figure 3 A flowchart illustrating the dynamic decision-making process for cross-modal fusion and the propagation path of the fusion results is shown. Figure 4 A schematic diagram of the fusion effect on the MSRS dataset is shown. Detailed Implementation

[0018] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0020] like Figure 1 As shown, this embodiment provides an image fusion method based on random channel enhancement and multi-scale fusion, including: The infrared and visible light images to be fused are obtained. The input infrared and visible light images are copied several times. Channel-level perturbation is applied to a portion of the copies. All copies are then stitched together along the channel dimension to generate the enhanced input tensor. A dual-branch U-Net network is used to encode the enhanced infrared and visible light images respectively. In each encoder, multiple dilated convolutions with different dilation rates are connected in parallel to extract multi-scale features and adaptively weighted and fused. The output feature maps of the two branches at the same scale are concatenated and compressed to obtain the fused features. The fused features at each scale are input into the U-Net decoder, upsampled step by step to restore resolution, and combined with skip connections to preserve details, outputting a fused image; The dual-branch U-Net network is trained end-to-end using a joint loss function that includes strength constraints and gradient constraints to optimize network parameters.

[0021] It should be noted that the infrared and visible light images mentioned can come from a variety of practical application scenarios, including but not limited to: security monitoring scenarios (such as nighttime perimeter protection, airport and train station security checks), intelligent driving and assisted driving scenarios (such as autonomous vehicles, mining engineering vehicles), power and industrial inspection scenarios (such as transmission line inspection, substation online monitoring), fire rescue and emergency search and rescue scenarios (such as fire rescue, forest search and rescue), medical imaging scenarios (such as infrared thermal imaging-assisted diagnosis), and agricultural and plant protection scenarios (such as farmland remote sensing monitoring), etc.

[0022] like Figure 2As shown, the input single-channel infrared or visible light image is copied N times to obtain N identical image copies. All copies are randomly divided into a hold group, a noise perturbation group, and a blur perturbation group according to a preset ratio. Preferably, the ratio of the number of copies in each group is 4:3:3. For each copy in the noise perturbation group, Gaussian noise is applied according to a preset noise intensity scheduling strategy, where different copies use different levels of noise variance, generating a multi-level noise degradation view from mild to severe. For each copy in the blur perturbation group, a Gaussian blur kernel is applied according to a preset blur intensity scheduling strategy, where different copies use blur kernels of different sizes, generating a multi-level blur degradation view from local detail preservation to global contour smoothing. Randomly... Two copies are selected, and a channel-level feature exchange operation is performed to generate a hybrid perturbation copy containing cross-copy information interaction. This involves randomly mixing some channel pixel values ​​from one copy with the corresponding channel pixel values ​​from the other copy to simulate the compounding effect of multi-source interference in real imaging. All N copies are then stitched together along the channel dimension, and a lightweight channel attention module is introduced to adaptively weight the stitched multi-channel tensor. This channel attention module learns an importance weight for each channel through global average pooling and a two-layer fully connected network, enabling the network to automatically focus on channels containing effective degradation information and suppress redundant or invalid perturbation channels. The multi-channel tensor after channel attention weighting is used as the final enhancement input for the current modality image.

[0023] It should be noted that a dual-branch U-Net network is used to encode the enhanced infrared and visible light images separately. An asymmetric dual-branch U-Net encoder is constructed, comprising an infrared branch and a visible light branch. The infrared and visible light branches learn parameters independently without weight sharing. Each encoder stage includes a multi-scale dilated convolution module to extract multi-level receptive field features. Preferably, the dilation rates of the three branches are set to 1, 2, and 3, respectively, and the kernel size of each branch is set to 3×3. The padding value is adjusted accordingly based on the dilation rate to ensure that the spatial size of the output feature map is consistent with the input. Each branch independently performs a depthwise separable dilated convolution operation, outputting a feature map of the same size as the input.

[0024] The enhanced feature map is obtained by fusing the outputs of the three branches through a content-aware adaptive weighting mechanism. Specifically, the content-aware adaptive weighting mechanism includes: compressing the input feature map into a global description vector of length C using global average pooling, representing the overall activation intensity of each channel. This global description vector is then sequentially fed into two fully connected layers: the first layer is used for dimensionality reduction and compression, and the second layer is used to restore the dimensionality and output the original weight scores of the three scale branches. Subsequently, the three scores are normalized using the Softmax function to obtain three normalized weight coefficients. The output feature maps of the three branches are then weighted and summed according to the learned weight coefficients. Preferably, after adaptive weighted fusion, a lightweight channel attention module is further introduced to recalibrate the channels of the fused feature map. This module generates a scaling factor between 0 and 1 for each channel through global average pooling, two fully connected layers, and a sigmoid activation function, suppressing irrelevant or noisy channels and enhancing key semantic channels.

[0025] After each encoder output, the infrared feature map and the visible light feature map are mapped to the common feature space through the projection head, and the mutual information estimator calculates the lower bound of the mutual information between the projected features. Maximizing the mutual information is used as a constraint to force the distribution of infrared and visible light modes to be closer in the feature space, thereby alleviating the feature mismatch problem between modes.

[0026] A lightweight reconstruction head is connected to the output of each stage of the infrared encoder and the visible light encoder to reconstruct the original input image. The multi-scale reconstruction loss of the infrared branch and the visible light branch is calculated separately as an auxiliary supervision signal to guide each branch encoder to retain modality-specific information. The weights of the reconstruction loss at different scales are allocated differently according to the spatial resolution of the features at that level. The shallow encoder focuses on detail reconstruction and has a higher weight, while the deep encoder focuses on semantic reconstruction and has a lower weight. At the same scale, the feature-aligned infrared feature map and the visible light feature map are concatenated along the channel dimension and then compressed and fused by one-dimensional convolution to generate a fused feature map.

[0027] Asymmetric dual-branch coding includes a dynamic routing mechanism. After each encoder output, based on the global correlation coefficient between the infrared and visible light feature maps, it dynamically determines whether the current stage should perform cross-modal fusion and the propagation path of the fusion result. For example... Figure 3 As shown, the infrared feature map and the visible light feature map are compressed into vectors by global average pooling, and the cosine similarity between the two vectors is calculated to generate a correlation score. ; Preset first threshold Second threshold As a learnable parameter of the network, it is adaptively adjusted during training, and a fusion strategy is selected based on the comparison between the relevance score and the threshold. when At that time, a deep fusion strategy is adopted, which concatenates the infrared feature map and the visible light feature map and then fuses them through one-dimensional convolution. The fusion result is simultaneously transmitted to the decoder and the next level encoder. when At this time, a selective fusion strategy is adopted, which only passes the fusion result to the decoder, and the two branches continue to encode independently; when At that time, a delayed fusion strategy is adopted, with the infrared and visible light branches being encoded completely independently. Each branch passes its feature map to the next level encoder and waits for the correlation of subsequent levels to improve before making a decision. Perform corresponding feature transfer and fusion operations according to the selected fusion strategy to ensure that feature pollution caused by forced fusion is avoided when modal correlation is low, and to make full use of cross-modal complementary information when modal correlation is high.

[0028] It should be noted that the fused features at each scale are input into the U-Net decoder, and the resolution is restored by upsampling at each level. Skip connections are then used to preserve details, and the fused image is output. The decoder adopts a three-level structure symmetrical to the encoder. Each level consists of an upsampling layer, a skip connection fusion module, a progressive feature refinement module, and a multi-scale dilated convolution module.

[0029] A three-level U-Net decoder symmetrical to the encoder is constructed. The multi-scale fused feature maps output from the fusion module are input into each level of the decoder in ascending order of resolution. In each level of the decoder, the input feature maps are upsampled and then interactively fused with the feature maps of the corresponding encoder levels via bidirectional skip connections. These bidirectional skip connections include forward feature propagation and backward error feedback. Forward feature propagation concatenates the feature map output from the corresponding encoder level with the upsampled feature map of the current decoder level along the channel dimension, and then performs compression and fusion through one-dimensional convolution to output the forward fused features. This allows the decoder to directly access the high-resolution detail information retained by the encoder, compensating for the spatial information lost during upsampling. Backward error feedback upsamples the feature map of the current decoder level to the resolution of the corresponding encoder level through a deconvolution layer, calculating the difference map between the upsampled decoder feature map and the encoder feature map. The difference map reflects the deviation between the decoder reconstruction result and the original encoder features. A gating module converts the difference map into feedback weights, and the element-wise product of the feedback weights and the difference map is superimposed onto the encoder feature map to update it. ,in, For encoder number Level feature map, For the difference graph, The feedback weights output by the gating module, The learnable coefficients are used as the forward fusion features, which are then passed to the next operation unit as the output feature map of the current level of the decoder. The updated encoder feature map is then used as the current feature representation of the nth level of the encoder for subsequent forward skip connection calculations.

[0030] The fused feature map is refined in both spatial and channel dimensions using a progressive feature refinement module, and further refined using a multi-scale dilated convolution module. Specifically, the progressive feature refinement module includes: a spatial attention submodule that performs global max pooling and global average pooling on the input feature map in the channel dimension; concatenating the two pooling results and then performing a 7×7 convolution followed by sigmoid activation to generate spatial attention weights; multiplying these spatial attention weights element-wise with the original feature map to obtain spatially enhanced features; and a channel attention submodule that performs global average pooling to compress the spatially enhanced features into channel description vectors; performing a two-layer fully connected network followed by sigmoid activation to generate channel attention weights; multiplying these channel attention weights channel-wise with the spatially enhanced features to obtain channel-enhanced features; and finally, a weighted sum of the original input feature map and the channel-enhanced features is taken as the module output. After each level of decoder output, a lightweight reconstruction head is connected to reconstruct the feature map at the current scale into an intermediate fused image of the same size as the original image. The semantic consistency loss between intermediate fused images at adjacent scales is calculated. The semantic consistency loss is based on a weighted combination of structural similarity index and L1 difference, so that the reconstruction results at different scales maintain semantic alignment. The feature map output by the first level decoder is mapped to a single channel through a one-dimensional convolutional layer and then activated by an activation function to generate the final fused image.

[0031] It should be noted that the dual-branch U-Net network is trained end-to-end using a joint loss function that includes both strength and gradient constraints. During the training phase, the Adam optimizer combined with cosine annealing scheduling is employed. A locally saliency-aware adaptive weighting mechanism is introduced, so that the reference image is no longer a fixed point-by-point maximum value, but is dynamically adjusted based on the content characteristics of local regions.

[0032] Local saliency is calculated based on the gradient magnitude in the neighborhood of each pixel location. Local saliency maps of infrared and visible light images are obtained. Based on the ratio of local saliency of infrared and visible light, adaptive weights are generated for each pixel location to construct an adaptive reference image. , in , For infrared images at pixel locations The local significance ratio, For visible light images at pixel locations The local significance ratio, It is a very small constant; Calculate the L1 distance between the fused image and the adaptive reference image to obtain the adaptive weighted intensity loss. : , in To merge images at pixel locations The intensity value, To adapt the reference image at pixel position The intensity value, Total number of pixels; The gradient magnitude maps and gradient direction angles of the fused image, infrared image, and visible light image are calculated separately. Using the same adaptive weighting strategy as the intensity loss, a gradient magnitude reference map is constructed, and the gradient magnitude loss is calculated. : , in To merge images at pixel locations gradient magnitude, For gradient magnitude reference image at pixel position The gradient magnitude; Using the same adaptive weighting strategy, an adaptive reference map of gradient directions is constructed based on the ratio of local saliency of gradient directions in infrared and visible light, and the gradient direction loss is calculated. ; , in To merge images at pixel locations gradient direction, For gradient direction adaptive reference map at pixel position The gradient direction; The gradient magnitude loss and gradient direction loss are weighted and combined to obtain the direction-aware gradient loss. ,in This is the balance weighting coefficient for directional loss.

[0033] It should be noted that the joint loss function also includes auxiliary losses, specifically including: Infrared and visible light feature maps are mapped to a common feature space using a projection head. A mutual information estimator calculates the lower bound of the mutual information between the two projected features based on the projected features, using negative mutual information as the mutual information maximization loss. The gradient backpropagation is used to align the features of the two modalities as much as possible in the common space.

[0034] A lightweight reconstruction head is connected to the output of each stage of the infrared encoder and the visible light encoder to reconstruct the original input image. The L1 loss between the reconstructed image and the original image is calculated as the modality-aware reconstruction loss. The weights of the reconstruction loss at different scales are allocated differently according to the spatial resolution of the features at that level. The reconstruction weights of the shallow encoder are higher, and the reconstruction weights of the deep encoder are lower.

[0035] After each stage output of the decoder, an intermediate fused image of the corresponding scale is generated. The difference between intermediate fused images of adjacent scales is calculated as the multi-scale consistency loss. ; , in , For the first Level, No. The intermediate fused image output by the multi-stage decoder, It is a structural similarity index. For downsampling operation, These are the balancing weighting coefficients.

[0036] The joint loss function is expressed as: , in, for At least one of them, These are the corresponding balance weight coefficients.

[0037] like Figure 4 As shown, to comprehensively evaluate the fusion quality, six no-reference metrics (EN, SD, SF, AG, SCD, and VIF) were selected and tested on the MSRS test set. During testing, infrared and visible light images were directly fed into the trained model without post-processing or scene fine-tuning. Quantitative results show that our approach achieved good results on several key metrics, with sharper textures and higher contrast for thermal targets at the visual level, verifying the effectiveness and generalization ability of the CDA self-enhancement and multi-scale U-Net collaborative mechanism under complex scene conditions.

[0038] The second embodiment of the present invention provides a computer-readable storage medium, which includes a program for an image fusion method based on random channel enhancement and multi-scale fusion. When the program for the image fusion method based on random channel enhancement and multi-scale fusion is executed by a processor, it implements the steps of the image fusion method based on random channel enhancement and multi-scale fusion.

[0039] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0040] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An image fusion method based on random channel enhancement and multi-scale fusion, characterized in that, Includes the following steps: The infrared and visible light images to be fused are obtained. The input infrared and visible light images are copied several times. Channel-level perturbation is applied to a portion of the copies. All copies are then stitched together along the channel dimension to generate the enhanced input tensor. A dual-branch U-Net network is used to encode the enhanced infrared and visible light images respectively. In each encoder, multiple dilated convolutions with different dilation rates are connected in parallel to extract multi-scale features and adaptively weighted and fused. The output feature maps of the two branches at the same scale are concatenated and compressed to obtain the fused features. The fused features at each scale are input into the U-Net decoder, upsampled step by step to restore resolution, and combined with skip connections to preserve details, outputting a fused image; The dual-branch U-Net network is trained end-to-end using a joint loss function that includes strength constraints and gradient constraints to optimize network parameters.

2. The image fusion method based on random channel enhancement and multi-scale fusion according to claim 1, characterized in that, The input infrared and visible light images are copied several times. Channel-level perturbations are applied to a subset of these copies. All copies are then concatenated along the channel dimension to generate the enhanced input tensor, specifically including: The input single-channel infrared or visible light image is copied N times to obtain N identical image copies. All copies are then randomly divided into a hold group, a noise perturbation group, and a blur perturbation group according to a preset ratio. For each copy in the noise disturbance group, Gaussian noise is applied according to a preset noise intensity scheduling strategy, wherein different copies adopt different levels of noise variance, generating a multi-level noise degradation view from mild to severe. For each copy in the fuzzy perturbation group, a Gaussian fuzz kernel is applied according to a preset fuzz intensity scheduling strategy, wherein different copies use fuzz kernels of different sizes to generate a multi-level fuzzy degradation view that ranges from local detail preservation to global contour smoothing. Two replicas are randomly selected from the holding group, and a channel-level feature exchange operation is performed to generate a hybrid perturbation replica that includes cross-replica information exchange. After concatenating all N copies along the channel dimension, a lightweight channel attention module is introduced to adaptively weight the concatenated multi-channel tensor. The multi-channel tensor after channel attention weighting is used as the final enhancement input of the current modality image.

3. The image fusion method based on random channel enhancement and multi-scale fusion according to claim 1, characterized in that, A dual-branch U-Net network is used to encode the enhanced infrared and visible light images separately, specifically including: An asymmetric dual-branch U-Net encoder comprising an infrared branch and a visible light branch is constructed. The infrared branch and the visible light branch learn parameters independently without weight sharing. Each encoder level includes a multi-scale dilated convolution module to extract multi-level receptive field features. The outputs of the three branches are fused through a content-aware adaptive weighting mechanism to obtain an enhanced feature map. After each encoder output, the infrared feature map and the visible light feature map are mapped to the common feature space through the projection head, and the mutual information estimator calculates the lower bound of the mutual information between the projected features. The maximum mutual information is used as a constraint to force the distribution of infrared and visible light modes to be closer in the feature space. A lightweight reconstruction head is connected to the output of each stage of the infrared encoder and the visible light encoder to reconstruct the original input image. The multi-scale reconstruction loss of the infrared branch and the visible light branch is calculated separately as an auxiliary supervision signal to guide each branch encoder to retain modality-specific information. At the same scale, the infrared feature map and the visible light feature map after feature alignment are stitched together along the channel dimension, and then compressed and fused by one-dimensional convolution to generate a fused feature map.

4. The image fusion method based on random channel enhancement and multi-scale fusion according to claim 3, characterized in that, Asymmetric dual-branch coding includes a dynamic routing mechanism. After each encoder output, based on the global correlation coefficient between the infrared and visible light feature maps, it dynamically determines whether to perform cross-modal fusion at the current stage and the propagation path of the fusion result. Specifically, this includes: The infrared feature map and the visible light feature map are compressed into vectors by global average pooling, and the cosine similarity between the two vectors is calculated to generate a correlation score. A first threshold and a second threshold are preset as learnable parameters of the network, which are adaptively adjusted during training. A fusion strategy is selected based on the comparison between the relevance score and the threshold. When the correlation score is greater than or equal to the first threshold, a deep fusion strategy is adopted to stitch the infrared feature map and the visible light feature map together and then fuse them through one-dimensional convolution. The fusion result is simultaneously transmitted to the decoder and the next level encoder. When the correlation score is between the first threshold and the second threshold, a selective fusion strategy is adopted, and only the fusion result is passed to the decoder, while the two branches continue to encode independently. When the correlation coefficient is less than or equal to the second threshold, a delayed fusion strategy is adopted, and the infrared and visible light branches are encoded completely independently. Each branch passes its feature map to the next level encoder and waits for the correlation of subsequent levels to improve before making a decision. Perform the corresponding feature transfer and fusion operations according to the selected fusion strategy.

5. The image fusion method based on random channel enhancement and multi-scale fusion according to claim 1, characterized in that, The fused features at each scale are input into the U-Net decoder, upsampled stepwise to restore resolution, and combined with skip connections to preserve details, outputting a fused image, specifically including: A three-level U-Net decoder symmetrical to the encoder is constructed, and the multi-scale fused feature maps output by the fusion module are input into each level of the decoder in order of resolution from low to high. In each level of the decoder, the input feature map is upsampled and then interacted with the feature map of the corresponding level of the encoder through bidirectional skip connections. The fused feature map is then refined in both spatial and channel dimensions through a progressive feature refinement module, and the refined feature map is further refined through a multi-scale dilated convolution module. After each level of decoder output, a lightweight reconstruction head is connected to generate an intermediate fused image of the corresponding scale. The semantic consistency loss between intermediate fused images of adjacent scales is calculated. The semantic consistency loss is based on a weighted combination of structural similarity index and L1 difference, so that the reconstruction results of different scales maintain semantic alignment. The feature map output from the first-level decoder is mapped to a single channel through a one-dimensional convolutional layer, and then the final fused image is generated through an activation function.

6. The image fusion method based on random channel enhancement and multi-scale fusion according to claim 5, characterized in that, The bidirectional skip connection includes forward feature propagation and backward error feedback, specifically including: The forward feature transfer concatenates the feature map output by the encoder at the corresponding level with the feature map upsampled at the current level of the decoder along the channel dimension, and then compresses and fuses them through one-dimensional convolution to output the forward fused features. The reverse error feedback upsamples the feature map of the current level of the decoder to the resolution of the corresponding level of the encoder through the deconvolution layer, and calculates the difference map between the upsampled decoder feature map and the encoder feature map. The difference map is converted into feedback weights through the gating module. The element-wise product of the feedback weights and the difference map is then superimposed onto the encoder feature map to update the encoder feature map. The forward fusion features are used as the output feature map of the current level of the decoder and passed to the next operation unit. The updated encoder feature map is used as the current feature representation of the nth level of the encoder for subsequent forward skip connection calculations.

7. The image fusion method based on random channel enhancement and multi-scale fusion according to claim 1, characterized in that, The dual-branch U-Net network is trained end-to-end using a joint loss function that includes strength constraints and gradient constraints, specifically including: Local saliency is calculated based on the gradient magnitude in the neighborhood of each pixel location. Local saliency maps of infrared and visible light images are obtained. Based on the ratio of local saliency of infrared and visible light, adaptive weights are generated for each pixel location to construct an adaptive reference image. Calculate the L1 distance between the fused image and the adaptive reference image to obtain the adaptive weighted intensity loss. : ， in To merge images at pixel locations The intensity value is used to adapt the reference image at the pixel location. The intensity value, Total number of pixels; Calculate the gradient magnitude maps and gradient direction angles of the fused image, infrared image, and visible light image respectively, construct a gradient magnitude reference map, and calculate the gradient magnitude loss. : ， in To merge images at pixel locations gradient magnitude, For gradient magnitude reference image at pixel position The gradient magnitude; Based on the ratio of local saliency of gradient directions in infrared and visible light, an adaptive reference map of gradient directions is constructed, and the gradient direction loss is calculated. ; The gradient magnitude loss and gradient direction loss are weighted and combined to obtain the direction-aware gradient loss. ,in This is the balance weighting coefficient for directional loss.

8. The image fusion method based on random channel enhancement and multi-scale fusion according to claim 7, characterized in that, The joint loss function also includes auxiliary losses, specifically: Infrared and visible light feature maps are mapped to a common feature space using a projection head. The lower bound of the mutual information between the two projected features is calculated, and negative mutual information is used as the mutual information maximization loss. ; A lightweight reconstruction head is connected to the output of each stage of the infrared encoder and the visible light encoder to reconstruct the original input image. The L1 loss between the reconstructed image and the original image is calculated as the modality-aware reconstruction loss. ; After each stage output of the decoder, an intermediate fused image of the corresponding scale is generated. The difference between intermediate fused images of adjacent scales is calculated as the multi-scale consistency loss. ; The joint loss function is expressed as: ， in, for At least one of them, These are the corresponding balance weight coefficients.

Images