A multi-modal controllable image fusion method based on a physical model drive

Abstract

The application provides a kind of multi-modal controllable image fusion method based on physical model driving, it is related to controllable image fusion technical field, including: camera response characteristics, saturated pixel distribution, halo diffusion and local texture attenuation are converted into visible light degradation representation weight map;Infrared structure effectiveness weight map is generated;Cross-modal fusion control weight map is obtained;The infrared feature and visible light feature after reweighting are carried out frequency domain decomposition, and low-frequency structure component and high-frequency texture component are obtained;Dynamic cross-modal attention interaction is executed;Single-channel gray fusion image is generated;In the training stage, the cross-modal fusion control weight map is used to impose regional differentiation fusion constraints on normal area and overexposure degradation area;The application can effectively suppress halo artifacts in the fusion image, while retaining the infrared key target structure and visible light effective texture.

Description

Technical Field

[0001] This invention relates to the field of controllable image fusion technology, and in particular to a multimodal controllable image fusion method driven by a physical model. Background Technology

[0002] Existing infrared and visible light image fusion methods suffer from the following problems in locally overexposed nighttime scenes: overexposed area detection often relies on simple rules such as brightness thresholds, failing to establish a unified physical quantification model based on imaging degradation mechanisms such as camera saturation, halo diffusion, and texture attenuation; visible light degradation and infrared structural effectiveness are two heterogeneous observations with different physical meanings and statistical scales, and existing methods lack the means to uniformly map them into comparable regional modal contribution variables; cross-modal attention interactions often adopt fixed directions, making it difficult to continuously adjust information contributions based on modal advantages at different locations in the same image; furthermore, existing methods often do not distinguish between low-frequency structures and high-frequency textures, directly unifying fusion and failing to simultaneously preserve infrared target contours and visible light texture details. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: According to the physical model-driven multimodal controllable image fusion method provided in this application, the method includes: Acquire strictly registered infrared and visible light images; A visible light overexposure degradation model is established based on the imaging degradation characteristics of the visible light image, and the camera response characteristics, saturated pixel distribution, halo diffusion and local texture attenuation are converted into a visible light degradation characterization weight map. An infrared structure effectiveness weight map is generated based on the saliency of thermal radiation of the target region relative to the local background and the edge response in the infrared image. The visible light degradation characterization weight map and the infrared structural effectiveness weight map are normalized and spatially consistent, respectively, and then coupled and smoothed position by position to obtain the cross-modal fusion control weight map. The shallow features of the infrared image and the visible light image are extracted respectively, and the shallow features of the two modes are differentially reweighted according to the cross-modal fusion control weight map; The reweighted infrared and visible light features are decomposed in the frequency domain to obtain low-frequency structural components and high-frequency texture components. Based on the cross-modal fusion control weight map, dynamic cross-modal attention interaction is performed on the low-frequency structural component and the high-frequency texture component respectively. Specifically, the dominance of the infrared low-frequency structural component is increased in the overexposed degradation region, the dominance of the visible light low-frequency structural component is increased in the normal region, unreliable visible light high-frequency response is suppressed in the overexposed degradation region and infrared target boundaries are preferentially preserved, and visible light edges, textures and local details are supplemented in the normal region. The low-frequency fusion features and high-frequency fusion features after frequency division and dynamic interaction are aggregated across frequency, and a single-channel grayscale fusion image is generated through the local edge enhancement and reconstruction module. During the training phase, the cross-modal fusion control weight map is used to apply regional differential fusion constraints to the normal region and the overexposed degradation region; the regional differential fusion constraints include physical model-driven intensity constraints, low-frequency structural consistency constraints, and high-frequency gradient preservation constraints.

[0004] The present invention has at least the following beneficial effects: The present invention provides a physical model-driven multimodal controllable image fusion method. This method generates a visible light degradation characterization weight map by establishing a visible light overexposure degradation model, and simultaneously generates an infrared structural effectiveness weight map based on the saliency of infrared thermal radiation and edge response. The method normalizes two types of heterogeneous physical observations to a unified spatial weight domain and couples them position-by-position to obtain a cross-modal fusion control weight map, solving the problem of difficulty in unified quantitative comparison of heterogeneous observations. This control weight map is used to reweight shallow features differentially, and dynamic cross-modal attention interactions are performed on the low-frequency structure and high-frequency texture after frequency decomposition. In overexposed areas, the infrared thermal target outline and boundary are preferentially preserved, while in normal areas, visible light texture and background details are fully utilized. Through cross-frequency aggregation, edge enhancement, and regional differential fusion constraints, the fused image effectively suppresses halo artifacts while preserving the infrared key target structure and visible light effective texture. Attached Figure Description

[0005] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0006] Figure 1 A flowchart of a physical model-driven multimodal controllable image fusion method provided in an embodiment of the present invention. Detailed Implementation

[0007] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0008] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0009] The following will refer to Figure 1 The flowchart shown is for a physical model-driven multimodal controllable image fusion method, which introduces such a method.

[0010] The physical model-driven multimodal controllable image fusion method includes the following steps: S100 acquires strictly registered infrared and visible light images.

[0011] The core of this step lies in image registration. Due to differences in imaging mechanisms, resolution, and field of view between infrared and visible light cameras, images of the same scene captured by both cameras are not naturally aligned in spatial position. Therefore, registration is necessary to transform the two images into the same spatial coordinate system, ensuring that the same target in the scene is located at the same pixel position in both images.

[0012] The general registration process consists of four steps: feature extraction → feature matching → transform model estimation → image resampling and transformation. The specific implementation methods of each step are explained below.

[0013] 1. Feature Extraction: Extracting discriminative image features from infrared and visible light images respectively. Commonly used features include: Point features: Feature points and their descriptors are extracted using the SIFT (Scale Invariant Feature Transform) algorithm. Considering the weak texture of infrared images, the SUSAN operator can be used to constrain the location of feature points before constructing SIFT descriptors.

[0014] Edge / Contour Features: The image contour curve is extracted by NSCT (Non-subsampled contour wave transform), and then the corner points on the contour are extracted as feature points using the CSS (Curvature Scale Space) corner detection algorithm.

[0015] Phase consistency features: Mapping images to a phase consistency space to extract structural features that are insensitive to changes in illumination and contrast.

[0016] 2. Feature Matching: Feature points extracted from the infrared and visible light images are paired. A nearest neighbor search method based on kd-trees is used to find the best matching point in the other image for each feature point. Reliable matching point pairs are filtered out by a distance ratio threshold. Geometric constraints such as angle and relative distance are further used to eliminate false matches.

[0017] 3. Transformation Model Estimation: Based on the matched feature point pairs, estimate the spatial transformation relationship between the two images. The transformation matrix is ​​solved iteratively using the RANSAC (Random Sample Consensus) algorithm, while outlier matching points are removed.

[0018] The transformation model typically employs affine transformation, incorporating parameters such as translation (Δx, Δy), rotation (θ), and scaling (k). For infrared and visible light binocular camera systems, camera calibration information (intrinsic and extrinsic parameters) can be used as prior constraints to improve registration accuracy.

[0019] Homography can also be used to describe the projection mapping relationship between two images.

[0020] 4. Image resampling and transformation Using the estimated transformation matrix, coordinate transformation and pixel interpolation are performed on one of the images (usually the image to be registered): Each pixel in the image to be registered is mapped to the coordinate space of the reference image according to the transformation matrix.

[0021] Bilinear interpolation is used to calculate the pixel values ​​at non-integer coordinate positions after transformation.

[0022] The transformed image and the reference image are aligned at the pixel level in space, which is called "strict registration".

[0023] The above process allows for the selection of different feature extraction methods and transformation models depending on the actual application scenario. For stereo systems with relatively fixed camera positions, transformation parameters can be obtained in advance through calibration and applied directly thereafter; for general scenarios, features need to be extracted again and the transformation matrix estimated each time registration is performed.

[0024] S200, Based on the imaging degradation characteristics of the visible light image, a visible light overexposure degradation model is established, and the camera response characteristics, saturated pixel distribution, halo diffusion and local texture attenuation are converted into a visible light degradation characterization weight map.

[0025] The goal of this step is to generate a two-dimensional weight map corresponding to the spatial location of the visible light image. A higher value indicates that the visible light information at that location is more "unreliable" (overexposure, halo, or loss of texture). Specifically, it is synthesized by weighted summation of three response components, and the specific implementation of each component is explained below.

[0026] Furthermore, the visible light degradation characterization weight map is generated in the following manner: Within the normalized luminance space, pixel saturation response is calculated using a local window. The halo diffusion response was calculated using a normalized Gaussian point diffusion kernel. The texture loss response is calculated using the local gradient magnitude decay. The weighted summation of the three types of responses, followed by normalization mapping, yields the visible light degradation characterization weight map. Its expression is: ; in, , and The weighting coefficients are non-negative and satisfy the following conditions: The visible light degradation characterization weight map The value is used to quantify the degree to which the visible light information at the corresponding location is affected by sensor saturation, optical scattering, lens glare, and low signal-to-noise ratio degradation. The higher the value, the more severe the damage to the visible light information.

[0027] Pixel saturation response — Identifying “dead white” areas: Pixel saturation response is used to mark bright, textureless areas in visible light images that are formed due to sensor output truncation.

[0028] Local window and adaptive threshold: Assume the input is a visible light brightness map. The size is For each pixel position (i, j), take a local window centered on it. Window radius Determine by the following formula: ; That is, take 1 / 64 of the shorter side of the image (integer division), and then... The limit is between 3 and 15. For example, for The graph has a shorter side of 512. 512 / / 64 = 8. Therefore... Window size is .

[0029] Within each window, calculate the average brightness of all pixels within the window. Simultaneously calculate the global average brightness of the entire image. The saturation threshold at this pixel location. for: ; Meaning: The lower limit of the threshold is 0.75, and the upper limit is 0.95. The specific value is determined by the weighted average of the local mean and the global mean (local weight 0.6, global weight 0.4). In other words, if a certain area is already very bright (local mean close to 0.9), the threshold will be raised to above 0.9 to avoid misjudging normal bright areas as saturated.

[0030] Furthermore, the pixel saturation response It is generated jointly by adaptive local thresholding and gradient decay, specifically as follows: Let the spatial size of the input visible light brightness image be... Define the local window radius and will The value range is limited to [3, 15]; for each pixel position (i, j), its local saturation threshold is... Based on the average brightness within the local window Compared with the global average brightness The weighted result is adaptively determined: ; in, The pixel saturation response The final expression is: ; Where H and W are the height and width of the visible light brightness image, respectively; Represents the local window, 1 [] For indicator functions, This represents the brightness gradient magnitude. This setting ensures that the pixel saturation response only occurs when the brightness within a local window exceeds an adaptive threshold and the neighborhood gradient tends to vanish. Achieve high values.

[0031] Saturation response calculation: In the window Within this range, all brightness values ​​exceeding the aforementioned threshold are counted. The pixel ratio. At the same time, for these pixels that exceed the threshold, the pixel density is reduced according to their gradient magnitude—the larger the gradient, the more texture variation there is, and it should not be judged as severely saturated; the gradient approaches 0, indicating that the area is indeed "dead white".

[0032] Specific steps: For each pixel exceeding the threshold within the window, calculate its gradient magnitude. Divide by the maximum gradient magnitude of the entire image Normalize, then multiply by a coefficient Then, a negative exponential decay is applied. Finally, the average decay value of all pixels exceeding the threshold within the window is taken as the decay value for that pixel position. .

[0033] In implementation, calculations can be performed by sliding a window pixel by pixel. When the window for boundary pixels exceeds the image area, only valid pixels within the image are used for calculation.

[0034] Physical effect: Only pixels that simultaneously satisfy the conditions of "brightness exceeding the adaptive threshold" and "neighborhood gradient near vanishing" are allowed to participate. Only then will it output a high value, effectively distinguishing between real white objects and overexposure artifacts.

[0035] Furthermore, the halo diffusion response The response is generated by parallel computation using a fixed multi-scale point diffusion kernel and taking the maximum value. Specifically: Three normalized Gaussian point diffusion kernels with different spatial scales are defined, and their standard deviations are respectively... , and All units are pixels; the three Gaussian kernels are respectively compared with the visible light brightness image. Convolution is performed to obtain multi-scale blurred responses. The halo diffusion response The final expression is: ; in, This setting selects the scale that most significantly increases brightness as the halo response of the current pixel, while utilizing... Remove positions that have already responded in the saturated core region to avoid redundant weighting.

[0036] Halo diffusion response — Identify the bright halo around a strong light source: A halo is a gradually bright area formed around a strong light source after it has been optically scattered by the lens.

[0037] ① Multi-scale Gaussian blur Set Gaussian kernels at three scales, with standard deviations of respectively. Pixels. These three kernels are used to analyze the visible light brightness map. Performing convolution (i.e. Gaussian blur) yields three blurred results. During convolution, the kernel radius is usually taken as... The left and right are approximately 3×3, 9×9, and 21×21 respectively.

[0038] ② Extract the brightness enhancement at each scale For each scale ,calculate The difference between the blurred and original brightness is calculated as: brightness after blurring. A positive difference indicates that the pixel has become brighter after blurring—meaning it is within the peripheral diffusion range of a bright light source (blurring diffuses the energy of the light source to the surrounding area). ReLU (positive values ​​are retained, negative values ​​are set to zero) is used to obtain the positive brightness increase at each scale.

[0039] ③ Take the maximum-scale response and exclude saturated cores. For each pixel, the maximum brightness increase across the three scales is taken as the pixel's halo response. This is then multiplied by... That is, if the pixel has been determined to be a saturation kernel ( If the value is close to 1, the halo response is suppressed to 0 to avoid duplicate counting in saturated and halo regions.

[0040] Physical effects: Small-scale nuclei ( Capturing subtle lens flare, large-scale nuclear ( It captures a wide-ranging diffuse halo, and automatically selects the scale that most significantly enhances the brightness of each pixel as its halo intensity.

[0041] Texture loss response — Identify areas where textures disappear due to overexposure: In overexposed areas, the edges of textures that should exist are "smoothed out" by strong light, resulting in local variance or gradient amplitude that is significantly lower than in normal areas.

[0042] In practical implementation, it can be adopted with A similar local windowing strategy involves calculating the variance of luminance values ​​within a local window for each pixel, or calculating the mean of local gradient magnitudes within a normalized luminance space. The local variance (or gradient mean) is then inversely normalized—the smaller the variance / gradient, the better. The higher. At the same time, The response area should be consistent with and The spatial locations roughly overlap (i.e., high weights are assigned only to the saturated core and its neighborhood to avoid erroneous responses in flat regions of the entire graph).

[0043] A simplified implementation is to directly take... and The complement of the local gradient decay within the coverage area, or directly obtained by normalizing the inverse of the local gradient magnitude after thresholding.

[0044] Three-response weighted synthesis : The above three response diagrams , , Pixel-weighted summation, weighting coefficients The weighted sum of the three factors must equal 1. The specific values ​​are adjusted based on the actual scenario (usually, saturation core has the highest weight, followed by halo, with texture attenuation as an auxiliary factor). After weighted summation, a global minimum-maximum normalization is performed to map the numerical range to [0,1], thus obtaining the final visible light degradation characterization weight map. .

[0045] The spatial dimensions are completely consistent with the original visible light image. The higher the value, the more severe the degradation of visible light at that location, and the less suitable it is as a basis for fusion.

[0046] S300, Generate an infrared structure effectiveness weight map based on the thermal radiation significance of the target region relative to the local background and the edge response in the infrared image.

[0047] The goal of this step is to generate a two-dimensional weighted map corresponding to the spatial location in the infrared image. The higher the value, the more "useful" the infrared information at that location (the presence of a significant thermal target or a clear target boundary).

[0048] Furthermore, the infrared structural effectiveness weighting map Generate in the following way: Mean filtering is used to estimate the local background of infrared images. Extracting infrared images Relative to the local background The difference in forward thermal radiation, combined with the infrared gradient amplitude for edge enhancement, is expressed as follows: ; in, Represents the spatial smoothing kernel, This indicates element-wise multiplication. The edge enhancement coefficient, For infrared image gradient, Represents minimum-maximum normalization; the infrared structure effectiveness weighting graph The value is used to indicate the reliability of the presence of a significant infrared target structure or target boundary at the corresponding location. The higher the value, the more suitable the infrared structure at that location is for fusion.

[0049] (a) Extracting differences in positive thermal radiation – identifying areas that are “hotter than the background” In infrared images, targets such as pedestrians and vehicles typically exhibit a positive thermal radiation difference relative to the surrounding background (i.e., the pixel value of the target area is higher than that of the background). If the target is cooler than the background (negative difference), it can be processed by inverting the value or taking the absolute value, but in nighttime scenes, the focus is mainly on positive differences.

[0050] ① Local background estimation Mean filtering is used for infrared images Smoothing is performed to obtain a local background estimate. : ; The window size should be chosen to match the scale of the target image, typically set to 1 / 32 to 1 / 16 of the shorter side of the image. This can be achieved by directly calling OpenCV's `blur()` function.

[0051] ② Calculate positive differences Pixel-by-pixel calculation For each pixel location, if the value is greater than 0 (i.e., the pixel brightness is higher than the local background), it indicates the possibility of a hot target at that location, and the difference is retained; if it is less than or equal to 0, it indicates that the location is not hotter than the background, and is directly set to 0. ; The ReLU operation ensures that only the "positive thermal radiation difference" is preserved, and the background region output is 0. The Diff is then subjected to a global min-max normalization, mapping it to the [0,1] interval.

[0052] (ii) Edge response enhancement – ​​contribution to strengthening the target contour The edges of hot targets (i.e., the transition zone between the target and the background) are equally important because these locations need to maintain sharp boundaries in subsequent fusion. Gradient magnitude is used to characterize edge intensity.

[0053] ① Calculate the gradient magnitude For the original infrared image Calculate gradient magnitude In implementation, the Sobel operator can be used to calculate the gradients in the x and y directions separately, and then their magnitudes can be determined. ; in, and These are the Sobel gradient responses in the horizontal and vertical directions, respectively.

[0054] ② Normalization and Enhancement The gradient magnitude is normalized using a global minimum-maximum normalization, mapped to the interval [0,1], denoted as... Then multiply by the edge enhancement factor. (Usually 2~4), then add 1 to get an enhancement factor that is always ≥1: ; ③ Element-by-element multiplication Multiply the thermal radiation significance map (normalized Diff) obtained in step (1) element-wise with the above enhancement factor: ; The physical meaning of this operation is that the base weights are determined by "how much hotter the target is than the background", while the edge positions (with larger gradients) will receive additional gains, which further increases the weight of the target contour region.

[0055] (III) Spatial Smoothing and Normalization – Generating the Final Weight Graph ① Spatial smoothness The above enhancement results are Gaussian smoothed, and the convolution kernel is... Standard deviation Typically, 1 to 3 pixels are used. ; The purpose of smoothing is to eliminate isolated high-response points caused by noise or image texture, making the weight map more continuous and stable in space.

[0056] ② Normalization Perform a global minimum-maximum normalization on the smoothed result, mapping it to the [0,1] interval, to obtain the infrared structure effectiveness weight map. : ; For example: a good It should have a high weight in hot target areas such as pedestrians and vehicles, and at their edges, while the weight should approach 0 in pure background areas (sky, road surface, etc.). It will be compared with previously generated... The coupling determines which modality information is ultimately used at each location.

[0057] S400, the visible light degradation characterization weight map and the infrared structural effectiveness weight map are normalized and mapped and spatially consistent, respectively, and then coupled and smoothed position by position to obtain the cross-modal fusion control weight map.

[0058] The goal of this step is to transform the two heterogeneous weighted graphs obtained in the first two steps. (The higher the value, the less reliable the visible light) and (Higher = more useful infrared) Combined into a unified control chart . A high value indicates that "visible light is damaged but infrared light is useful", while a low value indicates that "visible light is still usable or infrared light has no significant advantage".

[0059] Furthermore, the cross-modal fusion control weight graph The generation process includes a lightweight multi-scale processing strategy, specifically: First, the visible light degradation characterization weight map is... and the infrared structure effectiveness weighting diagram Min-max normalization was performed separately to unify the numerical range of both classes to the [0,1] interval. To reduce computational cost, bilinear interpolation was used to downsample the normalized weight maps to a fraction of their original spatial dimensions. Position-by-position coupling is performed in the low-resolution space: ; in, and These are the weighted images after downsampling; then... Median filtering is performed for noise reduction, and then guided upsampling is used to restore the original spatial dimensions to obtain the final cross-modal fusion control weight map. The guiding map uses the gradient edge map of the infrared image to maintain the sharpness of the control weight map at the target boundary; the cross-modal fusion control weight map High values ​​indicate that visible light information is impaired while the infrared structure is effective, while low values ​​indicate that visible light information is still usable or the infrared structure does not have a significant advantage.

[0060] (a) Normalization – Unifying the dimensions and Although both are within the [0,1] interval, their numerical distribution characteristics differ. We perform another global min-max normalization to ensure that their numerical scales are completely consistent: ; in, Take the minimum value (e.g.) This step is to prevent the denominator from being zero. It has no complex physical meaning and is purely for engineering numerical alignment.

[0061] Why is this step necessary? Although normalization was performed in the previous generation process, due to... The three-component weighting method and The enhancement coefficients η differ, and the criteria for judging "high / low" in their actual values ​​are not entirely consistent. Secondary normalization eliminates this difference, making the physical meaning of subsequent pixel-by-pixel multiplication clearer.

[0062] (ii) Downsampling – reducing computational load Bilinear interpolation is used to downsample the two normalized weighted maps to their original spatial dimensions. That is, the width and height are each reduced to 1 / 4 of their original size. For example, the original image... After downsampling, .

[0063] Why downsample to 1 / 4 instead of other ratios? 1 / 4 represents a balance between computational reduction and spatial information preservation. Downsampling by 1 / 4 reduces computation to 1 / 16 of its original value, while... The low-frequency characteristics (regional modal advantage) of the device itself do not require extremely fine spatial resolution; 1 / 4 is sufficient to preserve the macroscopic distribution of overexposed and normal areas. In practice, 1 / 2 or 1 / 8 can also be selected depending on the device's computing power, but 1 / 4 is the default value and can meet the needs of most scenarios.

[0064] (III) Position-by-position coupling - multiplication + normalization In the low-resolution space, multiply the two normalized weight maps pixel by pixel: ; Then perform a global min-max normalization on the multiplication result: ; The two diagrams have different physical meanings. Mark "bad" Mark "good". Only when a location simultaneously satisfies "poor visible light" and "good infrared" should it be considered a region "must be dominated by infrared". If additive methods are used, regions that are "medium-poor in visible light but also moderate in infrared" may receive a medium weight, resulting in an unclear definition. The logic of multiplication is: if both are strong, the result is strong; if either is weak, the result is weak. This aligns with the "modal dominance" determination logic.

[0065] (iv) Median filtering for noise reduction – eliminating isolated outliers Low resolution There may be a small number of isolated high or low value noise points (from noise or mismatches during feature extraction). Median filtering is used for denoising, with a window size typically set to... : ; The advantage of median filtering over mean filtering is that it smooths out noise without blurring edge boundaries, which is beneficial for subsequent upsampling.

[0066] (v) Guided upsampling – restoring the original size and maintaining sharp edges Low resolution Restore to the original spatial dimensions. If bilinear interpolation upsampling is used directly, the weight map will become blurred at the target edge (because interpolation smooths the boundary), resulting in insufficiently crisp mode switching at the target contour during subsequent fusion.

[0067] Therefore, guided upsampling is employed, with the guide image being the gradient edge map of the infrared image. Specific implementation details are as follows: Calculate the infrared gradient magnitude map: for the original infrared image Calculate the gradient magnitude using the Sobel operator. Normalized to [0,1].

[0068] Will A coarse version is obtained by upsampling to the original size using bilinear interpolation. .

[0069] Edge guidance correction: Edge-preserving filtering is applied to the coarse version, and the guidance image is as follows. A simple and effective implementation is the Joint Bilateral Filter: using the infrared gradient map as a guide, it smooths the coarse weight map while preserving its edges. The core of the Joint Bilateral Filter is to simultaneously consider the spatial distance and the gray-level similarity of the guide map—at locations with large infrared gradients (target edges), the filtering does not over-smooth the image, maintaining a steep transition in the weight map; at locations with small infrared gradients (flat regions), it performs normal smoothing filtering.

[0070] In implementation, you can directly call OpenCV's cv2.ximgproc.guidedFilter() or cv2.bilateralFilter() and provide a guide map. The final result will have the same dimensions as the original image. .

[0071] S500, extract the shallow features of the infrared image and the visible light image respectively, and perform differential reweighting on the shallow features of the two modes according to the cross-modal fusion control weight map.

[0072] The goal of this step is to use cross-modal fusion to control the weight graph. The shallow features of the two modalities are "pre-modulated"—before the features enter into deep interactions, the network learns which modality each location should depend on more. Simultaneously, a spatial transformation network is embedded as a fault-tolerance mechanism for inaccurate input registration.

[0073] Furthermore, the shallow feature extraction and differential reweighting process includes a spatial registration fault-tolerance mechanism, specifically: Infrared shallow features are extracted using convolutional layers, normalization layers, nonlinear activation layers, depthwise separable convolutional layers, and pointwise convolutional layers, respectively. and visible light shallow features Before differential reweighting, infrared shallow features are... and visible light shallow features After concatenation along the channel dimensions, the data is fed into a spatial transformation network, which predicts a 2×3 affine transformation parameter. And for infrared shallow features Perform spatial transformation to obtain the registered and corrected shallow infrared features. To tolerate a global registration error of no more than 5 pixels between infrared and visible light images; during the training phase, a structural consistency regularization term is applied to the spatial transformation network: ; in, The learning-aware image patch similarity metric is used; after registration and correction are completed, the cross-modal fusion control weight map is used. Interpolate to the same spatial dimension as the shallow features, denoted as And perform differential reweighting according to the following formula: ; in, and The control coefficient is non-negative and adjustable. and These are the reweighted infrared and visible light features, respectively.

[0074] (I) Shallow Feature Extraction The infrared and visible light branches share the same network structure, and each branch extracts shallow features from its respective input image.

[0075] Network structure: It includes the following operations in sequence: Convolutional layer: 3×3 convolution, input channel number is 1 (grayscale image), output channel number is set to... (Usually 32 or 64), step size is 1, padding is 1 to keep the size unchanged.

[0076] Normalization layer: Batch Normalization accelerates training and stabilizes the distribution.

[0077] Nonlinear activation layer: ReLU activation function.

[0078] Depthwise separable convolutional layers: First, perform channel-wise convolution (each channel performs a 3×3 convolution independently), then perform pointwise convolution (1×1 convolutions are performed across channels for fusion). Compared to ordinary convolution, depthwise separable convolutions significantly reduce the number of parameters and computational cost, making them suitable as lightweight feature extractors.

[0079] Pointwise convolutional layer: Finally, a 1×1 convolution is used to adjust the number of channels, and the output feature map size is... .

[0080] The input to each of the two branches is the original infrared image. Visible light brightness diagram (If the input is RGB, it is first converted to grayscale and brightness channels), and the outputs are infrared shallow layer features. and visible light shallow features Both have spatial dimensions. The number of channels is .

[0081] (II) Spatial Transformation Network - Registration Fault Tolerance Due to the physical parallax between infrared and visible light cameras, even with registration during preprocessing, it is difficult to guarantee perfect alignment of every pixel. Therefore, a lightweight Spatial Transformer Network (STN) is introduced to perform secondary fine-tuning in the feature space.

[0082] ①Localization Network Will and By splicing along the channel dimension, the resulting shape is... The tensor is fed into a small convolutional network (2-3 convolutional layers + global average pooling + fully connected layers) to regress a... affine transformation parameter matrix : ; in, and Control the translation in the horizontal and vertical directions respectively. Control rotation and scaling.

[0083] ② Mesh generation and sampling STN utilization Generate a sampling grid and input feature map Each pixel coordinate in the image is mapped to the output coordinate space. Then, bilinear interpolation is used to... By sampling pixel values, the transformed feature map is obtained. .

[0084] During training, Apply constraints to limit the corresponding translation amount to Within pixels (by applying a regularization term to the translation parameters or by direct cropping) The value of STN ensures that it is used only for fine-tuning registration errors, rather than arbitrarily distorting the image structure.

[0085] ③ Structural consistency regularization term STN alone cannot guarantee structural alignment between the transformed infrared and visible light features; therefore, an additional regularization loss is added during the training phase. ; LPIPS (Learned Perceptual Image Patch Similarity) is a pre-trained perceptual similarity measurement network that takes two feature maps as input and outputs a scalar value that measures the structural similarity between them. This loss forces STN to align the structure of infrared features with that of visible light features as much as possible during the transformation process. Norms, as a distance metric, compared to More robust to outliers.

[0086] (iii) Differentiated reweighting After completing the registration and correction, use The two modal features are modulated separately.

[0087] ① Weighted graph interpolation The size is (Consistent with the original image), and the spatial dimensions of shallow features are also... (Convolution does not change the spatial dimensions), therefore directly... Simply copy it to each channel of the feature map. If the feature map size is... Inconsistencies (e.g., downsampling in subsequent networks) require bilinear interpolation to correct them. Adjust to the same spatial dimensions as the feature map. The interpolated control weight map is denoted as... , shape is And broadcast to all according to the channel direction. One channel.

[0088] ② Modulation formula Enhanced modulation of infrared features and suppressed modulation of visible light features: ; ; in, This indicates element-wise multiplication (the broadcasting mechanism automatically multiplies the elements). Broadcast to every channel.

[0089] Parameter settings: and All are non-negative control coefficients, with the default value being... , . The design makes it possible to In the high-value region, the infrared enhancement is slightly greater than the visible light suppression, ensuring sufficient infrared information to compensate for overexposed areas. Modulation factor The range is [1, 1+a], meaning the infrared signature will only be enhanced or remain unchanged, not weakened. Modulation factor The range is [1-b,1], meaning that visible light features will only be suppressed or remain unchanged, and will not be enhanced.

[0090] Physical meaning: In In the region close to 1 (overexposed and infrared effective), the infrared feature is magnified (multiplied by 1+a), and the visible light feature is compressed (multiplied by 1-b); In regions close to 0 (normal), the two modal characteristics remain essentially unchanged, maintaining their original complementary relationship. All modulation is performed at the pixel level, with different proportions used at different locations.

[0091] S600 performs frequency domain decomposition on the reweighted infrared and visible light features to obtain low-frequency structural components and high-frequency texture components.

[0092] The goal of this step is to decompose the reweighted infrared and visible light features into two parts: "low-frequency structure" (macroscopic profile) and "high-frequency texture" (microscopic details), so that different fusion strategies can be adopted for the two frequency components in the future.

[0093] Furthermore, the frequency domain decomposition is achieved through channel-by-channel Gaussian low-pass filtering, specifically as follows: For reweighted infrared features and visible light characteristics Gaussian low-pass filtering is performed channel by channel to obtain low-frequency structural components. : ; in, A Gaussian low-pass kernel with a standard deviation set to 1 to 2 feature pixel scales; high-frequency texture components. Represented by the residual between the original features and the low-frequency structural components: ; The low-frequency structural components The high-frequency texture component is used to represent global brightness distribution, scene layout, and thermal target outlines. Used to represent edges, local textures, and fine-grained background details.

[0094] In this embodiment, infrared and visible light images have different information advantages at different frequencies—the main value of infrared lies in the overall outline of the thermal target (low frequency), while the main value of visible light lies in the background texture and edges (high frequency). If they are fused together without distinguishing between frequencies, low frequencies can easily dilute each other, and high frequencies can contaminate each other. Therefore, it is necessary to separate the features in the frequency domain first.

[0095] Low frequency components The "baseline" representing the feature map consists of large-scale brightness variations, scene layout, and the coarse outline of the target. High-frequency components... The “details” of the feature map are edges, textures, and fine-grained structures.

[0096] Reweighted feature map and Perform Gaussian low-pass filtering on each channel. The operation is performed independently for each channel. ; ① Generation of Gaussian kernels It is a two-dimensional Gaussian convolution kernel with a standard deviation of The unit is feature pixels, not original image pixels. This is because the operation here takes place in feature space, where one "feature pixel" corresponds to one receptive field region of the original image.

[0097] Set to between 1 and 2 feature pixels, the specific value depending on the low-frequency scale you want to retain. The timing filtering is relatively weak, retaining more mid-frequency components; The time filtering is stronger, resulting in purer low-frequency components. (Default setting) .

[0098] ② Convolution kernel size The size of the Gaussian kernel is usually taken as To ensure that most of the energy is covered. If Then the core size is That is, 11×11. In actual implementation, 5×5 or 7×7 can also be used.

[0099] ③ Boundary processing Padding is used during convolution to ensure that the output feature map has the same size as the input. Common padding methods include: Zero padding: Fill the boundary with zeros, which is simple and computationally efficient.

[0100] Reflection padding: Mirroring the boundary to avoid abrupt zero-value transitions. Since the feature map itself does not contain artifacts caused by zero-padding, reflection padding is recommended.

[0101] ④ Channel-by-channel operation Input feature map The shape is The Gaussian convolution kernel is applied independently to each channel, meaning that each channel uses the same kernel parameters. Output The shape is still .

[0102] The high-frequency components are obtained by subtracting the low-frequency components from the original features: ; This is an element-wise subtraction operation, and the shape remains the same. .

[0103] Mathematical meaning: High-frequency components represent all the information in the original features that was attenuated by the Gaussian filter—that is, components with drastic spatial variations. These components correspond to edges, textures, and fine-grained details in the image.

[0104] It should be noted that, due to yes The low-pass filtering results of the two are similar in value range, and the difference is... The values ​​are typically small (around 0). No need to... Additional normalization is sufficient to maintain the relative values, as subsequent cross-modal attention interactions will adjust automatically.

[0105] S700, according to the cross-modal fusion control weight map, performs dynamic cross-modal attention interaction on the low-frequency structural component and the high-frequency texture component respectively, wherein, in the overexposed degradation region, the dominant role of the infrared low-frequency structural component is increased, in the normal region, the dominant role of the visible light low-frequency structural component is increased, in the overexposed degradation region, unreliable visible light high-frequency response is suppressed and infrared target boundaries are preferentially preserved, and in the normal region, visible light edges, textures and local details are supplemented.

[0106] The goal of this step is to perform cross-modal attention interactions on the separated low-frequency and high-frequency features, but the interaction method is not fixed—it is determined by the control weight map. This determines whether each location is biased towards infrared or visible light. Low frequencies emphasize "whose structure to use," while high frequencies emphasize "whose details to use." The entire interaction is designed to be lightweight to ensure inference speed.

[0107] Furthermore, the dynamic cross-modal attention interaction is achieved by constructing infrared dominant candidate features and visible light dominant candidate features for low-frequency structural components and high-frequency texture components respectively, and then continuously interpolating them according to control weights, specifically as follows: For low-frequency structural branches and high-frequency texture branches The query, key, and value are obtained using linear mapping, and infrared dominant candidate features are calculated using scaled dot product attention. and visible light dominant candidate features The cross-modal fusion control weight map Interpolate to the same spatial size as the current frequency branch feature, denoted as And perform position-by-position dynamic fusion according to the following formula: ; in, and These are low-frequency fusion features and high-frequency fusion features, respectively; when When the value approaches 1, the corresponding region increases the contribution of infrared candidate features. When the value approaches 0, the corresponding region increases the contribution of visible light candidate features, thereby enabling continuous and adjustable fusion of different regions in the same image based on local modal advantages using different interaction ratios; the cross-modal attention interaction adopts a grouped depth-separable attention mechanism, dividing the number of channels into 4 groups for independent calculation and then stitching them together, and setting the number of heads of multi-head attention to 2, compressing the number of feature channels to 32 dimensions to reduce the amount of inference computation.

[0108] (I) Overall Logic - Each frequency branch independently follows the process of "generating two candidates → weighted interpolation". For low-frequency branches ( ) and high-frequency branches ( The operations are exactly the same, only the input characteristics are different: low-frequency branch input: and High-frequency branch input: and The output of each branch is a fused feature map. The size is the same as the input.

[0109] (ii) Generate two "candidate features" for each branch. For a certain frequency branch The input has infrared characteristics. and visible light characteristics (here) represent or Two candidates need to be generated: Infrared dominant candidate Information is primarily extracted from infrared features, while also referencing the context within visible light features.

[0110] Visible light dominant candidate Information is primarily extracted from visible light features, while also referencing the context in infrared features.

[0111] Implementation method – Cross-Attention ① Generate Q, K, V For input features and Perform linear mapping (1×1 convolution) on each to obtain their respective Query, Key, and Value.

[0112] For infrared dominant candidates The query comes from infrared, while the key and value come from visible light. The resulting output features are the result of "using infrared as the query and searching for relevant context in visible light". Essentially, it uses the context of visible light to enrich the infrared features, but it is still dominated by the infrared structure.

[0113] For visible light dominant candidates Conversely, the query comes from visible light, while the key and value come from infrared light.

[0114] Formula (taking infrared as the dominant factor as an example): ; ; ② Scaling Dot Product Attention: It is the dimension of the key (i.e., the number of channels per feature point divided by the number of heads), divided by To prevent the softmax gradient from saturating due to excessively large dot product values.

[0115] ③ Multi-head mechanism: Standard multi-head attention divides the channel into Calculate each element separately and then piece them together. (Here) The number of channels is compressed to 32 dimensions, which means that each head has a dimension of 16, greatly reducing the amount of computation.

[0116] Important detail: The spatial dimension of Query and Key is the number of pixel positions on the feature map. Attention is done in the spatial dimension—that is, each pixel position is processed by the attention mechanism to look for relevant information in all positions of the other modality, rather than just in its own position.

[0117] (III) Grouping Depth Separable Attention – Lightweight Core Dividing the number of channels into 4 groups for independent calculation is a further extension of the bullish attention.

[0118] The specific implementation is as follows: The channel dimension of the input features is divided into 4 groups, and the number of channels in each group is the total number of channels / 4 (here, 32 / 4 = 8).

[0119] The Cross-Attention calculation described above is performed independently for each group (i.e., each group has its own linear mapping weights).

[0120] Finally, the outputs of the four groups are spliced ​​back to the original 32 channels in the channel dimension.

[0121] The advantage of this design is that: In standard multi-head attention, each head performs a global projection on all channels, which involves a large amount of computation.

[0122] After grouping, each group only performs calculations on 8 channels, which is equivalent to reducing the amount of computation to 1 / 4 of the original.

[0123] Different groups can focus on different types of features (for example, one group focuses on horizontal textures, and another group focuses on vertical edges), similar to the effect of multiple convolutional kernels.

[0124] (iv) Controlled weight interpolation – fusing two candidates position by position Infrared dominant candidates were generated and visible light dominant candidate Afterwards, use Decide which one to use in each position.

[0125] ① Interpolation weight map: From the original size Spatial size of bilinear interpolation to the current feature map , recorded as After interpolation, it is broadcast along the channel dimension, and the shape becomes... .

[0126] ②Position-by-position fusion: ; This is a pixel-by-pixel linear interpolation. When When the value is close to 1 (at which visible light is degraded and infrared light is effective), the fusion result almost entirely uses infrared as the dominant candidate; when When the value is close to 0 (visible light is still good at this location), the fusion result almost entirely uses visible light as the dominant candidate. When the value is around 0.5, take half of each.

[0127] It should be noted that the fusion here occurs after the candidate features have been calculated using Cross-Attention, not during the QKV stage. In other words, the two candidates have already completed cross-modal information exchange, and the final interpolation only determines "which exchange method to use." This is the core of the solution—each position first independently calculates the "fusion result from the infrared view" and the "fusion result from the visible light view," and then uses control weights to perform a soft switch.

[0128] (v) The difference between low-frequency and high-frequency branches – reflected only in input characteristics Although the calculation process for the two branches is exactly the same, the results will naturally differ due to the different inputs: The low-frequency branch inputs are outlines and scene layouts, so the interaction result determines "whose macro structure to use at this location".

[0129] The high-frequency branch inputs are textures and edges, so the interaction result determines "whose micro-details are used at this location".

[0130] The advantage of separate processing is that one location can use infrared light at a low frequency (because the visible light outline of the overexposed area is already distorted), while simultaneously using the texture of the unexposed portion of the visible light at a high frequency (if the location itself is not fully saturated). However, since the control weights are the same... This difference is essentially determined by the distribution of the frequency characteristics themselves—in overexposed areas, low-frequency visible light characteristics are severely distorted, while high-frequency visible light characteristics may have disappeared (gradient approaches zero), therefore even In high-frequency branches, the quality of visible light candidates is also very poor, so the model naturally relies more on infrared candidates. The entire system relies on the quality of the features themselves and The guidance of these factors jointly completes the frequency-aware modality selection.

[0131] The S800 performs cross-frequency aggregation of the low-frequency fusion features and high-frequency fusion features after frequency division dynamic interaction, and generates a single-channel grayscale fusion image through the local edge enhancement and reconstruction module.

[0132] The goal of this step is to re-merge the low-frequency and high-frequency fusion features and output the final single-channel grayscale fusion image through reconstruction. Simultaneously, since the preceding frequency-division attention interaction may have smoothed the edges to some extent, a local edge enhancement is needed to compensate for this before reconstruction. The entire process consists of three sequential operations: cross-frequency aggregation → local edge enhancement → image reconstruction. These are explained below.

[0133] (a) Cross-frequency aggregation – piecing together high and low frequency features The input is low-frequency fused features and high frequency fusion features Both have the same shape (both are) The aggregation method is simple: directly concatenate (Concat) along the channel dimension to obtain... The tensor.

[0134] Then, a 1×1 convolution is used to reduce the number of channels from... Press back The purpose of this 1×1 convolution is to allow the network to learn the reweighted combination of each channel—which low-frequency channels need to be paired with which high-frequency channels, determined by the training process. Essentially, this step re-integrates the separated information at the feature level.

[0135] In implementation, simply call `torch.cat([F_f_L,F_f_H],dim=1)` followed by another... That's all.

[0136] (ii) Local edge enhancement – ​​compensating for edge blunting caused by attention The Softmax operation in cross-modal attention is essentially a weighted average, which can smooth out sharp edges in the feature map to some extent. Although the high-frequency branch has already processed the texture separately, there may still be a problem of reduced edge contrast in the aggregated features. The role of local edge enhancement is to "fill in" the missing edges.

[0137] The specific implementation involves attaching a residual structure to the aggregated features: ① After aggregation The features are fed into a 3×3 depthwise separable convolution (channel-wise convolution + pointwise convolution), and the output is still... .

[0138] ② Add the convolutional output to the original aggregated features element by element to form a residual connection: ; Why does this enhance edges? Depthwise separable convolution, during channel-wise convolution, essentially performs a weighted combination of local neighborhoods on the feature maps of each channel—similar to the local neighborhood operations in traditional Laplacian or Sobel edge detection, except the weights are learned through network training rather than preset manually. A 3×3 window is sufficient to capture the differences between adjacent pixels. Residual connections ensure that the original feature information is not lost, and if the added portion learns edge features, it's equivalent to superimposing an edge-enhancing signal on the original features.

[0139] Here, depthwise separable convolution is used instead of ordinary 3×3 convolution, mainly because of the smaller number of parameters (ordinary 3×3 convolution has 1000 parameters). Depth can be separated into (When C is large, the latter is much smaller), consistent with the lightweight design of the entire solution.

[0140] (III) Image Reconstruction – From Feature Map to Grayscale Image Feature map after edge enhancement Convert to a single-channel grayscale image.

[0141] Specific method: Use a 3×3 convolution to reduce the number of channels from... The value is reduced to 1, and then the Sigmoid activation function is applied to map the output value to the [0,1] interval: ; The 3×3 convolution performs a final local fusion in space, extracting the brightness information stored in the feature channels into pixel values. The output size is... The size is consistent with the input image (the spatial resolution of each previous layer has not been changed).

[0142] The entire S800 data stream is strung together in sequence: ; ; ; Final output It is the grayscale image after fusion.

[0143] S900, during the training phase, applies regional differential fusion constraints to normal regions and overexposed degradation regions using the cross-modal fusion control weight map; the regional differential fusion constraints include physical model-driven intensity constraints, low-frequency structural consistency constraints, and high-frequency gradient preservation constraints.

[0144] The goal of this step is to design a loss function that allows the network to know "which region should align with" during training—overexposed areas should align with infrared, while normally exposed areas should maintain a balance between the two. The entire loss function consists of three terms, controlling brightness, low-frequency structure, and high-frequency texture, respectively.

[0145] Furthermore, the total loss function of the regional differentiation fusion constraint Defined as: ; in, Strength constraint loss driven by the physical model For low-frequency structural consistency constraint loss, To preserve the constraint loss for high-frequency gradients, , and The weighting coefficients are positive, with initial weights set to 1, 1, and 10 respectively; the strength constraint loss The cross-modal fusion control weight map The low-value regions maintain the brightness balance between the fused image and the infrared and visible light images. The high-value regions guide the alignment of the intensity and low-frequency structure of the fused image with the infrared image; the low-frequency structure consistency constraint loss exist The high-value regions maintain the consistency between the low-frequency components of the fused image and the low-frequency components of the infrared image; the high-frequency gradient preservation constraint loss exist In low-value regions, edges and textures of the image with the larger gradient magnitude from the two source images are preferentially preserved. The gradient response of the infrared target edge is preferentially preserved in the high value region.

[0146] (a) Overall Loss Framework The total loss is the weighted sum of three terms: ; Weight The highest value indicates that the scheme prioritizes texture and edge preservation—the core quality metrics for image fusion. All three losses depend on... To differentiate the processing methods for different regions.

[0147] (ii) Strength constraint loss —Control overall brightness This loss constraint ensures that the pixel intensity (i.e., brightness value) of the fused image is within a reasonable range. The constraint objectives differ for different regions. Low Region (Reliable in Visible Light): The brightness of the fused image should be close to the average brightness of both infrared and visible light, maintaining a balance between the two.

[0148] high Region (Visible light degradation): The brightness of the fused image should be closer to that of the infrared region to avoid introducing overexposed high-brightness artifacts into the fusion result.

[0149] Specific calculation method: ① Perform binarization or soft partitioning: Low-weight mask (The larger the value, the more it belongs to the normal range); High-weight mask (The larger the value, the more it belongs to the overexposure degradation area).

[0150] ② Calculate the strength loss of each part separately: ; in, It is the pixel-wise average of the two source images. The first term (low-weight region) forces the fused image to approximate the average brightness of the two source images; the second term (high-weight region) forces the fused image to approximate the infrared brightness.

[0151] In practice, L1 loss is usually used directly, as it is more robust to outliers than L2 loss. (Adding the denominator...) Prevent division by zero.

[0152] Explanation of the choice between L1 and L2: L1 loss (absolute value) is insensitive to extreme pixel values ​​that may exist in overexposed areas, and the gradient will not explode as the error increases, resulting in more stable training; while L2 loss (squared) over-penalizes large errors, causing the network to tend to output overly smooth results to avoid penalizing extreme values, which weakens the sharpness of the target boundary. Therefore, this scheme adopts the L1 norm for all cases.

[0153] (iii) Low-frequency structural consistency loss —Protect the outline of infrared targets This lossy forced fusion image's low-frequency components match the infrared low-frequency components in overexposed areas, aiming to preserve the overall outline of infrared thermal targets. Only in... The high-level area is affected.

[0154] The specific calculation method is as follows: ①For fused images Infrared images Visible light images Perform Gaussian low-pass filtering on each component to extract their respective low-frequency components: ; Standard deviation of Gaussian kernel A larger value (such as 5~10 pixels) can be selected to ensure that the true macroscopic outline is extracted rather than the detailed texture.

[0155] ② Calculate the loss: ; Note that only this is used. The mask is applied only to the overexposed and degraded regions. The low-frequency components of visible light are not involved in this loss at all—because in the overexposed regions, the low-frequency structure of visible light has already been distorted by halos and saturation, and should not be considered as a monitoring target.

[0156] Low The region (normal region) does not require additional constraints on the low-frequency structure, because the intensity constraint has already been balanced at the brightness level, and the low frequency, as a macroscopic manifestation of brightness, will naturally be indirectly constrained.

[0157] (iv) High-frequency gradient preservation loss —Preserve texture and edges This loss has the highest weight among the three ( The goal is to preserve as much texture detail and edges as possible in the merged image. Different gradient objectives are used for different regions: Low Region (visible light reliable): Select the target with the larger amplitude between the infrared gradient and the visible light gradient (whichever texture is stronger is retained).

[0158] high Region (Visible light degradation): Only the infrared gradient is taken as the target (the visible light gradient is unreliable in the overexposed area, and may be the boundary of the halo artifact rather than the real texture).

[0159] The specific calculation method is as follows: ① Calculate the gradient magnitudes of the three images using the Sobel operator: ; The Sobel operator uses a 3×3 kernel to calculate the gradients in the x and y directions separately, and then calculates the magnitude. .

[0160] ② Generate gradient target map : ; That is, in the low-weight region, the larger of the two source gradients is taken, and in the high-weight region, only the infrared gradient is taken.

[0161] ③ Calculate the loss: ; Calculate the L1 distance between the gradient of the fused image and the gradient of the target image pixel by pixel.

[0162] Why set the gradient loss weights to 10? The human eye is most sensitive to the sharpness of textures and edges. Experiments show that too low gradient loss weights can lead to overly smooth, blurry images. Setting the gradient loss weights to 10 times the intensity forces the network to prioritize edge preservation during training, while intensity and low-frequency structures act as auxiliary constraints to prevent significant brightness deviations or contour distortions.

[0163] In each training iteration, the following steps are performed in order: Forward propagation yields the fused image ; Will , , and Input into the loss calculation module; Calculate the three losses separately: intensity loss (pixel-level L1), low-frequency loss (L1 after Gaussian blur), and gradient loss (Sobel gradient L1). according to Weighted summation; Backpropagation updates the parameters of the entire network (including all trainable parameters of shallow feature extraction, STN, attention module, and reconstruction module). All operations are differentiable—Gaussian blur, Sobel gradient, pixel-wise weighted summation—so gradients can be propagated end-to-end.

[0164] The parameter settings are shown in Table 1.

[0165] Table 1

[0166] In this embodiment, through the entire process design of physical modeling, unified weight mapping, frequency-division dynamic attention interaction, and regional differential supervision, the overall effect of the present invention is reflected in the following aspects: In local overexposed scenes at night, the fused image can effectively suppress brightness artifacts and texture distortion caused by sensor saturation, halo diffusion, and lens flare. At the same time, in the overexposed degradation area, the complete outline and sharp boundary of thermal targets such as pedestrians and vehicles in the infrared image are stably preserved. In the normal area, the rich background texture, edge details, and scene layout information in the visible light image are fully inherited. Moreover, the modal contribution switching between different areas is smooth and natural, without producing a harsh sense of spatial discontinuity. Its core advantage lies in the fact that the aforementioned effects are not achieved through empirical thresholds or fixed fusion rules, but rather through a weight map driven by a physical model that quantifies the degree of visible light degradation and the effectiveness of infrared structures pixel by pixel. Based on this, it adopts differentiated attention interaction strategies for low-frequency structures (contours) and high-frequency textures (details), giving the modality selection in the fusion process a clear physical basis and spatial adaptability. Combined with the regionally differentiated loss constraints during the training phase, the network can actively align to the infrared stable structure in overexposed areas, maintain dual-modal brightness balance in normal areas, and prioritize the retention of strong gradient responses. Ultimately, the fused image not only meets the requirements of naturalness and clarity for human subjective vision, but also has practical application value in providing high-quality input for downstream visual tasks such as object detection and semantic segmentation.

[0167] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0168] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A multimodal controllable image fusion method based on a physical model, characterized in that, include: Acquire strictly registered infrared and visible light images; A visible light overexposure degradation model is established based on the imaging degradation characteristics of the visible light image, and the camera response characteristics, saturated pixel distribution, halo diffusion and local texture attenuation are converted into a visible light degradation characterization weight map. An infrared structure effectiveness weight map is generated based on the saliency of thermal radiation of the target region relative to the local background and the edge response in the infrared image. The visible light degradation characterization weight map and the infrared structural effectiveness weight map are normalized and spatially consistent, respectively, and then coupled and smoothed position by position to obtain the cross-modal fusion control weight map. The shallow features of the infrared image and the visible light image are extracted respectively, and the shallow features of the two modes are differentially reweighted according to the cross-modal fusion control weight map; The reweighted infrared and visible light features are decomposed in the frequency domain to obtain low-frequency structural components and high-frequency texture components. Based on the cross-modal fusion control weight map, dynamic cross-modal attention interaction is performed on the low-frequency structural component and the high-frequency texture component respectively. Specifically, the dominance of the infrared low-frequency structural component is increased in the overexposed degradation region, the dominance of the visible light low-frequency structural component is increased in the normal region, unreliable visible light high-frequency response is suppressed in the overexposed degradation region and infrared target boundaries are preferentially preserved, and visible light edges, textures and local details are supplemented in the normal region. The low-frequency fusion features and high-frequency fusion features after frequency division and dynamic interaction are aggregated across frequency, and a single-channel grayscale fusion image is generated through the local edge enhancement and reconstruction module. During the training phase, the cross-modal fusion control weight map is used to apply regional differential fusion constraints to the normal region and the overexposed degradation region; the regional differential fusion constraints include physical model-driven intensity constraints, low-frequency structural consistency constraints, and high-frequency gradient preservation constraints.

2. The method according to claim 1, characterized in that, The visible light degradation characterization weight map is generated in the following manner: Within the normalized luminance space, pixel saturation response is calculated using a local window. The halo diffusion response was calculated using a normalized Gaussian point diffusion kernel. The texture loss response is calculated using the local gradient magnitude decay. The weighted summation of the three types of responses, followed by normalization mapping, yields the visible light degradation characterization weight map. Its expression is: ； in, , and The weighting coefficients are non-negative and satisfy the following conditions: The visible light degradation characterization weight map The value is used to quantify the degree to which the visible light information at the corresponding location is affected by sensor saturation, optical scattering, lens glare, and low signal-to-noise ratio degradation. The higher the value, the more severe the damage to the visible light information.

3. The method according to claim 2, characterized in that, The pixel saturation response It is generated jointly by adaptive local thresholding and gradient decay, specifically as follows: Let the spatial size of the input visible light brightness image be... Define the local window radius and will The value range is limited to [3, 15]; for each pixel position (i, j), its local saturation threshold is... Based on the average brightness within the local window Compared with the global average brightness The weighted result is adaptively determined: ； in, The pixel saturation response The final expression is: ； Where H and W are the height and width of the visible light brightness image, respectively; Represents the local window, 1 [] For indicator functions, This represents the brightness gradient magnitude. This setting ensures that the pixel saturation response only occurs when the brightness within a local window exceeds an adaptive threshold and the neighborhood gradient tends to vanish. Achieve high values.

4. The method according to claim 2, characterized in that, The halo diffusion response The response is generated by parallel computation using a fixed multi-scale point diffusion kernel and taking the maximum value. Specifically: Three normalized Gaussian point diffusion kernels with different spatial scales are defined, and their standard deviations are respectively... , and All units are pixels; The three Gaussian kernels are respectively compared with the visible light brightness image. Convolution is performed to obtain multi-scale blurred responses. The halo diffusion response The final expression is: ； in, This setting selects the scale that most significantly increases brightness as the halo response of the current pixel, while utilizing... Remove positions that have already responded in the saturated core region to avoid redundant weighting.

5. The method according to claim 1, characterized in that, The infrared structural effectiveness weighting diagram Generate in the following way: Mean filtering is used to estimate the local background of infrared images. Extracting infrared images Relative to the local background The difference in forward thermal radiation, combined with the infrared gradient amplitude for edge enhancement, is expressed as follows: ； in, Represents the spatial smoothing kernel, This indicates element-wise multiplication. The edge enhancement coefficient, For infrared image gradient, Represents minimum-maximum normalization; the infrared structure effectiveness weighting graph The value is used to indicate the reliability of the presence of a significant infrared target structure or target boundary at the corresponding location. The higher the value, the more suitable the infrared structure at that location is for fusion.

6. The method according to claim 1, characterized in that, The cross-modal fusion control weight map The generation process includes a lightweight multi-scale processing strategy, specifically: First, the visible light degradation characterization weight map is... and the infrared structure effectiveness weighting diagram Min-max normalization was performed separately to unify the numerical range of both classes to the [0,1] interval. To reduce computational cost, bilinear interpolation was used to downsample the normalized weight maps to a fraction of their original spatial dimensions. Position-by-position coupling is performed in the low-resolution space: ； in, and These are the weighted images after downsampling; then... Median filtering is performed for noise reduction, and then guided upsampling is used to restore the original spatial dimensions to obtain the final cross-modal fusion control weight map. The guiding map uses the gradient edge map of the infrared image to maintain the sharpness of the control weight map at the target boundary; the cross-modal fusion control weight map High values ​​indicate that visible light information is impaired while the infrared structure is effective, while low values ​​indicate that visible light information is still usable or the infrared structure does not have a significant advantage.

7. The method according to claim 1, characterized in that, The shallow feature extraction and differential reweighting process includes a spatial registration fault-tolerant mechanism, specifically: Infrared shallow features are extracted using convolutional layers, normalization layers, nonlinear activation layers, depthwise separable convolutional layers, and pointwise convolutional layers, respectively. and visible light shallow features Before differential reweighting, infrared shallow features are... and visible light shallow features After concatenation along the channel dimensions, the data is fed into a spatial transformation network, which predicts a 2×3 affine transformation parameter. And for infrared shallow layer features Perform spatial transformation to obtain the registered and corrected shallow infrared features. To tolerate a global registration error of no more than 5 pixels between infrared and visible light images; During the training phase, a structural consistency regularization term is applied to the spatial transformation network: ； in, The learning-aware image patch similarity metric is used; after registration and correction are completed, the cross-modal fusion control weight map is used. Interpolate to the same spatial dimension as the shallow features, denoted as And perform differential reweighting according to the following formula: ； in, and The control coefficient is non-negative and adjustable. and These are the reweighted infrared and visible light features, respectively.

8. The method according to claim 1, characterized in that, The frequency domain decomposition is achieved through channel-by-channel Gaussian low-pass filtering, specifically: For reweighted infrared features and visible light characteristics Gaussian low-pass filtering is performed channel by channel to obtain low-frequency structural components. : ； in, A Gaussian low-pass kernel with a standard deviation set to 1 to 2 feature pixel scales; high-frequency texture components. Represented by the residual between the original features and the low-frequency structural components: ； The low-frequency structural components The high-frequency texture component is used to represent global brightness distribution, scene layout, and thermal target outlines. Used to represent edges, local textures, and fine-grained background details.

9. The method according to claim 1, characterized in that, The dynamic cross-modal attention interaction is achieved by constructing infrared dominant candidate features and visible light dominant candidate features for low-frequency structural components and high-frequency texture components respectively, and then continuously interpolating them according to control weights. Specifically: For low-frequency structural branches and high-frequency texture branches The query, key, and value are obtained using linear mapping, and infrared dominant candidate features are calculated using scaled dot product attention. and visible light dominant candidate features ; The cross-modal fusion control weight map Interpolate to the same spatial size as the current frequency branch feature, denoted as And perform position-by-position dynamic fusion according to the following formula: ； in, and These are low-frequency fusion features and high-frequency fusion features, respectively. when When the value approaches 1, the corresponding region increases the contribution of infrared candidate features. When the value approaches 0, the corresponding region increases the contribution of visible light candidate features, thereby enabling continuous and adjustable fusion of different regions in the same image based on local modal advantages using different interaction ratios; the cross-modal attention interaction adopts a grouped depth-separable attention mechanism, dividing the number of channels into 4 groups for independent calculation and then stitching them together, and setting the number of heads of multi-head attention to 2, compressing the number of feature channels to 32 dimensions to reduce the amount of inference computation.

10. The method according to claim 1, characterized in that, The total loss function of the regional differentiation fusion constraint Defined as: ； in, Strength constraint loss driven by the physical model For low-frequency structural consistency constraint loss, To preserve the constraint loss for high-frequency gradients, , and The weighting coefficients are positive, with initial weights set to 1, 1, and 10 respectively; the strength constraint loss The cross-modal fusion control weight map The low-value regions maintain the brightness balance between the fused image and the infrared and visible light images. The high-value regions guide the alignment of the intensity and low-frequency structure of the fused image with the infrared image; the low-frequency structure consistency constraint loss exist The high-value regions maintain the consistency between the low-frequency components of the fused image and the low-frequency components of the infrared image; the high-frequency gradient preservation constraint loss exist In low-value regions, edges and textures of the image with the larger gradient magnitude from the two source images are preferentially preserved. The gradient response of the infrared target edge is preferentially preserved in the high value region.

Images