Intelligent fusion calculation method for multi-modal image data

Abstract

The invention provides an intelligent fusion calculation method for multi-modal image data, and relates to the technical field of multi-modal image fusion, and the method comprises the steps: carrying out the foreground segmentation of an infrared and visible light image pair, and automatically generating a foreground mask; the extraction module captures rich semantic representations; cross-modal semantic feature interaction is realized; utilizing semantic information to dynamically adjust normalization parameters; the normalization parameters comprise a scaling parameter gamma and an offset parameter beta, so that the fusion features fully mine deep semantic information; restoring the spatial resolution of the deep features through bilinear interpolation to obtain a foreground segmentation image, and generating a fusion image through residual fidelity in combination with the shallow features of the semantic information; constructing an overall loss function to optimize the model; according to the method, the problem that a GIFNet class depends on underlying visual task supervision and has a limited promotion effect on a subsequent advanced visual task is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of multimodal image fusion technology, and in particular to an intelligent fusion calculation method for multimodal image data. Background Technology

[0002] Infrared and visible light images have modal complementarity. Visible light images are rich in texture and color details but are easily affected by weather and other factors, while infrared images are robust but have low resolution and weak texture details. Therefore, image fusion technology has been widely studied to improve its applicability to downstream computer vision tasks. Among the current deep learning-based fusion methods, GIFNet uses a three-branch structure based on the Swin-Transformer to achieve feature interaction through pixel-level supervision of low-level visual tasks such as digital photography fusion. Although it does not rely on abstract semantics and has wide applicability, its promotion effect on subsequent high-level visual tasks is limited. PSFusion, based on a CNN-designed two-branch structure, uses multiple high-level visual tasks to jointly supervise the network to extract semantic features, but it requires training new fusion models with specific features for different downstream tasks and preparing various labeled data for the dataset, resulting in weak generalization ability. In summary, existing infrared and visible light image fusion methods cannot balance versatility and adaptability to high-level visual tasks, and cannot meet the requirements of various downstream high-level visual tasks for fused images without retraining the model. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: According to a first aspect of this application, an intelligent fusion computing method for multimodal image data is provided, the method comprising the following steps: Step 1: Use a pre-trained U2-NetP saliency detection network to perform foreground segmentation on infrared and visible light image pairs and automatically generate foreground masks; Step 2: An improved ResNet is used as the backbone for feature extraction. The spatial resolution is preserved by removing the shallow feature extraction modules of the first few downsampling operations, and the rich semantic representation is captured by the deep feature extraction modules. Step 3: Use a reversible neural network to process shallow features to avoid information loss, and use modal attention to process deep features to achieve cross-modal semantic feature interaction; Step 4: Dynamically adjust the normalization parameters using semantic information; the normalization parameters include scaling parameter γ and offset parameter β, so that the fused features can fully extract deep semantic information; Step 5: The deep features are restored to obtain the foreground segmentation map by bilinear interpolation to restore the spatial resolution. The shallow features with semantic information are combined to generate a fused image by residual fidelity. At the same time, gradient loss, structural similarity loss and intensity loss are used to constrain the generation of the fused image. Cross-entropy loss is used to constrain the distinguishability between the foreground and the background, and higher loss weights are given to the foreground pixels. The overall loss function is constructed to optimize the model.

[0004] Furthermore, the average intersection-union ratio of the foreground mask generated by U2-NetP in step 1 and the manually labeled mask on the MSRS dataset reaches a preset value.

[0005] Furthermore, the improved ResNet described in step 2 only removes the downsampling operations of the first few layers in the shallow feature extraction module, while the deep feature extraction module retains the original downsampling logic to capture deep semantics.

[0006] Furthermore, the modal attention mentioned in step 3 is an attention mechanism that combines self-attention and cross-modal attention.

[0007] Furthermore, in step 5, the gradient loss is calculated using the Sobel operator to measure the L1 distance between the gradient of the fused image and the maximum gradient of the input image; the structural similarity loss is measured by normalized cross-correlation to determine the global structural consistency between the fused image and the input image; and the intensity loss is calculated by mean square error to enable the fused image to utilize the advantageous parts of the infrared and visible light images.

[0008] Furthermore, the overall loss function described in step 5 is a weighted sum of all loss terms, and the weight hyperparameters can be adjusted according to the requirements of downstream tasks to regulate the network's attention to the foreground target.

[0009] Furthermore, a preprocessing step is included before step 1: the training samples of the MSRS, M3FD, or VIRcraft dataset are randomly cropped and rotated into 256×256 image patches, the training epoch is set to 500 and the batch size to 24, SGD is used as the optimizer, and a warm-up strategy is used to adjust the initial learning rate.

[0010] Furthermore, the initial learning rate of the preheating strategy is a preset value of 10% of the total iteration steps, and the weight decay parameter is configured according to the training requirements.

[0011] Furthermore, the method is applicable to downstream advanced vision tasks such as semantic segmentation, object detection, and key point detection, without requiring retraining of the fusion model for different tasks.

[0012] The present invention has at least the following beneficial effects: The intelligent fusion computing method for multimodal image data of this invention can specifically solve the core problems existing in the prior art: First, by using a pre-trained U2-NetP saliency detection network to automatically generate foreground masks, there is no need for manual annotation of foreground data. At the same time, there is no need to retrain specific fusion models for different downstream advanced vision tasks such as semantic segmentation, object detection, and key point detection. This effectively overcomes the shortcomings of PSFusion-like methods, which rely on multiple manually annotated data, have weak generalization ability, and require repeated model training, thus significantly improving the model's versatility. Second, by constructing shallow feature extraction modules (SFEB) and deep feature extraction modules (DFEB) through an improved ResNet, combined with a reversible neural network, the method effectively addresses the core problems existing in the prior art. (INN) processes shallow features to avoid information loss, and modal attention (M-Att) extracts cross-modal semantic interaction from deep features. Combined with the injection of normalization parameters that dynamically adjust semantic information, and multi-dimensional loss constraints consisting of gradient loss, structural similarity loss, intensity loss, and cross-entropy loss (including foreground pixel weighting), it not only preserves image details and spatial resolution, but also enhances the mining of deep semantic information and the prominence of foreground targets. It effectively solves the problem that GIFNet-like systems rely on supervision from low-level visual tasks and have limited promotional effects on subsequent high-level visual tasks. Ultimately, it enables fused images to present better results in a variety of downstream high-level visual tasks, achieving a balance between versatility and adaptability to high-level visual tasks. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of an intelligent fusion calculation method for multimodal image data provided in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0017] The following will refer to Figure 1 The flowchart shown is for an intelligent fusion calculation method for multimodal image data, which introduces such a method.

[0018] The intelligent fusion calculation method for multimodal image data may include the following steps: Step 1: Use a pre-trained U2-NetP saliency detection network to perform foreground segmentation on infrared and visible light image pairs and automatically generate foreground masks.

[0019] Furthermore, the average intersection-union ratio (IUU-NetP) of the foreground mask generated by UU-NetP and the manually labeled mask on the MSRS dataset reaches a preset value.

[0020] In this embodiment, U2-NetP is a lightweight saliency detection network. In this step, the pre-trained U2-NetP model is used directly, eliminating the need to retrain the network for the infrared / visible light fusion task. During operation, infrared and visible light image pairs (such as day and night traffic scene image pairs from the MSRS dataset, road vehicle image pairs from the M3FD dataset, and aircraft fuselage image pairs from the VIRcraft dataset) are input into the pre-trained U2-NetP. This network extracts saliency features from the images through an encoder-decoder structure, distinguishing between foreground (e.g., cars, pedestrians, aircraft fuselages) and background (e.g., roads, sky, airport ground) regions, and outputting a binary foreground mask (pixel value 1 represents foreground, 0 represents background).

[0021] For example, on the MSRS dataset, the average intersection-union ratio of the mask generated in this way with the manually labeled mask can reach 70.72%, without the need for manual labeling of foreground data.

[0022] This step first addresses the core issue in existing technologies where PSFusion requires preparing various manually labeled data for different downstream tasks. Automatically generating foreground masks eliminates the high costs and time associated with manual labeling, significantly improving labeling efficiency. Simultaneously, as the driving task for fusion, the foreground mask eliminates the need to redesign labeling systems or train dedicated mask generation models for different downstream tasks such as semantic segmentation, object detection, and keypoint detection, overcoming the limitation of PSFusion's weak generalization ability. Furthermore, the lightweight nature of U2-NetP ensures computational efficiency in mask generation, preventing significant increases in the overall fusion method's time consumption due to additional segmentation steps, thus balancing practicality and versatility.

[0023] Step 2: An improved ResNet is used as the backbone for feature extraction. The spatial resolution is preserved by removing the shallow feature extraction modules of the first few downsampling layers, and the rich semantic representation is captured by the deep feature extraction modules.

[0024] In this embodiment, ResNet50 (or ResNet18) is selected as the basic feature extraction network and improved to construct a shallow feature extraction module (SFEB) and a deep feature extraction module (DFEB). For SFEB, downsampling operations are removed from the first 2-3 convolutional layers of ResNet; for example, the original stride of the convolutional layer with a stride of 2 is adjusted to a stride of 1, and the pooling layer is deleted. When processing 256×256 infrared / visible light image patches, the feature map size output by SFEB is still 256×256 (128×128 after conventional ResNet processing), thus preserving the texture details of the visible light image (such as the texture of car paint) and the target contours of the infrared image (such as the contours of pedestrian hot zones). For DFEB, the network structure of the last few layers of ResNet (including downsampling and residual blocks) is used. Deep semantic information is captured through multi-layer convolution and residual connections, such as identifying abstract semantic features such as "car", "pedestrian", and "aircraft key points" from the image. The residual connections avoid the gradient vanishing problem of deep networks.

[0025] This step addresses the issues of GIFNet's reliance on low-level visual task supervision leading to the loss of shallow details and insufficient extraction of high-level semantic features: SFEB removes downsampling operations, maximizing the preservation of spatial resolution and pixel-level details in both infrared and visible light images, laying the foundation for detail restoration in subsequent fused images and compensating for GIFNet's insufficient preservation of shallow features; DFEB effectively captures the semantic information of multimodal images, compensating for the weakness of semantic representation in infrared images and avoiding the problem of visible light images' semantics being easily affected by weather and lighting, enabling the extracted features to possess both detailed and semantic attributes, providing a high-quality feature foundation for subsequent cross-modal fusion, and significantly improving the adaptability of fused features to downstream high-level visual tasks.

[0026] Furthermore, the improved ResNet only removes the downsampling operations of the first few layers in the shallow feature extraction module, while the deep feature extraction module retains the original downsampling logic to capture deep semantics.

[0027] Taking the improved ResNet based on ResNet50 as an example, we first clarify the hierarchical division of the shallow feature extraction module (SFEB) and the deep feature extraction module (DFEB): the conv1 and conv2_x stages of ResNet50 are classified as SFEB (shallow), and the conv3_x, conv4_x, and conv5_x stages are classified as DFEB (deep).

[0028] For SFEB, only the downsampling operations of the first few layers within this module are removed / adjusted: ① Change the stride of the convolution kernel in the conv1 layer from the usual 2 to 1, and cancel the default 3×3 max pooling layer (stride 2) after the conv1 layer. ② Change the stride of the convolutional layer in the first residual block of the conv2_x stage from 2 to 1, delete the 1×1 convolutional layer used for downsampling in the residual block, and keep the other parameters such as the number of convolutional kernels and activation functions consistent with the native ResNet50.

[0029] Taking a 256×256 infrared / visible image block as an example, the resolution of the output feature map of the native ResNet50 after passing through conv1+maxpool+conv2_x will drop to 64×64, while the resolution of the shallow feature map output by the improved SFEB remains at 256×256, fully preserving pixel-level details.

[0030] For DFEB, the original downsampling logic of ResNet50 is completely adopted: the first residual block of the conv3_x, conv4_x, and conv5_x stages retains a convolutional layer with a stride of 2 and a 1×1 downsampling convolutional layer, and the feature map resolution is reduced from 256×256 to 128×128, 64×64, 32×32, and 16×16 respectively. The receptive field of the network is expanded by multiple downsampling, and the abstract deep semantic features such as "car category", "pedestrian outline" and "aircraft key points" are captured by combining residual connections, without modifying any downsampling-related network structure and parameters in DFEB.

[0031] This improvement addresses the core shortcomings of existing technologies: Firstly, SFEB removes the first few downsampling layers, maximizing the preservation of shallow details such as the target thermal outline in infrared images and the texture and color in visible light images. This avoids the problem of GIFNet losing details due to shallow downsampling, which affects the performance of downstream advanced vision tasks (such as keypoint detection). It also eliminates the need for multi-task supervision to compensate for insufficient details, unlike PSFusion. Secondly, DFEB maintains the original downsampling logic, reducing feature map dimensionality and computational overhead through multiple downsampling layers while ensuring the network has a sufficient receptive field to capture deep semantics. This not only compensates for the weak semantic representation of infrared images but also avoids the problems of insufficient semantic extraction capabilities and poor model generalization caused by the lack of deep downsampling. Furthermore, the layered downsampling design eliminates the need to modify the network structure for different downstream tasks such as semantic segmentation and object detection. A single improved ResNet can simultaneously meet the requirements of shallow detail preservation and deep semantic extraction, overcoming the limitation of existing methods that require customized feature extraction networks for different tasks, and significantly improving the model's versatility and adaptability.

[0032] Step 3: Use a reversible neural network to process shallow features to avoid information loss, and use modal attention to process deep features to achieve cross-modal semantic feature interaction.

[0033] In this embodiment, the shallow features output in step 2 are processed by a reversible neural network (INN): the shallow features (such as 256×256 visible light texture features and infrared target contour features) are split into two sub-feature blocks, which are then fused through reversible convolution and transformation operations. After fusion, the original shallow features can be restored through inverse transformation.

[0034] For example, when performing INN fusion on "car texture features" from visible light images and "car hot zone contour features" from infrared images, it integrates detailed information from both modalities without losing the original shallow features of either modality. For deep features, modal attention (M-Att) is used: M-Att combines self-attention and cross-modal attention. Self-attention focuses on the internal structure of single-modal deep features (such as the semantic association of the "car" region in an infrared image), while cross-modal attention focuses on the association between infrared and visible light deep features (such as the correspondence between visible light "car texture" and infrared "car hot zone"). By calculating attention weights, key semantic features are strengthened and redundant information is weakened, achieving cross-modal semantic interaction.

[0035] This step specifically addresses the problems of shallow feature fusion easily losing details and insufficient cross-modal interaction of deep semantic features in existing fusion methods: the reversibility of INN fundamentally avoids the loss of shallow detail features, ensuring the integrity of low-level visual information such as texture and contour of the fused image, and making up for the shortcomings of GIFNet in preserving low-level features; M-Att does not need to rely on multiple high-level task supervision to extract semantic features like PSFusion, and can achieve accurate cross-modal interaction of deep semantics through attention mechanism alone, so that the fused semantic features have both the robustness of infrared images and the richness of visible light images, which reduces the complexity of network training and improves the generality of the model.

[0036] Furthermore, the modal attention is an attention mechanism that combines self-attention and cross-modal attention.

[0037] Modal attention (M-Att) is composed of a self-attention submodule and a cross-modal attention submodule connected in series. The input is the infrared deep features (denoted as F) output by the deep feature extraction module (DFEB) in step 2. ir (Example dimension: 2048×16×16) and visible light depth features (denoted as F) vis (Example dimension: 2048×16×16), the specific implementation process is as follows: Self-attention submodule calculation: respectively for F ir and F vis Perform unimodal self-attention computation. Using F... ir For example, the feature map is first reshaped into a two-dimensional matrix of channel number × pixel number (C×N, N=H×W), and then a query matrix Q is generated through linear transformation. ir Key matrix K ir Value matrix V ir Then calculate the attention weights: Attention self-ir =Softmax(Q ir ×K T ir / d k 1 / 2 )(d k (where V is the dimension of the key matrix, used for normalization), and finally the weights are multiplied by V. ir Multiplying these results yields the infrared feature self-attention enhancement result F. ir-self Similarly, for F vis Complete the self-attention calculation and obtain F. vis-self This process focuses on semantic associations within a single modality, such as enhancing the distinction between "pedestrian hotspots" and the surrounding background in infrared features, or the continuity of "aircraft fuselage texture" in visible light features.

[0038] Cross-modal attention submodule computation: Centered on cross-modal interaction, constructing cross-modal query-key-value mapping relationships: ① Using F ir-self To query matrix Q, F vis-self Given the key matrix K and the value matrix V, calculate the cross-modal weights for Attention. cross-ir =Softmax(Q ir-self ×K T vis-self / d k 1 / 2 ), with V vis-self Multiplication yields the result F of infrared feature fusion with visible light semantics. ir-cross ; ② with F vis-self To query matrix Q, F ir-self Given the key matrix K and value matrix V, the result F obtained by fusing visible light features with infrared semantics is similarly obtained. vis-cross This process focuses on semantic complementarity between modalities. For example, it matches the infrared "car hotspot" feature with the location information of the visible light "car paint texture" and matches the visible light "pedestrian outline" feature with the semantic category information of the infrared "pedestrian hotspot".

[0039] Feature fusion output: The single-modal self-attention result and the cross-modal attention result are added element-wise to obtain the enhanced features (F) of each modality. ir-m =F ir-self +F ir-cross F vis-m =F vis-self +F vis-cross Then, by adding elements one by one or concatenating channels, the final cross-modal fusion deep feature \(F_{fusion}\) is output.

[0040] For example, in the aircraft keypoint detection scenario of the VIRcraft dataset, infrared deep features can only identify the "aircraft fuselage hot zone" but cannot distinguish the positions of the nine keypoints; visible light deep features can clearly present the keypoint textures but are easily affected by lighting conditions. Through a self-attention submodule, infrared features enhance the boundary semantics of the fuselage hot zone, and visible light features enhance the texture semantics of the keypoints; then, through a cross-modal attention submodule, infrared hot zone features are fused with visible light keypoint position information, and visible light texture features are fused with infrared fuselage semantic information, ultimately achieving F... fusion It possesses both the robustness of infrared and the detail precision of visible light.

[0041] This modal attention mechanism addresses the core issues of existing technologies by combining self-attention and cross-modal attention: the self-attention submodule strengthens the semantic representation within a single modality, compensating for the weak semantic details in infrared images and the susceptibility of visible light images to environmental interference (such as lighting and weather), eliminating the need for multiple high-level visual task supervision to extract semantic features like PSFusion; the cross-modal attention submodule enables precise interaction between deep semantics in infrared and visible light, allowing fused features to possess both the robustness of infrared and the richness of visible light, solving the problem of insufficient high-level semantic interaction and poor performance on downstream high-level tasks caused by GIFNet's reliance on low-level visual task supervision; simultaneously, the structure of this attention mechanism does not require adjustments for different downstream tasks such as semantic segmentation, object detection, and keypoint detection, allowing a single mechanism to adapt to multiple scenarios, overcoming the limitations of PSFusion, which requires training dedicated models for different tasks and has weak generalization ability; furthermore, the lightweight computational methods of self-attention and cross-modal attention eliminate the need for additional manual annotation or complex multi-task training processes, controlling the computational cost of the model while ensuring semantic extraction performance, further enhancing the practicality and versatility of the method.

[0042] Step 4: Dynamically adjust the normalization parameters using semantic information; the normalization parameters include scaling parameter γ and offset parameter β, so that the fused features can fully extract deep semantic information.

[0043] In this embodiment, semantic information (such as semantic representations of "pedestrian," "car," and "airplane keypoints") is first extracted from the deep fusion semantic features processed in step 3. This semantic information is then input into the parameter generation module, which outputs a normalized scaling parameter γ and an offset parameter β for the current image patch through a fully connected layer. For example, in a semantic segmentation task, when the image contains the semantic meaning of "pedestrian," the parameter generation module adjusts γ to increase the weight of pedestrian region features and adjusts β to offset the feature distribution of the pedestrian region. In an object detection task, γ and β are adjusted for the semantic meaning of "car" to enhance the feature contrast of the car region. Subsequently, γ and β are applied to the batch normalization (BN) operation of the deep and shallow fusion features, replacing the fixed parameters of traditional BN, and achieving dynamic adaptation of the normalization parameters.

[0044] This step addresses the problem of fixed normalization parameters in existing methods, which cannot adapt to different semantic scenarios: dynamically adjusted γ and β can adapt the normalization process according to the actual semantic content of the image, allowing the fused features to more accurately mine deep semantic information. For example, in the aircraft keypoint detection task of the VIRcraft dataset, after adjusting the parameters for "aircraft fuselage keypoints", the fused features of the keypoint region have higher contrast and clearer boundaries, improving the downstream detection accuracy. At the same time, this approach does not require designing exclusive normalization strategies for different downstream tasks. It can adapt to multiple scenarios simply by driving parameter adjustment through semantic information, further enhancing the model's versatility and solving the defect of GIFNet's poor performance in advanced tasks due to a lack of semantic adaptation.

[0045] Step 5: The deep features are restored to obtain the foreground segmentation map by bilinear interpolation to restore the spatial resolution. The shallow features with semantic information are combined to generate a fused image by residual fidelity. At the same time, gradient loss, structural similarity loss and intensity loss are used to constrain the generation of the fused image. Cross-entropy loss is used to constrain the distinguishability between the foreground and the background, and higher loss weights are given to the foreground pixels. The overall loss function is constructed to optimize the model.

[0046] Deep features are used to restore spatial resolution to obtain a foreground segmentation map, and shallow features combined with semantic information are used to generate a fused image through residual fidelity. ; (1) in For the generated fused image, This is a feature of fusion.

[0047] This invention employs gradient loss. Structural similarity loss and intensity loss The generation of images is based on common constraints. The gradient loss is defined as: ; (2) In the formula The Sobel operator loss preserves salient edge information by calculating the L1 distance between the gradient of the fused image and the maximum gradient of the input image. Structural similarity loss. The global structural consistency between the fused graph and the input image is measured using normalized cross-correlation, aiming to maximize feature correlation. An intensity loss is also designed to ensure the fused graph utilizes the "advantageous" parts of both inputs as much as possible. ; (3) In the formula Indicates mean square error. and These represent the height and width of the image, respectively.

[0048] Meanwhile, cross-entropy loss is used to constrain the distinguishability between foreground and background: ; (4) in It is a binary mask that labels each pixel as belonging to the foreground or background. This represents the foreground probability predicted by the model. To emphasize salient foreground regions, pixels marked as foreground in the mask are assigned higher loss weights. The overall loss function is: (5) in These are the weight hyperparameters for each loss term. For mask weights, This indicates pixel-by-pixel multiplication.

[0049] In this embodiment, the deep semantic features (e.g., 16×16 resolution) processed in step 4 are first bilinearly interpolated to restore them to a resolution of 256×256, consistent with the shallow features, to obtain a foreground segmentation map. Then, a residual fidelity module is constructed. The shallow features combined with semantic information and the foreground segmentation map are input into this module. The final fused image is generated through residual connection (fused image = shallow features + residual term). The residual term is used to compensate for the deep and shallow fusion error (e.g., to compensate for the lack of aircraft fuselage details caused by the low resolution of infrared images).

[0050] The loss function is constructed as follows: Gradient loss is calculated using the Sobel operator to measure the L1 distance between the gradient of the fused image and the maximum gradient of the input image (e.g., the gradient of the car edge in the fused image needs to be close to the gradient of the visible light image edge); Structural similarity loss is measured by normalized cross-correlation to measure the global structural consistency between the fused image and the input image; Intensity loss is calculated by mean square error to make the fused image take into account both the brightness features of the infrared image and the color features of the visible light image; Cross-entropy loss is calculated for the foreground segmentation map and the ground truth mask, and foreground pixels (e.g., key points of pedestrians and airplanes) are assigned a loss weight of 2 times that of the background; Finally, the overall loss function is a weighted sum of the loss terms (e.g., gradient loss weight 0.3, structural similarity loss weight 0.4, intensity loss weight 0.1, and cross-entropy loss weight 0.2).

[0051] This step addresses the problems of poor detail restoration, lack of foreground object prominence, and poor performance on advanced tasks caused by a single loss constraint in existing methods: Bilinear interpolation restores the resolution of deep features, and the residual fidelity module ensures both the spatial resolution of the fused image and reduces feature fusion errors through residual compensation, allowing the fused image to possess the detail advantages of both infrared and visible light images; the design of multi-dimensional loss functions includes gradient loss to preserve significant image edges, structural similarity loss to ensure global structural consistency, intensity loss to maximize the advantages of both modalities, and cross-entropy loss (including foreground weighting) to enhance the distinction between foreground and background, making foreground objects (such as pedestrians and aircraft keypoints) more prominent in the fused image. It is adaptable to various downstream advanced vision tasks such as semantic segmentation, object detection, and keypoint detection, without the need to design dedicated loss functions for different tasks like PSFusion, thus balancing versatility and task performance, and solving the core problem of GIFNet's poor performance in advanced vision tasks.

[0052] Furthermore, the gradient loss is calculated using the Sobel operator to measure the L1 distance between the gradient of the fused image and the maximum gradient of the input image; the structural similarity loss is measured by normalized cross-correlation to assess the global structural consistency between the fused image and the input image; and the intensity loss is calculated by mean square error to enable the fused image to utilize the advantageous portions of the infrared and visible light images.

[0053] Furthermore, the overall loss function is a weighted sum of all loss terms, and the weight hyperparameters can be adjusted according to the requirements of downstream tasks to regulate the network's attention to the foreground target.

[0054] 1. Calculation and Implementation of Gradient Loss: Gradient loss uses the Sobel operator as its core tool, focusing on preserving the salient edges and contour information of the image. The specific steps are as follows: ① Prepare the Sobel horizontal operator (Sobel_x, such as a 3×3 matrix: [-1 0 1;-20 2;-1 0 1]) and the vertical operator (Sobel_y, such as a 3×3 matrix: [-1 -2 -1;0 0 0;1 2 1]); ② Process the input infrared image (I ir Visible light images (I) vis ) and the generated fused image (I fusion ① Perform convolution operations to extract the gradients of the infrared image, the visible light image, and the fused image; ② Determine the maximum gradient of the input image pixel by pixel; ③ Calculate the L1 distance as the gradient loss.

[0055] For example, in the nighttime traffic scene of the MSRS dataset, the gradient value of "car paint texture edge" in the visible light image is high (clear texture), and the gradient value of "pedestrian hot zone contour" in the infrared image is high (not affected by dark light). The maximum gradient will select the visible light gradient at the car pixel position and the infrared gradient at the pedestrian pixel position. The gradient loss constrains the gradient of the fusion map to be consistent with the maximum gradient at these positions, ensuring that the fusion map has both clear car texture edges and robust pedestrian contours.

[0056] 2. Implementation of Structural Similarity Loss Calculation The structural similarity loss is improved based on normalized cross-correlation (NCC), and the core metric is the global structural consistency between the fused image and the input image. The specific steps are as follows: ① Calculate local statistical features in blocks: I fusion I ir I vis Divide the area into several non-overlapping local windows (e.g., 8×8 pixels), and calculate the mean, standard deviation, and covariance for each window; ② Calculate the single-modal structural similarity (SSIM); ③ Calculate the structural similarity loss: L ssim .

[0057] For example, in the road vehicle detection scenario of the M3FD dataset, the visible light image presents a "spatial arrangement structure of multiple vehicles in parallel", and the infrared image presents a "distribution structure of vehicle hot zones". The structural similarity loss constraint fusion map neither distorts the spatial arrangement of vehicles nor disrupts the distribution of hot zones, thus avoiding structural errors such as "vehicle overlap" and "hot zone shift".

[0058] 3. Calculation and implementation of strength loss The intensity loss is designed based on mean squared error (MSE). Its core principle is to maximize the utilization of the advantages of infrared and visible light in the pixel intensity (grayscale / brightness value) of the fused image. The specific steps are as follows: ① Define the intensity advantage rule: Infrared images have more stable pixel intensity in low-light, foggy, and rainy conditions (e.g., pedestrian hotspots at night), while visible light images have richer pixel intensity under normal lighting conditions (e.g., vehicle textures during the day); ② Calculate the intensity loss: L intensity =(1 / (H×W))∑ H i=1 ∑ W j=1 (I) fusion (i,j)-max(I ir (i, j), I vis (I,j))) 2 (Weighting can also be applied based on the scene, such as giving infrared intensity a weight of 0.7 and visible light a weight of 0.3 in low-light scenes).

[0059] For example, in an airport scene under heavy fog (VIRcraft dataset), the intensity value of the "aircraft fuselage texture" in the visible light image is low due to fog obscuring, while the intensity value of the "aircraft hot zone" in the infrared image is stable. The intensity loss constraint fusion map has an intensity value close to the infrared value at the fuselage pixel position and close to the visible light value at the airport ground position without fog, maximizing the absorption of the intensity advantages of the two modes.

[0060] The three losses are ultimately weighted according to a preset weight (e.g., L). gradient Weight 0.3, L ssim Weight 0.4, L intensity A weight of 0.1 is included in the overall loss function to jointly constrain the generation of fused images.

[0061] The collaborative design of these three loss functions specifically addresses the core shortcomings of existing technologies: Gradient loss uses the Sobel operator to lock in the optimal edge / contour features of the input image, compensating for the problems of blurred edges in fused images and low localization accuracy in downstream high-level tasks (such as keypoint detection) caused by GIFNet's reliance on low-level visual task supervision; Structural similarity loss ensures global structural consistency through normalized cross-correlation, avoiding the structural distortion and weak generalization ability of fused images caused by multi-task supervision in PSFusion; Intensity loss maximizes the intensity advantages of infrared and visible light through mean squared error, solving the shortcomings of low resolution and weak texture of infrared images while avoiding the defects of visible light images affected by lighting / weather interference; At the same time, the three types of losses do not require adjustment of the calculation logic for different downstream tasks such as semantic segmentation, object detection, and keypoint detection. A single loss system can be adapted to multiple scenarios, eliminating the need to design dedicated loss functions for different tasks like PSFusion, significantly improving the model's versatility; In addition, the loss calculation is based on pixel-level / Local statistical features eliminate the need for additional manual annotation or complex semantic parsing processes, ensuring fusion results while controlling computational costs. This allows fused images to retain both detail and structural integrity, while also being adaptable to various downstream advanced vision tasks, overcoming the limitations of existing methods that struggle to balance detail preservation, structural consistency, and multi-task adaptability.

[0062] Furthermore, a preprocessing step is included before step 1: the training samples of the MSRS, M3FD, or VIRcraft dataset are randomly cropped and rotated into 256×256 image patches, the training epoch is set to 500 and the batch size to 24, SGD is used as the optimizer, and a warm-up strategy is used to adjust the initial learning rate.

[0063] The preprocessing steps perform standardized operations on the MSRS, M3FD, and VIRcraft datasets, and are mainly divided into two parts: data augmentation and training hyperparameter configuration. ① Data augmentation: First, the original dataset samples are read (e.g., 1444 pairs of infrared / visible light images for MSRS, 4200 pairs of images for M3FD, and 216 pairs of images for VIRcraft). Synchronous random cropping and rotation are performed on each pair of images. During random cropping, a 256×256 pixel region is randomly selected from the original image (e.g., MSRS original size 512×512, M3FD original size 640×480) to ensure that the cropping region coordinates of the infrared and visible light images are completely consistent. During random rotation, 0°, 90°, 180°, or 270° rotation operations are randomly applied to the cropped image blocks, and the rotation angles of the infrared and visible light images are synchronized to avoid modal misalignment. For example, for aircraft fuselage image pairs with an original size of 400×400 in the VIRcraft dataset, cropping to 256×256 and then randomly rotating by 90° preserves the core features of the aircraft key points while increasing sample diversity. ② Training hyperparameter configuration: The enhanced image patches are used as training samples, and the number of training epochs is set to 500, meaning all training samples complete 500 full iterations; the batch size is set to 24, meaning 24 sets of infrared / visible light image patches are input simultaneously in each iteration; the optimizer is stochastic gradient descent (SGD), with a default momentum parameter of 0.9 (which can be fine-tuned according to downstream tasks, such as setting the momentum to 0.95 for key point detection tasks), and a warm-up strategy is started before the SGD optimizer is executed to adjust the initial learning rate.

[0064] This preprocessing step specifically addresses the core issues of existing image fusion methods, such as limited training samples, model overfitting, and poor cross-dataset adaptability. The simultaneous enhancement operation of random cropping and rotation avoids feature extraction bias caused by inconsistent sample sizes in the original dataset (e.g., directly inputting 640×480 images from M3FD and 400×400 images from VIRcraft can lead to inconsistent feature map sizes). Furthermore, by increasing the number of sample variants, it mitigates the overfitting risk of small datasets like VIRcraft (only 216 pairs). A uniform epoch setting of 500 ensures sufficient convergence of the model across all three datasets, avoiding the feature map distortion issues caused by insufficient training epochs in GIFNet and PSFusion. The problem of insufficient extraction is addressed; the batch size of 24 balances memory usage and training efficiency (improving iteration speed compared to batch size of 16 and reducing memory pressure compared to batch size of 32), adapting to the training needs of ordinary computing devices; the choice of the SGD optimizer is more suitable for image fusion convolutional / attention networks than optimizers such as Adam, effectively avoiding gradient oscillations in the later stages of training and improving model convergence stability; at the same time, the entire preprocessing process does not require adjusting the logic for different downstream tasks such as semantic segmentation (MSRS), object detection (M3FD), and keypoint detection (VIRcraft), ensuring the uniformity of model training and further enhancing the versatility of the method.

[0065] Furthermore, the initial learning rate of the preheating strategy is a preset value of 10% of the total iteration steps, and the weight decay parameter is configured according to the training requirements.

[0066] The warm-up strategy, as the pre-learning rate adjustment mechanism of the SGD optimizer, is mainly divided into two parts: learning rate warm-up and weight decay configuration. ① Learning rate warm-up calculation: First, calculate the total number of iterations (total number of iterations = total number of training samples ÷ batch size × epoch). For example, taking the MSRS dataset with 1000 pairs of training samples as an example, the total number of iterations = 1000 ÷ 24 × 500 ≈ 20833 steps. Set 10% of the total number of iterations (about 2083 steps) as the warm-up stage. The initial learning rate of this stage is a preset value (e.g., the initial lr in the warm-up stage is 0.001, and the base lr in the non-warm-up stage is 0.01). During the warm-up stage, the learning rate increases linearly from the preset value to the base learning rate. After the warm-up is over, it is restored to the base learning rate and decayed according to the normal strategy (e.g., the learning rate is halved every 100 epochs). ② Weight Decay Parameter Configuration: The weight decay parameter (Weight Decay) is flexibly adjusted according to the needs of downstream tasks and has no fixed value: For semantic segmentation tasks (MSRS), the weight decay is set to 1e-4 to strengthen the regularization of deep semantic features; for object detection tasks (M3FD), the weight decay is set to 5e-5 to avoid excessive regularization leading to object bounding box localization deviation; for keypoint detection tasks (VIRcraft), the weight decay is set to 1e-5 to prioritize the integrity of pixel-level features of keypoints.

[0067] This preheating strategy specifically addresses the problems of gradient explosion, unstable model convergence, and poor cross-task adaptability in existing methods during the initial training phase. By setting the preheating stage to 10% of the total iterations and using a low initial learning rate, it avoids gradient explosion or model parameter oscillations caused by excessively high learning rates in the early training phase. This allows the model to adapt to preprocessed standardized samples before entering the regular training phase. Compared to PSFusion's training method without a preheating strategy, the model convergence speed is improved by approximately 20%, and the fluctuation of the loss function is reduced. The weight decay parameter is flexibly configured according to downstream tasks, which solves the overfitting / ... It overcomes the underfitting problem (such as overfitting when the fixed 1e-4 is fixed on the VIRcraft small sample dataset), and does not require redesigning the optimizer framework for different tasks. It can adapt to scenarios such as semantic segmentation, object detection, and key point detection by adjusting only a single parameter. In addition, the combination of the warm-up strategy and the SGD optimizer further ensures that the model maintains stable gradient updates throughout 500 epochs of training, avoiding the decrease in accuracy in later training. Ultimately, the fusion model can achieve optimal convergence in downstream tasks of the three types of datasets, breaking through the limitation of existing methods that are difficult to balance the stability and accuracy of multi-task training.

[0068] Furthermore, the method is applicable to downstream advanced vision tasks such as semantic segmentation, object detection, and key point detection, without requiring retraining of the fusion model for different tasks.

[0069] The core of this method, which eliminates the need to retrain the fusion model for different downstream tasks, lies in its task-free design with foreground segmentation as the general driving task. During the model training phase, training is completed solely around the general objective of "infrared-visible light image fusion + foreground-background differentiation." After training, it can be directly adapted to three types of downstream advanced vision tasks: semantic segmentation, object detection, and keypoint detection. The entire process requires no modification to the fusion model's network structure or parameters, or retraining. The specific adaptation process is as follows: 1. Adapted for semantic segmentation tasks (MSRS dataset) The MSRS dataset contains 1444 pairs of infrared / visible image pairs, labeled with 9 semantic categories (Car, Person, Bike, etc.). The core requirement is to distinguish the pixel affiliation of different semantic categories. The fused image output from the trained fusion model can be directly input into a general semantic segmentation network (such as DeepLabV3+ or UNet) without any adjustments to the fusion model. The fused image retains visible light texture details (such as the paint texture of a Car) and robust infrared semantics (such as the thermal outline of a Person), and the semantic boundaries are clear thanks to foreground segmentation. The semantic segmentation network can directly extract semantic features for each category based on the fused image. For example, on the MSRS dataset, the output image of the fusion model, which was not retrained for semantic segmentation, achieved an mIoU of 69.89% after inputting into DeepLabV3+, significantly higher than methods such as PSFusion (67.44%) and GIFNet (63.65%), which require fine-tuning for semantic segmentation.

[0070] 2. Adapted for object detection tasks (M3FD dataset) The M3FD dataset contains 4200 image pairs labeled with bounding boxes for objects such as car, truck, and person. The core requirement is to locate and identify the object category. The fused image output from the fusion model can be directly input into a general object detection network (such as YOLOv8 or Faster R-CNN) without retraining the fusion model. The fused image preserves the edge contours of the objects (such as the border of a car) through gradient loss, maximizes the use of infrared target hotspots (the hotspot of a person in foggy / nighttime scenes) through intensity loss, and enhances the distinction between the target and the background through foreground segmentation. The detection network can directly perform bounding box regression and category classification based on the fused image. For example, on the M3FD dataset, the output image of the fusion model that has not been retrained for object detection, when input into YOLOv8, achieves a detection accuracy of 0.553 for the "all" category, which is better than methods such as PSFusion (0.551) and GIFNet (0.543) that require adaptation to the detection task.

[0071] 3. Adaptation for keypoint detection tasks (VIRcraft dataset) The VIRcraft dataset contains 216 image pairs labeled with 9 keypoints on the aircraft fuselage. The core requirement is accurate pixel-level keypoint localization. The fused image output from the fusion model is directly input into a general keypoint detection network (such as HRNet or Hourglass) without retraining the fusion model. The fused image integrates infrared fuselage semantics (avoiding illumination interference) and visible light keypoint textures (ensuring localization accuracy) through modal attention. Foreground segmentation enhances the features of the foreground region where the keypoints are located. The detection network can directly perform keypoint coordinate regression based on the fused image. For example, on the VIRcraft dataset, the output image from the fusion model without keypoint detection retraining achieves an mAP [50,95] of 66.18% after inputting into HRNet, higher than methods requiring specific adjustments such as PSFusion (65.44%) and GIFNet (64.85%).

[0072] Technically, the core support for this adaptability lies in: ① Foreground segmentation, as a general tractor task, covers the core requirements of three types of downstream tasks (semantic segmentation requires category boundaries, object detection requires object-background differentiation, and keypoint detection requires foreground region localization), without needing to customize tractor targets for a single task; ② Deep and shallow feature extraction, modal attention, and multi-dimensional loss functions are all general designs (e.g., modal attention does not bind to task-specific features such as "semantic category," "object box," and "keypoint"), and the extracted fusion features simultaneously meet the feature requirements of three types of tasks; ③ Standardization of training steps such as preprocessing and optimizer configuration ensures that the fusion model has "general feature representation capabilities" after training, rather than being only adapted to a certain type of task.

[0073] This feature specifically addresses the core pain points of existing technologies: This method addresses the issue of "weak generalization ability" in PSFusion-like methods: PSFusion requires training a dedicated fusion model for different downstream tasks and preparing corresponding labeled data for each task (e.g., category labeling for semantic segmentation and bounding box labeling for object detection). In contrast, this method only requires one training iteration to adapt to three types of tasks, without the need for additional labeling or retraining, reducing model training costs (time / computing power / labeling) by approximately 60% and significantly improving its practicality in engineering. This method addresses the issue of "poor adaptability to high-level tasks" in GIFNet-like methods: GIFNet relies on low-level visual task supervision and can only adapt to a small number of low-level tasks, with limited benefits for high-level tasks such as semantic segmentation and object detection. However, this method improves performance in all three types of high-level tasks through generalized feature design, overcoming the limitation that "low-level task supervision cannot adapt to high-level tasks". Improved engineering deployment efficiency: In practical applications (such as intelligent transportation and airport security), a single fusion model can simultaneously support multiple tasks such as "road semantic segmentation", "vehicle / pedestrian detection" and "aircraft key point localization", without the need to deploy multiple dedicated fusion models, thus reducing hardware resource consumption and maintenance costs; It retains the flexibility of task adaptation: even if the downstream task scenario changes (such as adding "non-motorized vehicle detection" and "building key point detection"), only the downstream task network needs to be adjusted (such as the number of categories in the detection network), and the fusion model still does not need to be retrained, which further enhances the versatility and scalability of the method.

[0074] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0075] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. An intelligent fusion calculation method for multimodal image data, characterized in that, The method includes the following steps: Step 1: Use a pre-trained U2-NetP saliency detection network to perform foreground segmentation on infrared and visible light image pairs and automatically generate foreground masks; Step 2: An improved ResNet is used as the backbone for feature extraction. The spatial resolution is preserved by removing the shallow feature extraction modules of the first few downsampling operations, and the rich semantic representation is captured by the deep feature extraction modules. Step 3: Use a reversible neural network to process shallow features to avoid information loss, and use modal attention to process deep features to achieve cross-modal semantic feature interaction; Step 4: Dynamically adjust the normalization parameters using semantic information; the normalization parameters include scaling parameter γ and offset parameter β, so that the fused features can fully extract deep semantic information; Step 5: The deep features are restored to obtain the foreground segmentation map by bilinear interpolation to restore the spatial resolution. The shallow features with semantic information are combined to generate a fused image by residual fidelity. At the same time, gradient loss, structural similarity loss and intensity loss are used to constrain the generation of the fused image. Cross-entropy loss is used to constrain the distinguishability between the foreground and the background, and higher loss weights are given to the foreground pixels. The overall loss function is constructed to optimize the model.

2. The method according to claim 1, characterized in that, The average intersection-union ratio of the foreground mask generated by U2-NetP in step 1 and the manually labeled mask on the MSRS dataset reaches the preset value.

3. The method according to claim 1, characterized in that, The improved ResNet described in step 2 only removes the downsampling operations of the first few layers in the shallow feature extraction module, while the deep feature extraction module retains the original downsampling logic to capture deep semantics.

4. The method according to claim 1, characterized in that, The modal attention mentioned in step 3 is an attention mechanism that combines self-attention and cross-modal attention.

5. The method according to claim 1, characterized in that, The gradient loss described in step 5 calculates the L1 distance between the gradient of the fused image and the maximum gradient of the input image using the Sobel operator. The structural similarity loss measures the global structural consistency between the fused image and the input image using normalized cross-correlation. The intensity loss uses mean squared error to enable the fused image to utilize the advantageous parts of the infrared and visible light images.

6. The method according to claim 1, characterized in that, The overall loss function described in step 5 is a weighted sum of all loss terms. The weight hyperparameters can be adjusted according to the requirements of downstream tasks to regulate the network's attention to the foreground target.

7. The method according to claim 1, characterized in that, Before step 1, a preprocessing step is also included: randomly cropping and rotating the training samples of the MSRS, M3FD or VIRcraft dataset into 256×256 image patches, setting the training epoch to 500 and the batch size to 24, using SGD as the optimizer, and using a warm-up strategy to adjust the initial learning rate.

8. The method according to claim 7, characterized in that, The initial learning rate of the preheating strategy is a preset value of 10% of the total number of iterations, and the weight decay parameter is configured according to the training requirements.

9. The method according to claim 1, characterized in that, The proposed method is applicable to downstream advanced vision tasks such as semantic segmentation, object detection, and keypoint detection, without requiring retraining of the fusion model for different tasks.

Images