Multi-modal image fusion model construction method

Abstract

The invention discloses a multi-modal image fusion model construction method, and particularly relates to the technical field of image fusion model construction. Interactive modeling of a modal contribution imbalance coefficient and a confidence coefficient estimation anomaly coefficient is introduced in a multi-modal image fusion process; a causal association between confidence prediction fluctuation and modal weight extreme is converted into a quantifiable cross-modal complementarity degeneration index, and the cross-modal complementarity degeneration index is used as a core adjustment mechanism to dynamically optimize a fusion weight and training regularization, so that the problem that a single mode is excessively amplified or mistakenly weakened is effectively inhibited on a fusion strategy level; according to the method, the structure retention capability of the model in edge details and texture regions is remarkably improved, and artifacts and information loss caused by modal complementarity degradation are avoided.

Description

Technical Field

[0001] This invention relates to the field of image fusion model construction technology, and more specifically, to a method for constructing a multimodal image fusion model. Background Technology

[0002] In existing multimodal image fusion models, a confidence prediction mechanism is typically used to weight features from different modalities in order to dynamically balance the contributions of each modality during the fusion process. However, in practice, the estimation of the confidence branch often exhibits instability. When the confidence prediction fluctuates abnormally or converges to extreme values, it directly leads to an imbalance in modal weight allocation, resulting in one modality being over-amplified while another is incorrectly weakened. Since the modal contribution itself depends entirely on the adjustment of the confidence factor, this estimation anomaly and contribution imbalance are not independent but rather exhibit a closely logical causal relationship: incorrect confidence prediction directly leads to extreme modal weights, thereby destroying the complementarity of cross-modal features. As a result, the fused image gradually degenerates into an approximate unimodal representation, losing the edge enhancement and detail compensation advantages brought by multimodal collaboration, and is more prone to artifacts and information loss in textured regions and structural edges. This "cross-modal complementarity degradation problem" caused by the interaction of confidence anomalies and modal contribution imbalance has become a core defect restricting the effectiveness of multimodal image fusion models. Summary of the Invention

[0003] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a method for constructing a multimodal image fusion model to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: A method for constructing a multimodal image fusion model includes the following steps: Radiometric normalization and geometric alignment are performed on the original multimodal images to generate a multimodal image set with uniform spatial and numerical scales; Multi-scale feature encoding is performed on a multimodal image set. Multi-scale features of each modality are extracted through residual convolutional networks and pyramid structures. Feature alignment and interoperability are achieved between scales to form a multi-scale feature group. Multi-scale feature groups are input into the cross-modal interaction module. The complementary information between modalities is enhanced through the attention mechanism to generate interaction features. Pixel-level confidence estimation and uncertainty prediction are performed on the interaction features to form stable confidence information. Modal contribution weights are calculated by combining interaction features and confidence information, and modal contribution imbalance coefficients are generated based on the weights. At the same time, confidence estimation anomaly coefficients are calculated based on the confidence distribution. The cross-modal complementarity degradation index is calculated by interactively weighting the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, thereby quantifying the degree of degradation of the fusion effect due to the interaction between the two. The modal fusion weights and training regularization are dynamically optimized based on the cross-modal complementarity degradation index to obtain optimized fusion features. The fusion features are then mapped to the final high-quality fusion image through a convolutional decoding network.

[0005] In a preferred embodiment, the multimodal image set is processed by its respective modality-specific residual coding backbone, and stacked residual units are used to preserve details and stabilize gradients to generate a primary feature map set for each modality. ; The initial feature map set is used to construct a multi-scale representation through pyramid downsampling and scale residual iteration. After downsampling at each scale, several residual blocks are appended to form a pyramid feature map set with scales s=1...S. ; The pyramid features are projected onto a unified channel baseline at each scale using channel projection and modality normalization. First, a 1×1 convolution projects each modality feature onto a unified channel dimension D. Then, modality-aware normalization is performed on the projected features to generate unified channel features at each scale. ; The unified channel features are aligned across modal spaces at each scale, and the aligned features and alignment confidence maps at each scale are output. ,in For alignment features, To align confidence levels; The alignment features and alignment confidence maps are jointly enhanced intra-scale and inter-scale. A lightweight self-attention mechanism is used to calculate weighted inter-modal complementarity and cross-scale information transfer along top-down and bottom-up paths, generating interactive enhancement features for each scale. It also outputs the complementary confidence map for each scale. ; The interactive enhancement features are weighted at each scale by joint channel and spatial attention and fused using adaptive normalization, with channel weights generated using channel attention. And in combination with spatial attention Calculate the final weighted features and synthesize the final multi-scale feature set according to the weighted summation rule. .

[0006] In a preferred embodiment, multi-scale feature groups The input cross-modal feature interaction module performs channel linear projection on different modal features at each scale to form Query, Key, and Value tensors. , , ; The Query, Key, and Value tensors are used to perform inter-modal self-attention computation at each scale. The similarity scores are normalized using Softmax and weighted by the Value tensor to output the preliminary interaction features at each scale. ; Interaction features at each scale The input is a multi-scale fusion network, which fuses features from both top-down and bottom-up information flows layer by layer to obtain enhanced features that simultaneously possess local details and global semantics. ; For the enhanced features Pixel-level confidence estimation is performed by predicting the modal confidence per pixel using a small convolutional subnetwork. The confidence uncertainty is calculated based on local variance, cross-modal consistency, and characteristic residuals. ; The confidence and uncertainty information at each scale are smoothly fused within and between scales, and a stable confidence map is generated by combining the spatial neighborhood mean and scale weighting factor. With stability uncertainty diagram ; Integrating stable confidence and uncertainty plots with interactive enhancement features Joint encapsulation forms the final cross-modal interaction features. and stable confidence information , .

[0007] In a preferred embodiment, cross-modal interaction features With stable confidence plot The joint input modality contribution weight calculation module weights the feature importance of each modality through channel and spatial attention mechanisms, generating a pixel-level contribution weight map for each modality. ; Pixel-level contribution weight Perform intra-scale statistical analysis to calculate the average contribution of each mode. and standard deviation ; Average contribution value of each mode After inter-scale normalization, the modal contribution imbalance coefficient is calculated. This is used to quantify the skewness of the modal contribution distribution: ; Based on stable confidence plot Statistical analysis was performed on the confidence distribution at each scale to calculate the confidence estimation anomaly coefficient. : ,in This is a very small constant to prevent division by zero; Modal contribution imbalance coefficient Estimating the anomaly coefficient with confidence level Encapsulated as a quantitative indicator of cross-modal complementarity degradation.

[0008] In a preferred embodiment, the cross-modal complementarity degradation index is calculated by cross-weighting the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, as shown in the following formula: In the formula It is a cross-modal complementarity degradation index. Contributes imbalance coefficient to modalities. To estimate the outlier coefficient for confidence level, represent the preset proportionality coefficients of the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, respectively, and All are greater than 0.

[0009] In a preferred embodiment, the cross-modal complementarity degradation index is combined with the cross-modal interaction feature. and stable confidence information The input is an adaptive mapper that generates a set of adjustment signals based on the cross-modal complementarity degradation index: a temperature factor per scale. Weight upper limit threshold Smoothing coefficient set of regular coefficients ; The adjustment signal is compared with the previously obtained pixel-level contribution weight. and the corresponding uncertainty Combine, calculate the updated normalized fusion weights element by element. ; The updated fusion weights Apply pixel-by-pixel to the corresponding cross-modal interaction features Multi-scale fusion features are generated by weighted summation in the modality dimension. ; The fusion features The input feature refinement module performs three parallel techniques on each scale and fuses the results, outputting the optimized fused features. ; The optimized fusion features The input is a multi-scale convolutional decoder, which uses an upsampling + skip connections + attention gate structure to reconstruct the image layer by layer, generating an initial fused image. ; The initial fused image The image is fed into the sensor-level post-processing pipeline and outputs the final fused image. .

[0010] The technical effects and advantages of this invention are as follows: 1. This invention introduces interactive modeling of modal contribution imbalance coefficient and confidence estimation anomaly coefficient during multimodal image fusion. It transforms the causal relationship between confidence prediction fluctuation and modal weight extremism into a quantifiable cross-modal complementarity degradation index. Using this as the core adjustment mechanism, it dynamically optimizes the fusion weights and training regularization. This effectively suppresses the problem of excessive amplification or erroneous weakening of a single modality at the fusion strategy level. This method not only significantly improves the model's ability to preserve structure in edge details and texture regions and avoids artifacts and information loss caused by modal complementarity degradation, but also exhibits stronger stability and robustness in complex scenes. The fusion results can fully reflect the synergistic advantages of multimodal fusion, achieving a comprehensive improvement in image clarity, contrast, and information integrity. This provides higher quality and more reliable support for applications such as intelligent perception, medical diagnosis, and remote sensing interpretation of multimodal images. Attached Figure Description

[0011] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings; Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Example: Figure 1 This invention presents a method for constructing a multimodal image fusion model, comprising the following steps: Radiometric normalization and geometric alignment are performed on the original multimodal images to generate a multimodal image set with uniform spatial and numerical scales; Multi-scale feature encoding is performed on a multimodal image set. Multi-scale features of each modality are extracted through residual convolutional networks and pyramid structures. Feature alignment and interoperability are achieved between scales to form a multi-scale feature group. Multi-scale feature groups are input into the cross-modal interaction module. The complementary information between modalities is enhanced through the attention mechanism to generate interaction features. Pixel-level confidence estimation and uncertainty prediction are performed on the interaction features to form stable confidence information. Modal contribution weights are calculated by combining interaction features and confidence information, and modal contribution imbalance coefficients are generated based on the weights. At the same time, confidence estimation anomaly coefficients are calculated based on the confidence distribution. The cross-modal complementarity degradation index is calculated by interactively weighting the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, thereby quantifying the degree of degradation of the fusion effect due to the interaction between the two. The modal fusion weights and training regularization are dynamically optimized based on the cross-modal complementarity degradation index to obtain optimized fusion features. The fusion features are then mapped to the final high-quality fusion image through a convolutional decoding network.

[0014] Radiometric normalization and geometric alignment are performed on the original multimodal images to generate a multimodal image set with uniform spatial and numerical scales; In this embodiment of the invention, sensor-level radiometric correction is performed on the original multimodal image set. Based on dark current subtraction and flat-field correction, the grayscale count is corrected by combining the sensor's nonlinear inversion polynomial, and the digital count is adjusted according to the calibration coefficient. Converted to radiance: ,in Radiance, The black level is the background response value of the sensor when there is no incident light, which is mainly caused by dark current, electronic noise and system bias. This is the gain coefficient (a linear scaling factor obtained during sensor calibration, used to convert the corrected DN value into physical radiation). For the nonlinear correction term (used to compensate for the nonlinear deviation of the sensor under high dynamic range or extreme brightness conditions), generate the sensor-calibrated image set; The sensor-calibrated image set is standardized in the modal physics domain. Atmospheric correction and surface reflectivity conversion are performed on the visible light mode. Radiance to temperature conversion is performed on the thermal infrared mode based on the Planck function. Scattering coefficient images are obtained for the SAR mode through antenna gain correction and speckle suppression, forming a physically consistent image set. Multi-scale local contrast normalization is performed on the physically uniform image set. A robust normalization method combining Retinex decomposition and adaptive histogram equalization is adopted to improve local contrast and eliminate intermodal gain differences, thereby generating a contrast-normalized image set. For coarse registration of contrast-normalized image sets, the initial geographic alignment is first established using IMU and GPS metadata. Then, the translational deviation is estimated through phase correlation. Finally, SIFT / SuperPoint feature matching and RANSAC are combined to solve the global affine or homography transformation, generating coarse registration image sets and global geometric transformation parameters. We perform detailed multi-scale modality-invariant registration on the coarse-registered image set, construct cross-modal robust feature representations using MIND and Census descriptors, apply Lucas-Kanade iteration and mutual information maximization to optimize the sub-pixel displacement field at each layer of the image pyramid, and predict fine-grained displacements through lightweight U-Net regression and output pixel-level registration confidence maps to obtain dense registration results. The energy function is constructed by combining the dense registration results with the radiation adjustment parameters: ,in Let be the total energy function. For structural similarity weighting coefficients, It is a structural similarity index (which measures the similarity between two images in terms of brightness, contrast, and structure). Mutual information weighting coefficients Mutual information (measures the correlation between the gray-level distributions of two images; a larger value indicates more shared information). For radiation consistency weighting coefficient, The radiation residual function is represented by parameters The residual between the adjusted radiation value and the target consistency benchmark, This is the displacement field smoothness weighting coefficient. The gradient of the displacement field (representing the displacement field) The gradient in space reflects the rate of change of local deformation. Through Adam iteration, joint optimization is performed between registration residual energy and radiometric consistency to obtain the final geometrically corrected image set, radiometric adjustment parameters and registration uncertainty map. The final geometrically corrected image set, radiometric adjustment parameters, and registration uncertainty map are used for consistency detection and reliable region division. Stable regions are selected based on local gradient correlation and mutual information threshold, and a reliable fusion mask is generated by uncertainty clustering. The mask is then cleaned and smoothed by morphological dilation and erosion as well as time series exponential moving average (EMA) to form a multimodal image set with a unified spatial and numerical scale.

[0015] Multi-scale feature encoding is performed on a multimodal image set. Multi-scale features of each modality are extracted through residual convolutional networks and pyramid structures. Feature alignment and interoperability are achieved between scales to form a multi-scale feature group. In this embodiment of the invention, the multimodal image set is processed by its respective modality-specific residual coding backbone, and stacked residual units (each unit consists of two layers of convolution + normalization + activation, which are added to the input) are used to preserve details and stabilize gradients, generating a primary feature map set for each modality. ,in For modal indexing, The total number of modes, For the first The primary feature map extracted from each modality using the primary residual coding backbone (same spatial size as the original image, but with the same number of channels as the primary image). ); The initial feature map set is used to construct a multi-scale representation through pyramid downsampling and scale residual iteration. Each scale is downsampled once (strided convolution or pooling) and then followed by several residual blocks to form a pyramid feature map set with scales s=1..S. This allows for a multi-layered representation, from details to semantics; The pyramid features are projected onto a unified channel baseline at each scale via channel projection and modality normalization. First, a 1×1 convolution projects each modality feature onto a unified channel dimension D. Then, modality-aware normalization (e.g., InstanceNorm / GroupNorm with learnable affine coefficients or a separable modality adaptive normalization layer) is applied to the projected features to generate unified channel features at each scale. (The number of channels is D); The unified channel features are aligned across modal spaces at each scale: an offset field (or displacement field) is generated based on local correlation, and the spatial position of the features is corrected by deformable convolution or displacement sampling driven by the offset field, thereby outputting the aligned features and alignment confidence map at each scale. ,in For alignment features, To align confidence levels; The alignment features and alignment confidence maps are jointly enhanced intra-scale and inter-scale. A lightweight self-attention mechanism is used to calculate weighted inter-modal complementarity and cross-scale information transfer along top-down and bottom-up paths (similar to bidirectional feature aggregation), generating interactive enhancement features for each scale. It also outputs the complementary confidence map for each scale. ; The interaction enhancement features are weighted at each scale by joint channel and spatial attention and fused using adaptive normalization. Channel weights are generated using channel attention (a two-layer MLP based on GAP). And in combination with spatial attention Calculate the final weighted features and synthesize the final multi-scale feature set according to the weighted summation rule. This feature set retains complementary details of each modality and also has scale continuity.

[0016] Multi-scale feature groups are input into the cross-modal interaction module. The complementary information between modalities is enhanced through the attention mechanism to generate interaction features. Pixel-level confidence estimation and uncertainty prediction are performed on the interaction features to form stable confidence information. In this embodiment of the invention, multi-scale feature groups The input cross-modal feature interaction module performs channel linear projection on different modal features at each scale to form Query, Key, and Value tensors. , , In order to establish attention mapping relationships between modalities; It should be noted that the Query tensor The Key tensor represents the "query features" of modality m at scale s, used to send information requests to other modalities or calculate similarity; The "key features" of mode m at scale s serve as a reference benchmark for information provision; Value tensor The "value feature" of modality m at scale s represents the information that will actually be weighted and integrated into the final interaction feature. The Query, Key, and Value tensors are used to perform inter-modal self-attention computation at each scale. Softmax is used to normalize the similarity scores and weight the Value tensor, thereby enhancing inter-modal information complementarity and outputting preliminary interaction features at each scale. ; In an optional example, the similarity scores are normalized using Softmax and weighted into a Value tensor, as shown in the following formula: ,in This is the attention scaling factor (usually equal to the channel dimension D).

[0017] Interaction features at each scale The input is a multi-scale fusion network, which fuses features from both top-down and bottom-up information flows layer by layer to obtain enhanced features that simultaneously possess local details and global semantics. ; For the enhanced features Pixel-level confidence estimation is performed by predicting the modal confidence per pixel using a small convolutional subnetwork. The confidence uncertainty is calculated based on local variance, cross-modal consistency, and characteristic residuals. ; It should be noted that local variance is used to measure the stability of local information at a location. It is obtained by calculating the variance of each modality's features within the neighborhood window W of the pixel (x,y). Cross-modal consistency is obtained by calculating a consistency measure (such as cosine similarity or mutual information) for features of different modalities at the same location. Feature residual is obtained by taking the residual between the original features of the pixel and the features after cross-modal attention fusion. The final uncertainty is obtained by normalizing the above three items and combining them in a weighted manner.

[0018] The confidence and uncertainty information at each scale are smoothly fused within and between scales, and a stable confidence map is generated by combining the spatial neighborhood mean and a scale weighting factor (which can be learned or empirically set to reflect the importance of information at different scales). With stability uncertainty diagram This is used to guide cross-modal weight allocation; Integrating stable confidence and uncertainty plots with interactive enhancement features Joint encapsulation forms the final cross-modal interaction features. and stable confidence information , This provides a consistent and reliable input for subsequent modal contribution weight calculation and multimodal fusion.

[0019] Modal contribution weights are calculated by combining interaction features and confidence information, and modal contribution imbalance coefficients are generated based on the weights. At the same time, confidence estimation anomaly coefficients are calculated based on the confidence distribution. In this embodiment of the invention, cross-modal interaction features are included. With stable confidence plot The joint input modality contribution weight calculation module weights the feature importance of each modality through channel and spatial attention mechanisms, generating a pixel-level contribution weight map for each modality. ; It should be noted that global average pooling plus two layers of MLP are used to predict channel weights. Spatial attention is used to predict spatial weights using convolution. The two are then multiplied element-wise to obtain a joint attention map. : ,in This represents the element-wise multiplication of the channel dimension and the spatial dimension; then a stable confidence plot is used. right Perform pixel-by-pixel correction: .

[0020] Pixel-level contribution weight Perform intra-scale statistical analysis to calculate the average contribution of each mode. and standard deviation To quantify the overall dominance and imbalance of modes at the current scale; It should be noted that, , These represent the height and width of the feature map at scale s, respectively.

[0021] Average contribution value of each mode After inter-scale normalization, the modal contribution imbalance coefficient is calculated. This is used to quantify the skewness of the modal contribution distribution: ; It should be noted that the modal contribution imbalance coefficient at each scale... A larger value indicates a more uneven distribution of modal contributions; M represents the number of modes. This represents the maximum average contribution value across all modes.

[0022] Based on stable confidence plot Statistical analysis was performed on the confidence distribution at each scale to calculate the confidence estimation anomaly coefficient. The instability of confidence prediction is measured by pixel-level confidence entropy values. ,in This is to prevent division by zero by a very small constant (generally taken as...). ); Modal contribution imbalance coefficient Estimating the anomaly coefficient with confidence level The quantitative index of cross-modal complementarity degradation is encapsulated and used to adjust the fusion weights and training regularization in the future to ensure that the modal contributions are balanced and the confidence prediction is stable, thus forming a numerical signal that can be directly used for fusion optimization.

[0023] The cross-modal complementarity degradation index is calculated by interactively weighting the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, thereby quantifying the degree of degradation of the fusion effect due to the interaction between the two. In this embodiment of the invention, the cross-modal complementarity degradation index is calculated by interactively weighting the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, as shown in the following formula: In the formula It is a cross-modal complementarity degradation index. Contributes imbalance coefficient to modalities. To estimate the outlier coefficient for confidence level, represent the preset proportionality coefficients of the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, respectively, and All are greater than 0; The cross-modal complementarity degradation index is used to comprehensively reflect the combined degradation effect of modal contribution imbalance and confidence anomaly. It should be noted that the above formulas are all dimensionless calculations. Commonly used methods for removing dimensions include Min-Max normalization and Z-Score standardization, which will not be elaborated here. The settings should be tailored to the specific circumstances. For example, an expert-empowered approach could be adopted, where experts in relevant fields are invited to determine the pre-defined proportions for each indicator through professional opinion surveys and comprehensive evaluations. It can be 0.5 or 0.5.

[0024] The modal fusion weights and training regularization are dynamically optimized based on the cross-modal complementarity degradation index to obtain optimized fusion features. The fusion features are then mapped to the final high-quality fusion image through a convolutional decoding network.

[0025] In this embodiment of the invention, the cross-modal complementarity degradation index and cross-modal interaction features are combined. and stable confidence information The input is an adaptive mapper (which can be a small MLP + LayerNorm or a learnable lookup table), which generates a set of adjustment signals based on the cross-modal complementarity degradation exponent: a temperature factor per scale. Weight upper limit threshold Smoothing coefficient set of regular coefficients ; The adjustment signal is compared with the previously obtained pixel-level contribution weight. and the corresponding uncertainty Combine, calculate the updated normalized fusion weights element by element. ; In an optional example, the fusion weights The acquisition logic is as follows: First, use temperature scaling to soften the weight distribution: perform temperature scaling on each pixel position according to the modality. , here prevent Then, inverse variance correction is performed using the uncertainty to obtain the unnormalized weights. ; Finally, normalization is performed in the modal dimension: .

[0026] The updated fusion weights Apply pixel-by-pixel to the corresponding cross-modal interaction features Multi-scale fusion features are generated by weighted summation in the modality dimension. : ; The fusion features The input feature refinement module performs three parallel methods on each scale and fuses the results: (A) Residual detail flow: Use shallow residual blocks to extract and enhance high-frequency information to prevent the loss of details caused by weighted averaging; (B) Edge Consistency Correction: Calculate the difference between the fused features and the high-frequency gradients of each modality and use gradient preservation loss to guide convolution correction to restore edge sharpness; (C) Cross-scale band injection: Employ "maximum response selection" or bandpass fusion (e.g., select the highest response mode per pixel or inject by threshold) in high-frequency subbands to compensate for the energy decay caused by average fusion.

[0027] After refinement, apply channel / spatial attention gates to the refined output and fuse it with the original output using residual connections. Output optimized fusion features ; The optimized fusion features The input is a multi-scale convolutional decoder, which uses an upsampling + skip connections + attention gate structure to reconstruct the image layer by layer, generating an initial fused image. The following loss combination is used for optimization during training, and the cross-modal complementarity degradation index is used as the control signal for the adaptive regularizer: Task loss (e.g., L1 reconstruction + SSIM); edge preservation loss (to ensure gradient consistency); perceptual loss (e.g., VGG feature difference); CCDI guided regularization; contribution balance regularization.

[0028] The initial fused image The image is fed into a perceptual-level post-processing pipeline, which includes optional adversarial fine-tuning (using a small PatchGAN discriminator to refine textures), structured filtering (bilateral / guided filtering to restore details), and dynamic contrast and color mapping; the post-processing output is the final fused image. .

[0029] This invention introduces interactive modeling of modal contribution imbalance coefficient and confidence estimation anomaly coefficient during multimodal image fusion. It transforms the causal relationship between confidence prediction fluctuations and modal weight extremism into a quantifiable cross-modal complementarity degradation index. Using this as the core adjustment mechanism, it dynamically optimizes the fusion weights and training regularization, effectively suppressing the problem of excessive amplification or erroneous weakening of a single modality at the fusion strategy level. This method not only significantly improves the model's ability to preserve structure in edge details and texture regions, avoiding artifacts and information loss caused by modal complementarity degradation, but also exhibits stronger stability and robustness in complex scenes. The fusion results fully reflect the synergistic advantages of multimodal fusion, achieving a comprehensive improvement in image clarity, contrast, and information integrity. This provides higher quality and more reliable support for applications such as intelligent perception, medical diagnosis, and remote sensing interpretation of multimodal images.

[0030] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0031] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0032] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing a multimodal image fusion model, characterized in that: Includes the following steps: Radiometric normalization and geometric alignment are performed on the original multimodal images to generate a multimodal image set with uniform spatial and numerical scales; Multi-scale feature encoding is performed on a multimodal image set. Multi-scale features of each modality are extracted through residual convolutional networks and pyramid structures. Feature alignment and interoperability are achieved between scales to form a multi-scale feature group. Multi-scale feature groups are input into the cross-modal interaction module. The complementary information between modalities is enhanced through the attention mechanism to generate interaction features. Pixel-level confidence estimation and uncertainty prediction are performed on the interaction features to form stable confidence information. Modal contribution weights are calculated by combining interaction features and confidence information, and modal contribution imbalance coefficients are generated based on the weights. At the same time, confidence estimation anomaly coefficients are calculated based on the confidence distribution. The cross-modal complementarity degradation index is calculated by interactively weighting the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, thereby quantifying the degree of degradation of the fusion effect due to the interaction between the two. The modal fusion weights and training regularization are dynamically optimized based on the cross-modal complementarity degradation index to obtain optimized fusion features. The fusion features are then mapped to the final high-quality fusion image through a convolutional decoding network.

2. The method for constructing a multimodal image fusion model according to claim 1, characterized in that: The multimodal image sets are processed by their respective modality-specific residual coding backbones, and stacked residual units are used to preserve details and stabilize gradients to generate primary feature map sets for each modality. ; The initial feature map set is used to construct a multi-scale representation through pyramid downsampling and scale residual iteration. After downsampling at each scale, several residual blocks are appended to form a pyramid feature map set with scales s=1...S. ; The pyramid features are projected onto a unified channel baseline at each scale using channel projection and modality normalization. First, a 1×1 convolution projects each modality feature onto a unified channel dimension D. Then, modality-aware normalization is performed on the projected features to generate unified channel features at each scale. ; The unified channel features are aligned across modal spaces at each scale, and the aligned features and alignment confidence maps at each scale are output. ,in For alignment features, To align confidence levels; The alignment features and alignment confidence maps are jointly enhanced intra-scale and inter-scale. A lightweight self-attention mechanism is used to calculate weighted inter-modal complementarity and cross-scale information transfer along top-down and bottom-up paths, generating interactive enhancement features for each scale. It also outputs the complementary confidence map for each scale. ; The interactive enhancement features are weighted at each scale by joint channel and spatial attention and fused using adaptive normalization, with channel weights generated using channel attention. And in combination with spatial attention Calculate the final weighted features and synthesize the final multi-scale feature set according to the weighted summation rule. .

3. The method for constructing a multimodal image fusion model according to claim 2, characterized in that: Multiscale feature groups The input cross-modal feature interaction module performs channel linear projection on different modal features at each scale to form Query, Key, and Value tensors. , , ; The Query, Key, and Value tensors are used to perform inter-modal self-attention computation at each scale. The similarity scores are normalized using Softmax and weighted by the Value tensor to output the preliminary interaction features at each scale. ; Interaction features at each scale The input is a multi-scale fusion network, which fuses features from both top-down and bottom-up information flows layer by layer to obtain enhanced features that simultaneously possess local details and global semantics. ; For the enhanced features Pixel-level confidence estimation is performed by predicting the modal confidence per pixel using a small convolutional subnetwork. The confidence uncertainty is calculated based on local variance, cross-modal consistency, and characteristic residuals. ; The confidence and uncertainty information at each scale are smoothly fused within and between scales, and a stable confidence map is generated by combining the spatial neighborhood mean and scale weighting factor. With stability uncertainty diagram ; Integrating stable confidence and uncertainty plots with interactive enhancement features Joint encapsulation forms the final cross-modal interaction features. and stable confidence information , .

4. The method for constructing a multimodal image fusion model according to claim 3, characterized in that: Cross-modal interaction features With stable confidence plot The joint input modality contribution weight calculation module weights the feature importance of each modality through channel and spatial attention mechanisms, generating a pixel-level contribution weight map for each modality. ; Pixel-level contribution weight Perform intra-scale statistical analysis to calculate the average contribution of each mode. and standard deviation ; Average contribution value of each mode After inter-scale normalization, the modal contribution imbalance coefficient is calculated. This is used to quantify the skewness of the modal contribution distribution: ; Based on stable confidence plot Statistical analysis was performed on the confidence distribution at each scale to calculate the confidence estimation anomaly coefficient. : ,in This is a very small constant to prevent division by zero; Modal contribution imbalance coefficient Estimating the anomaly coefficient with confidence level Encapsulated as a quantitative indicator of cross-modal complementarity degradation.

5. The method for constructing a multimodal image fusion model according to claim 4, characterized in that: The cross-modal complementarity degradation index is calculated by cross-weighting the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, as shown in the following formula: In the formula It is a cross-modal complementarity degradation index. Contributes imbalance coefficient to modalities. To estimate the outlier coefficient for confidence level, represent the preset proportionality coefficients of the modal contribution imbalance coefficient and the confidence estimation anomaly coefficient, respectively, and All are greater than 0.

6. The method for constructing a multimodal image fusion model according to claim 5, characterized in that: Combining the cross-modal complementarity degradation index with cross-modal interaction features and stable confidence information The input is an adaptive mapper that generates a set of adjustment signals based on the cross-modal complementarity degradation index: a temperature factor per scale. Weight upper limit threshold Smoothing coefficient set of regular coefficients ; The adjustment signal is compared with the previously obtained pixel-level contribution weight. and the corresponding uncertainty Combine, calculate the updated normalized fusion weights element by element. ; The updated fusion weights Apply pixel-by-pixel to the corresponding cross-modal interaction features Multi-scale fusion features are generated by weighted summation in the modality dimension. ; The fusion features The input feature refinement module performs three parallel techniques on each scale and fuses the results, outputting the optimized fused features. ; The optimized fusion features The input is a multi-scale convolutional decoder, which uses an upsampling + skip connections + attention gate structure to reconstruct the image layer by layer, generating an initial fused image. ; The initial fused image The image is fed into the sensor-level post-processing pipeline and outputs the final fused image. .

Images