Multi-modal image fusion method and system based on three-domain collaborative nonlinear reconstruction

Abstract

This invention relates to the field of image fusion technology, specifically disclosing a multimodal image fusion method and system based on three-domain collaborative nonlinear reconstruction. The method includes acquiring a first-modality medical image and a second-modality medical image to obtain basic shallow representations; inputting the basic shallow representations into a multi-level coding structure, with the output of each coding level being downsampled and passed to the next level; setting a dual adaptive enhancement module at the output of each coding level to obtain enhanced cross-layer connectivity features; inputting the deepest coding features into a bottleneck aggregation structure; inputting the output of the bottleneck aggregation structure into a multi-level decoding structure, with each decoding level upsampled and fused with the enhanced features output from the corresponding coding level; and mapping the final decoding result to a single-channel fused image through pointwise convolution and an activation function. This invention employs a reconstruction method combining hierarchical coding, deep aggregation, and step-by-step decoding to improve the structural integrity, texture clarity, and visual naturalness of the fused image.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, specifically to a multimodal image fusion method and system based on three-domain collaborative nonlinear reconstruction. Background Technology

[0002] The development of medical imaging technology has enabled different modalities of images to reflect the characteristic information of human tissues or lesion areas from different perspectives. However, a single modality of image usually cannot simultaneously capture both anatomical structural resolution and functional metabolic characterization. For example, CT images can clearly present bony structures and density differences, but their ability to distinguish soft tissue details is relatively limited; MRI images have strong soft tissue resolution, and different sequences can reflect different tissue states, but they are insufficient in expressing functional activity; functional imaging methods such as PET and SPECT can reveal tissue metabolism, perfusion, or receptor distribution, but their spatial resolution is low and their anatomical localization ability is relatively weak. Therefore, unifying the superior information from different modalities of images to form a fused image that combines structural clarity and functional significance is of great importance for lesion identification, auxiliary diagnosis, and treatment planning.

[0003] Regarding the problem of multimodal medical image fusion, existing technologies have generally evolved from traditional methods to data-driven approaches. Early fusion methods were mostly based on signal transformation and artificial rules, with typical methods including wavelet transform, Laplacian pyramid, principal component analysis, sparse representation, and related optimization models. These methods typically decompose or map the source image first, and then combine information from different scales, frequency bands, or subspaces according to preset criteria. Their advantage lies in the relatively intuitive physical meaning and the relatively clear computational process. However, these methods are inherently difficult to escape the limitations of artificial feature design and fixed fusion criteria. They often struggle to fully represent the nonlinear correlations, local structural differences, and semantic complementarity between modalities present in complex medical images, thus easily leading to problems such as insufficient preservation of structural information, weakened functional information, or poor naturalness of the fused image in complex scenarios.

[0004] In recent years, deep learning methods have been widely introduced into the field of medical image fusion, with related research mainly focusing on techniques based on convolutional neural networks, autoencoders, and generative adversarial models. Compared with traditional methods, these methods can automatically learn image features through end-to-end training, reducing the reliance on manually designed fusion rules to some extent, and achieving significant improvements in fusion quality. However, based on existing research, these methods still have several unresolved issues: First, many methods rely mainly on local convolutional operations for feature extraction. While this effectively captures neighborhood texture and edge information, it is insufficient for modeling long-range correlations between different regions, making it difficult to fully characterize the global contextual relationships at the organ or lesion level. Second, most fusion methods focus more on spatial domain feature superposition or channel domain weighting, lacking deeper collaborative modeling of potential structural correspondences and functional complementarities between different modalities, resulting in limited cross-modal information alignment capabilities. Third, in complex medical image scenarios, there are often obvious nonlinear coupling relationships between different modalities. Traditional convolutional mapping or general linear transformation structures have inherent limitations in expressing high-order relationships, making it difficult to meticulously characterize complex tissue response differences. Fourth, although existing methods often introduce multi-scale strategies, the synergistic utilization of information at different scales is still insufficient. In particular, during the transfer and reconstruction of deep features, problems such as gradual attenuation of shallow details, insufficient response of deep salient regions, and decreased structural integrity are prone to occur.

[0005] Furthermore, in practical medical image fusion tasks, the relationship between different modalities is not simply a grayscale overlay, but rather contains multi-level information such as local edges, regional structures, overall semantics, and salient targets. Without joint modeling of global information, local information, and higher-order nonlinear relationships, the fusion results often fail to simultaneously meet the requirements of structural fidelity, clear texture, and enhanced saliency. Especially when the lesion area is small, the tissue background is complex, or there are significant differences between modalities, existing techniques are more prone to problems such as loss of important information, blurred boundaries, or unbalanced responses.

[0006] Therefore, there is an urgent need for an end-to-end medical image fusion architecture that can jointly utilize Transformer global modeling, dynamic graph neural network (GNN) anatomical structure perception, KAN high-order nonlinear function approximation and multi-scale attention mechanism. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal image fusion method and system based on three-domain collaborative nonlinear reconstruction, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A multimodal image fusion method based on three-domain collaborative nonlinear reconstruction, the method comprising: The first and second modal medical images are acquired, and they are concatenated into a dual-channel input tensor. The basic shallow representation is obtained through convolution mapping. The basic shallow representation is input into a multi-level coding structure, which includes at least one shallow coding level for extracting local texture and edge information and at least one deep coding level for global context modeling, structural relationship propagation and high-order nonlinear feature transformation. The output of each coding level is downsampled and passed to the next level. A dual adaptive enhancement module is set at the output of each coding level to obtain enhanced cross-layer connectivity features; The deepest encoded features are input into a bottleneck aggregation structure, which consists of at least two cascaded three-domain collaborative feature enhancement modules for the integration and reconstruction of high-level semantics. The output of the bottleneck aggregation structure is input into a multi-level decoding structure, which includes at least one first decoding level using a three-domain collaborative feature enhancement module and at least one second decoding level using a convolutional decoding module. Each decoding level is upsampled and fused with the enhanced features output from the corresponding encoding level.

[0009] The final decoding result is mapped to a single-channel fused image through pointwise convolution and activation functions.

[0010] As a further embodiment of the present invention, the multi-level coding structure has five levels. The first and second levels are shallow coding levels, which adopt convolutional coding modules; the third, fourth and fifth levels are deep coding levels, which respectively adopt three-domain collaborative feature enhancement modules.

[0011] As a further embodiment of the present invention, the multi-level decoding structure has five levels, with each of the first three levels employing a three-domain collaborative feature enhancement module, and the last two levels employing a convolutional decoding module.

[0012] As a further embodiment of the present invention, the three-domain collaborative feature enhancement module includes a hybrid attention module, an enhanced convolution module, and an SE-KAN module; the hybrid attention module has a Swing Transformer branch and a graph neural network branch in parallel, and the outputs of the two branches are concatenated, then convolved and connected by residuals; the enhanced convolution module expands the receptive field through double convolution and flattens the features into a sequence representation; the SE-KAN module performs a high-order nonlinear transformation through a KAN linear layer based on B-splines, and then adds it to the input residual after channel recalibration.

[0013] As a further embodiment of the present invention, the graph neural network branch dynamically models the input feature map as a graph structure, constructs an inflated k-NN adjacency graph based on feature similarity, and updates the node representation by aggregating the difference information of neighboring nodes and using multi-head weighted updating.

[0014] As a further embodiment of the present invention, each unary function of the KAN linear layer in the SE-KAN module is represented as a scaling factor multiplied by the sum of a smooth nonlinear activation term and a learnable B-spline term.

[0015] As a further embodiment of the present invention, the dual adaptive enhancement module includes two cascaded residual stages: the first stage: sequentially passing through channel recalibrated convolution, local standard convolution, and three depthwise separable dilated convolution branches with different kernel sizes, and then connecting the fused residuals; the second stage: applying channel attention and pixel attention in parallel, concatenating and fusing the weighted features, and then connecting them with the input residuals.

[0016] The present invention also provides a multimodal image fusion system based on three-domain collaborative nonlinear reconstruction, the system comprising: The feature acquisition unit is used to acquire the first modality medical image and the second modality medical image, concatenate the two into a dual-channel input tensor, and obtain the basic shallow representation through convolution mapping; The encoding unit is used to input the basic shallow representation into a multi-level encoding structure. The multi-level encoding structure includes at least one shallow encoding level for extracting local texture and edge information and at least one deep encoding level for global context modeling, structural relationship propagation and high-order nonlinear feature transformation. The output of each encoding level is downsampled and passed to the next level. An enhancement unit is used to set up dual adaptive enhancement modules at the output of each coding level to obtain enhanced cross-layer connection features; the deepest coding features are input into a bottleneck aggregation structure, which consists of at least two cascaded three-domain collaborative feature enhancement modules for the integration and reconstruction of high-level semantics; A decoding unit is used to input the output of the bottleneck aggregation structure into a multi-level decoding structure. The multi-level decoding structure includes at least one first decoding level using a three-domain collaborative feature enhancement module and at least one second decoding level using a convolutional decoding module. Each decoding level is upsampled and fused with the enhanced features output from the corresponding encoding level. The output unit is used to map the final-level decoding result into a single-channel fused image through pointwise convolution and activation functions.

[0017] Compared with the prior art, the beneficial effects of the present invention are: by adopting the processing method of uniformly entering the main framework at the input end of dual-source images, different modal information can enter the same representation space at the shallow stage, which is beneficial to reduce the modal shift and information fragmentation that may be caused during subsequent independent branch processing. By using a dual adaptive enhancement module to perform detail compensation and saliency enhancement on cross-layer features at all levels, it is beneficial to enhance the response of edge textures, small-scale targets and key regions; The reconstruction method that combines hierarchical encoding, deep aggregation, and step-by-step decoding helps to balance deep semantic recovery and shallow texture reconstruction, thereby improving the structural integrity, texture clarity, and visual naturalness of the fused image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0019] Figure 1 This is a flowchart of a multimodal image fusion method based on three-domain collaborative nonlinear reconstruction provided in an embodiment of the present invention.

[0020] Figure 2 This is a structural diagram of the three-domain collaborative feature enhancement module provided in an embodiment of the present invention.

[0021] Figure 3 This is a structural diagram of the hybrid attention module provided in an embodiment of the present invention.

[0022] Figure 4 The diagram shows the structure of the dual adaptive enhancement module provided in this embodiment of the invention. Detailed Implementation

[0023] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0024] In this embodiment of the invention, a multimodal image fusion method based on three-domain collaborative nonlinear reconstruction is provided, the method comprising: The first and second modal medical images are acquired, and they are concatenated into a dual-channel input tensor. The basic shallow representation is obtained through convolution mapping. The basic shallow representation is input into a multi-level coding structure, which includes at least one shallow coding level for extracting local texture and edge information and at least one deep coding level for global context modeling, structural relationship propagation and high-order nonlinear feature transformation. The output of each coding level is downsampled and passed to the next level. A dual adaptive enhancement module is set at the output of each coding level to obtain enhanced cross-layer connectivity features; The deepest encoded features are input into a bottleneck aggregation structure, which consists of at least two cascaded three-domain collaborative feature enhancement modules for the integration and reconstruction of high-level semantics. The output of the bottleneck aggregation structure is input into a multi-level decoding structure, which includes at least one first decoding level using a three-domain collaborative feature enhancement module and at least one second decoding level using a convolutional decoding module. Each decoding level is upsampled and fused with the enhanced features output from the corresponding encoding level. The final decoding result is mapped to a single-channel fused image through pointwise convolution and activation functions.

[0025] In this embodiment, a first modality medical image and a second modality medical image to be fused are acquired, and the two images are subjected to size unification and normalization processing. Subsequently, the first modality image and the second modality image are stitched together along the channel dimension to form a dual-channel input tensor, which is then input into the initial feature mapping module to obtain basic shallow representations through continuous convolution operations.

[0026] The initial shallow representations are sequentially input into a five-level coding structure. This five-level coding structure employs a layer-by-layer downsampling approach for feature extraction. The first and second coding levels primarily extract local texture and edge information, while the third, fourth, and fifth coding levels utilize a three-domain collaborative feature enhancement module to achieve global context modeling, structural relationship propagation, and high-order nonlinear feature transformation. Before entering the next layer, each coding output undergoes downsampling to reduce spatial resolution and increase channel dimensions, thereby gradually expanding the receptive range and semantic abstraction capabilities.

[0027] A dual adaptive enhancement module is set after each level of encoding output. The dual adaptive enhancement module includes two cascaded residual enhancement stages. The first stage uses channel recalibrated convolution, local convolution, and multi-scale dilated depthwise convolution branches to extract feature responses under different receptive ranges, and then fuses them through point-by-point mapping. The second stage reweights salient features through channel attention and pixel attention, and then integrates them through point-by-point mapping to obtain enhanced cross-layer connectivity features, thereby improving the attenuation problem of shallow details during deep layer transmission.

[0028] The deepest encoded features are input into a bottleneck aggregation structure. This bottleneck aggregation structure consists of multiple cascaded three-domain collaborative feature enhancement modules, which are used to further integrate and reconstruct multimodal information at a high semantic level, thereby enhancing the structural expressive power and cross-regional dependency modeling capabilities of deep features.

[0029] The output of the bottleneck aggregation structure is sequentially input into the five-level decoding structure. Each decoding level includes two processes: upsampling and cross-layer fusion. The first three decoding stages use a three-domain collaborative feature enhancement module for deep semantic recovery, while the last two decoding stages use convolutional decoding for shallow texture reconstruction. Each decoding level is concatenated and fused with the enhanced features of the corresponding encoding stage to achieve step-by-step recovery and reconstruction of multi-scale information.

[0030] The final decoding result is input into the output mapping layer, and the features are compressed into a single-channel fused image through pointwise convolution. The output value is then constrained to a predetermined range by the hyperbolic tangent activation function to obtain the final fusion result.

[0031] This invention employs a unified encoder-decoder processing framework, which jointly represents the two source images to be fused at the input end. Through a shallow feature mapping module, hierarchical encoding extraction, a three-domain collaborative enhancement module, a dual adaptive enhancement module, deep semantic aggregation, and step-by-step decoding reconstruction, it achieves the collaborative expression of structural, textural, and functional information in multimodal images, thereby obtaining a fusion result that combines edge clarity, structural integrity, and target saliency.

[0032] In a preferred embodiment of the present invention, the multi-level coding structure has five levels. The first and second levels are shallow coding levels, which employ convolutional coding modules. The third, fourth, and fifth levels are deep coding levels, which respectively employ three-domain collaborative feature enhancement modules.

[0033] In this embodiment, the shallow stage is mainly used to establish the basic texture and edge response of the input image; the encoding stage adopts a hierarchical feature extraction method from shallow to deep, where the first two stages focus on the representation of local structure and texture details, while the middle and high levels realize global dependency modeling, spatial topological relationship propagation, and high-order nonlinear mapping expression through a three-domain collaborative feature enhancement module; at the output of each level of encoding, a dual adaptive enhancement module is further introduced to compensate and enhance local detail information and salient region information; at the deepest level, a bottleneck aggregation structure composed of multiple three-domain collaborative enhancement modules is set to improve the organization ability of high-level semantic features and the cross-regional information interaction ability.

[0034] The first modality input is an MRI medical image, and the second modality input is a SPECT medical image. First, the two modalities are spatially registered and resized, and then normalized to a predetermined grayscale range. Subsequently, they are concatenated as two input channels in the channel dimension to form a two-dimensional dual-channel tensor. This dual-channel tensor is fed into a shallow convolutional mapping module to extract basic edge and local texture information.

[0035] The features after shallow mapping are sequentially fed into a five-level encoder. The first and second coding levels use convolutional coding modules to extract local texture and edge information; the third, fourth, and fifth coding levels use a three-domain collaborative feature enhancement module. In each level, the input features are processed through a windowed Transformer branch to establish long-range dependencies, through a dynamic graph convolutional branch to model the dynamic topological adjacency relationships between feature points, and through a spline-type nonlinear mapping branch to reconstruct higher-order features, ultimately forming the enhanced encoded features. In a preferred embodiment of the present invention, the three-domain collaborative feature enhancement module includes a hybrid attention module, an enhanced convolution module, and an SE-KAN module. The hybrid attention module has a Swing Transformer branch and a graph neural network branch in parallel. The outputs of the two branches are concatenated, mapped by convolution, and connected by residuals. The enhanced convolution module expands the receptive field through double convolution and flattens the features into a sequence representation. The SE-KAN module performs a high-order nonlinear transformation through a KAN linear layer based on B-splines, and then adds it to the input residual after channel recalibration.

[0036] The graph neural network branch dynamically models the input feature map as a graph structure, constructs an inflated k-NN adjacency graph based on feature similarity, and updates the node representation by aggregating the difference information of neighboring nodes and using multi-head weighting.

[0037] In this embodiment, the three-domain collaborative feature enhancement module (a hybrid feature enhancement module specifically designed for multimodal image fusion, aiming to collaboratively model the global semantics, local structure, and nonlinear feature representation of an image) first inputs the input feature map into a hybrid attention module. The Swing Transform branch captures long-range semantic dependencies, while the GNN branch dynamically models the feature map as a k-NN graph, using graph convolution to characterize nonlocal geometric and topological relationships. Subsequently, the outputs of the two branches are concatenated along the channel dimension and fused and dimensionality-reduced via an FFN module. Then, residual connections are used to combine the outputs with the original input to preserve details. Following this, the fused feature input is enhanced by the EFConv module, which expands the local receptive field and converts it into a sequence form for input into the SE-KAN module; where the Kolmogorov–Arnold Network (KAN) performs high-order nonlinear refinement.

[0038] like Figure 3 As shown, in the hybrid attention module, the input features are first fed in parallel into two complementary context modeling branches: the Swin Transformer branch and the GNN branch. The feature map x is fed into the Swin Transformer module, where layer normalization is first applied, and multi-head attention (MSA) is used to efficiently model long-range dependencies. The output features are then added to the original input to alleviate gradient vanishing and preserve low-level features.

[0039] ; Where x represents the input feature; Representation layer normalization operation; y1 represents the multi-head self-attention mapping; y1 represents the intermediate features after global dependency modeling; "+" represents the residual connection, which is used to preserve the original low-level features and improve training stability.

[0040] The residual output is normalized again and then fed into the MLP module to perform a nonlinear transformation on the features. Finally, the output of the MLP is added to the input again through the residual connection.

[0041] ; in, y1 represents the multilayer perceptual mapping; y2 represents the output feature after global context modeling and nonlinear transformation. This equation shows that the global branch enhances the ability of features to express long-range dependencies through a process of "normalization—attention modeling—residual superposition—feedforward mapping—residual superposition again".

[0042] On the other hand, the original input feature x simultaneously enters the GNN branch, dynamically modeling the input feature map x as a graph structure. Subsequently, a dilated k-NN adjacency graph is dynamically constructed based on feature space similarity to find the k nearest neighbor nodes. A v will be added to each one. i to v i+1 The edge e i,i+1 For the constructed graph representation , where e i,i+1 It belongs to E. By aggregating the characteristics of neighboring nodes and updating the node representation, information is transmitted and integrated.

[0043] Updated features It can be represented as: ; Where, x i Indicates the first The input features corresponding to each node; Represents node x i The neighborhood set; Represents the neighborhood aggregation function; This represents the feature update function; Represents the learnable weights during the aggregation phase; This represents the learnable weights during the update phase; This represents the updated node features; the 'x' at the end represents the original input residual term.

[0044] The neighborhood aggregation function can be expressed as: ; in, x represents the intermediate feature obtained after neighborhood difference aggregation of the i-th node; j Indicates the relationship with node x i Features of the j-th adjacent node; represents a convolutional mapping; max represents the maximum response aggregation within the neighborhood.

[0045] This formula shows that neighborhood propagation considers not only the current node itself, but also the differences between neighboring nodes and the current node.

[0046] Correspondingly, the update function can be expressed as: ; Among them, b i This represents the bias term corresponding to the i-th node; This indicates updating the mapping weights. The formula represents updating the node representation by using a linear mapping to update the neighborhood aggregation result.

[0047] To further improve the expressive power of different feature subspaces, aggregated features will be used. The feature is divided into h parts, each with different weights for updating. Then, all parts are concatenated to obtain the updated feature. ; in, Represented as Split head, These represent different weights, and b1, b2, ..., b i These represent different biases.

[0048] For the input feature x, the output feature y3 after processing by the Grapher module can be represented as: ; ; Among them, W in and W out These represent the weights of the fully connected network, b in and b out These represent different biases; This represents graph convolution operations; y3 represents the Gaussian error linear activation function; y3 represents the graph topology propagation branch output feature.

[0049] This approach expands the receptive field of nodes while maintaining computational efficiency, thereby effectively modeling long-range geometric dependencies between image features.

[0050] This is followed by a feedforward network (FFN) module, which contains two 1×1 convolutional blocks and a GELU activation function, used to efficiently realize information exchange between channels and enhance the expression of local structure and edge details. ; Where BN represents batch normalization operation; y4 represents a 1×1 convolutional mapping; y4 represents the output feature of the graph branch after feedforward enhancement.

[0051] The feature maps output from the two branches are concatenated along the channel dimension, and then a 3×3 convolutional layer is used for cross-branch feature selection and compression to restore the original number of channels. Subsequently, residual connections are introduced to preserve the original low-frequency information and inject enhanced high-frequency details, forming a preliminary enhanced fused feature. ; in, This indicates a channel-dimensional splicing operation; represents a 3×3 convolutional mapping; Y represents the output after passing through the hybrid attention module. This formula reflects the synergistic fusion of global dependency modeling and graph topological relationship propagation within a unified feature space.

[0052] In a preferred embodiment of the present invention, each unary function of the KAN linear layer in the SE-KAN module is represented as a scaling factor multiplied by the sum of a smooth nonlinear activation term and a learnable B-spline term.

[0053] In this embodiment, the fused features in the SE-KAN module are normalized by LayerNorm and then input into the EFConv (Enhanced Feature Convolution) module, where the receptive field is expanded through double convolution and the features are flattened into a sequence form; ; in, This represents the enhanced convolutional mapping process, used to expand the local receptive field and convert spatial features into sequence representations.

[0054] The sequence is then fed into the SE-KAN module, where KANLayer uses a B-spline-based KANLinear instead of a traditional MLP, and combines depthwise convolution with channel attention mechanisms to achieve adaptive, highly expressive nonlinear feature transformation. The definition of the KAN linear transformation is as follows: ; in, This represents the result of a higher-order nonlinear mapping; Let represent a learnable univariate nonlinear function; m and n represent the output feature dimension and input feature dimension, respectively. This equation shows that the mapping relationship between the dimensions of the input features is no longer limited to a fixed linear combination, but is jointly determined by the learnable nonlinear function matrix.

[0055] The parameterized representation of each unary function is as follows: ; in, Indicates the scaling factor; This represents a smoothing nonlinear activation term; This represents the learnable B-spline function term. This equation illustrates that the mapping result is composed of the fundamental nonlinear response and the spline refinement term, thereby improving the ability to approximate complex cross-modal relationships.

[0056] Finally, the KAN output is reshaped back into a spatial tensor and added to the residual of the previous fusion result to form an enhanced feature that combines global consistency, local structural fidelity, and strong nonlinear expressive power.

[0057] ; ; in, This represents a higher-order linear mapping constructed based on the Kolmogorov–Arnold concept; Indicates the channel recalibration process; * indicates element-wise multiplication; Y1 represents the channel weight vector; Y1 represents the output feature after nonlinear refinement and channel reweighting.

[0058] As a preferred embodiment of the present invention, the dual adaptive enhancement module includes two cascaded residual stages: the first stage: sequentially passing through channel recalibrated convolution, local standard convolution, and three depth-separable dilated convolution branches with different kernel sizes, and then connecting the fused residuals; the second stage: applying channel attention and pixel attention in parallel, concatenating and fusing the weighted features, and then connecting them with the input residuals.

[0059] In this embodiment, the dual adaptive enhancement module passes through a first normalization module, a first convolution module, a second convolution module, a multi-scale dilated convolution module, and a first fusion module sequentially from input to output. The multi-scale dilated convolution module includes three parallel convolution branches with different kernel sizes and dilation rates. The output of the first fusion module is connected to the input via a residual. Subsequently, it passes through a second normalization module, a dual attention module, and a second fusion module sequentially. The dual attention module includes a channel attention mechanism and a pixel attention mechanism. The output of the dual attention module is fused with the input of the dual attention module via a residual to obtain the final result.

[0060] Its structure consists of two cascaded residual stages: the first stage first performs channel recalibration through 1×1 convolution, and then captures the local context through 5×5 standard convolution. ; Here, x3 represents the intermediate feature after the first stage of convolution transformation.

[0061] Then, three depthwise separable convolutional branches with the same dilation rate (dilation=3) but different kernel sizes (3×3, 5×5, 7×7) are input in parallel to capture multi-scale contextual feature information without increasing the number of parameters. ; ; ; Among them, f7, f 13 f 19 These represent multi-scale features obtained from different equivalent receptive fields; "d=3" indicates an inflation rate of 3. This parallel structure is used to expand the feature extraction range without significantly increasing the number of parameters.

[0062] The three outputs are concatenated and then fused across scales via a feedforward neural network consisting of two 1×1 convolutions and GELU activations. Finally, residual connections are used to preserve the original information.

[0063] ; ; Among them, f cat Indicates multi-scale splicing features; x represents the nonlinear activation function; x4 represents the output feature of the first residual stage.

[0064] The second stage applies channel attention (CA) and pixel spatial attention (PA) in parallel after normalization: the former models the inter-channel dependencies through global averaging and max pooling, while the latter learns the spatial saliency map through a lightweight convolutional subnet.

[0065] ; Pixel attention can be represented as: ; in, and These represent average pooling and max pooling, respectively; X c (x4) represents the channel attention weight; X p (x4) represents the pixel attention weight.

[0066] The two processes weight the features separately and then concatenate them. They are then fused again through a feedforward neural network, and residual connections are used again to preserve the underlying details.

[0067] ; ; Here, x5 represents the stitched features after channel and pixel weighting; Y2 represents the final output of the dual adaptive enhancement process. This process can simultaneously take into account local detail preservation, salient region enhancement, and key channel response selection.

[0068] It effectively balances local details, global context, and adaptive selection of key channels and spatial locations, significantly enhancing the discriminativeness and robustness of feature representation.

[0069] The deepest encoded features are input into the bottleneck layer, where the encoder features are further aggregated and recombined, and then sequentially input into the five-level decoding structure.

[0070] In a preferred embodiment of the present invention, the multi-level decoding structure has five levels, with each of the first three levels employing a three-domain collaborative feature enhancement module, and the last two levels employing a convolutional decoding module.

[0071] In this embodiment, each decoding stage includes upsampling and skip connection fusion operations. The first three decoding stages employ a three-domain collaborative feature enhancement module, while the last two stages employ a convolutional decoding module to balance deep semantic recovery capabilities with shallow texture reconstruction efficiency. Each decoding stage is concatenated and fused with the enhanced features from the corresponding encoding stage to achieve multi-scale information fusion. The final decoding result is processed by an output mapping module. Specifically, a 1×1 convolution is first used to compress the feature channels into a single-channel output, and then the Tanh activation function is used to generate the final fused image. The resulting fusion result can simultaneously preserve the tissue structure boundaries in the MRI image and the functionally significant information in the SPECT image.

[0072] The decoding stage achieves a gradual recovery from deep semantic information to shallow texture information through step-by-step upsampling and cross-layer feature fusion; finally, a fused image is generated through the output mapping process.

[0073] To verify the effectiveness of the method of this invention, experiments were conducted using an MRI-SPECT medical image fusion dataset, and several representative and advanced image fusion methods were selected as comparison methods, including CDDFuse, DDcGAN, FATFasion, MACTFFasion, GeSeNet, MATR, SwinFuse, SwinFusion, U2Fusion, and EMFusion. During the experiments, a combination of qualitative analysis and quantitative evaluation was used to comprehensively compare the fusion performance of each method, in order to evaluate the actual effect of the method of this invention in terms of information preservation, structural representation, and visual quality. Quantitative evaluation indicators included information entropy (EN), standard deviation (SD), spatial frequency (SF), mutual information (MI), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), visual information fidelity (VIF), and edge information preservation. wait.

[0074] Table 1. Quantitative comparison results of MRI-SPECT fusion. Bold and underlined values ​​represent the optimal and suboptimal values, respectively.

[0075]

[0076] Experimental results show that the method of the present invention achieves superior results in multiple evaluation metrics, indicating that it has good comprehensive performance in multimodal medical image fusion tasks. Specifically, the method of the present invention improves mutual information (MI), peak signal-to-noise ratio (PSNR), visual information fidelity (VIF), and edge information preservation. It exhibits optimal performance across various metrics, achieving or being at the best level in structural similarity (SSIM) and maintaining high levels in information entropy (EN) and spatial frequency (SF). This demonstrates that the method of this invention can effectively integrate complementary information from MRI and SPECT images, enhancing the richness of information in the fused image while effectively preserving tissue structure, edge contours, and significant region responses.

[0077] From the image characteristics reflected by various evaluation indicators, the high mutual information (MI) value indicates that the fused image inherits more effective information from the two source images, demonstrating the strong cross-modal information integration capability of the method of this invention; the high peak signal-to-noise ratio (PSNR) value indicates that the fused image has less distortion and higher reconstruction quality; the high structural similarity (SSIM) value indicates that the fused result maintains good consistency with the source images in terms of organizational structure, contour morphology, and spatial distribution; the high visual information fidelity (VIF) value indicates that the fused image retains more useful information at the visual perception level and has a good subjective visual effect; and the edge information preservation degree... A higher value indicates that the method of the present invention has significant advantages in edge detail preservation, texture transfer, and local structure enhancement. At the same time, the information entropy (EN) and spatial frequency (SF) are also at a high level, further illustrating that the fused image has a rich amount of information and a clearer ability to express details and textures.

[0078] The experimental results above demonstrate that the method of this invention not only effectively improves the information preservation capability of fused images in MRI-SPECT medical image fusion tasks, but also exhibits good overall performance in terms of structural consistency, edge sharpness, texture detail representation, and preservation of salient targets. This indicates that the method of this invention can generate fused images that combine structural integrity and functional saliency, and has good application prospects and practical value.

[0079] This invention also provides a multimodal image fusion system based on three-domain collaborative nonlinear reconstruction, the system comprising: The feature acquisition unit is used to acquire the first modality medical image and the second modality medical image, concatenate the two into a dual-channel input tensor, and obtain the basic shallow representation through convolution mapping; The encoding unit is used to input the basic shallow representation into a multi-level encoding structure. The multi-level encoding structure includes at least one shallow encoding level for extracting local texture and edge information and at least one deep encoding level for global context modeling, structural relationship propagation and high-order nonlinear feature transformation. The output of each encoding level is downsampled and passed to the next level. An enhancement unit is used to set up dual adaptive enhancement modules at the output of each coding level to obtain enhanced cross-layer connection features; the deepest coding features are input into a bottleneck aggregation structure, which consists of at least two cascaded three-domain collaborative feature enhancement modules for the integration and reconstruction of high-level semantics; A decoding unit is used to input the output of the bottleneck aggregation structure into a multi-level decoding structure. The multi-level decoding structure includes at least one first decoding level using a three-domain collaborative feature enhancement module and at least one second decoding level using a convolutional decoding module. Each decoding level is upsampled and fused with the enhanced features output from the corresponding encoding level. The output unit is used to map the final-level decoding result into a single-channel fused image through pointwise convolution and activation functions.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal image fusion method based on three-domain collaborative nonlinear reconstruction, characterized in that, The method includes: The first and second modal medical images are acquired, and they are concatenated into a dual-channel input tensor. The basic shallow representation is obtained through convolution mapping. The basic shallow representation is input into a multi-level coding structure, which includes at least one shallow coding level for extracting local texture and edge information and at least one deep coding level for global context modeling, structural relationship propagation and high-order nonlinear feature transformation. The output of each coding level is downsampled and passed to the next level. A dual adaptive enhancement module is set at the output of each coding level to obtain enhanced cross-layer connectivity features; The deepest encoded features are input into a bottleneck aggregation structure, which consists of at least two cascaded three-domain collaborative feature enhancement modules for the integration and reconstruction of high-level semantics. The output of the bottleneck aggregation structure is input into a multi-level decoding structure, which includes at least one first decoding level using a three-domain collaborative feature enhancement module and at least one second decoding level using a convolutional decoding module. Each decoding level is upsampled and fused with the enhanced features output from the corresponding encoding level. The final decoding result is mapped into a single-channel fused image through pointwise convolution and activation functions; The three-domain collaborative feature enhancement module includes a hybrid attention module, an enhanced convolution module, and a SE-KAN module; The hybrid attention module has a parallel Swing Transformer branch and a graph neural network branch. The outputs of the two branches are concatenated, then mapped by convolution and connected by residuals. The enhanced convolutional module expands the receptive field and flattens the features into sequence representations through double convolution; The SE-KAN module performs a high-order nonlinear transformation through a KAN linear layer based on B-splines, and then adds it to the input residual after channel recalibration; The dual adaptive enhancement module comprises two cascaded residual stages: The first stage involves sequentially performing channel recalibrated convolution, local standard convolution, and three depth-separable dilated convolutions with different kernel sizes in parallel branches, followed by fusion and residual connection. The second stage involves applying channel attention and pixel attention in parallel, concatenating and fusing the weighted features, and then connecting them to the input residual.

2. The multi-modal image fusion method based on three-domain collaborative nonlinear reconstruction according to claim 1, characterized in that, The multi-level coding structure has five levels. The first and second levels are shallow coding levels, which use convolutional coding modules. The third, fourth and fifth levels are deep coding levels, which use three-domain collaborative feature enhancement modules respectively.

3. The multi-modal image fusion method based on three-domain collaborative nonlinear reconstruction according to claim 1, characterized in that, The multi-level decoding structure has five levels. The first three levels each use a three-domain collaborative feature enhancement module, and the last two levels use a convolutional decoding module.

4. The multi-modal image fusion method based on three-domain cooperative nonlinear reconstruction according to claim 1, characterized in that, The graph neural network branch dynamically models the input feature map as a graph structure, constructs an inflated k-NN adjacency graph based on feature similarity, and updates the node representation by aggregating the difference information of neighboring nodes and using multi-head weighting.

5. The multi-modal image fusion method based on three-domain collaborative nonlinear reconstruction according to claim 1, characterized in that, Each unary function of the KAN linear layer in the SE-KAN module is represented as a scaling factor multiplied by the sum of a smoothed nonlinear activation term and a learnable B-spline term.

6. A multi-modal image fusion system based on three-domain collaborative non-linear reconstruction, for implementing the multi-modal image fusion method based on three-domain collaborative non-linear reconstruction according to any one of claims 1-5, characterized in that, The system includes: The feature acquisition unit is used to acquire the first modality medical image and the second modality medical image, concatenate the two into a dual-channel input tensor, and obtain the basic shallow representation through convolution mapping; The encoding unit is used to input the basic shallow representation into a multi-level encoding structure. The multi-level encoding structure includes at least one shallow encoding level for extracting local texture and edge information and at least one deep encoding level for global context modeling, structural relationship propagation and high-order nonlinear feature transformation. The output of each encoding level is downsampled and passed to the next level. An enhancement unit is used to set up dual adaptive enhancement modules at the output of each coding level to obtain enhanced cross-layer connection features; the deepest coding features are input into a bottleneck aggregation structure, which consists of at least two cascaded three-domain collaborative feature enhancement modules for the integration and reconstruction of high-level semantics; A decoding unit is used to input the output of the bottleneck aggregation structure into a multi-level decoding structure. The multi-level decoding structure includes at least one first decoding level using a three-domain collaborative feature enhancement module and at least one second decoding level using a convolutional decoding module. Each decoding level is upsampled and fused with the enhanced features output from the corresponding encoding level. The output unit is used to map the final-level decoding result into a single-channel fused image through pointwise convolution and activation functions.

Images