Medical image fusion method and system with cross-modal parameter coupling

Abstract

The application discloses a kind of medical image fusion methods and systems of cross-modal parameter coupling, belong to artificial intelligence and medical image processing technical field.The method of the application includes: obtaining and pre-processing multi-modal medical image;Extract multi-level feature through encoding network and retain shallow feature;Through cross-modal state space modeling, modulation parameter of current mode is dynamically generated from another modal feature, and the feature is updated through four-direction scanning fusion;The updated feature is enhanced by local convolution, multiplied and added to fusion and state space refinement, to generate a fusion feature map in combination with shallow features;Through decoding network, it is gradually up-sampled to reconstruct, and the final fusion image is obtained by adding residual error.The application also provides a corresponding system, electronic equipment, storage medium and computer program product.The application solves the problem of insufficient cross-modal information coordination and high computational complexity in multi-modal fusion, and improves the structural consistency and information integrity of the fusion image.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and medical image processing technology. More specifically, this application relates to a medical image fusion method and system with cross-modal parameter coupling. Background Technology

[0002] Multimodal medical image fusion technology integrates complementary information from different imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and single-photon emission computed tomography (SPECT) to obtain more comprehensive tissue structure and functional expression. It has been widely used in clinical diagnosis, lesion analysis, and decision support. How to efficiently fuse complementary information from multimodal medical images is an important research direction in the field of medical image analysis.

[0003] Early medical image fusion methods relied on manually designed features and mainly included two types of techniques: multi-scale transformation and sparse representation. Multi-scale transformation methods decomposed the image into multi-scale sub-bands and processed structural and detailed information separately, while sparse representation methods completed feature reconstruction through dictionary construction and sparse coding. These methods depended on shallow features and manually designed rules, which could not effectively characterize the complex nonlinear relationships between multimodal data, and their fusion effect was limited in scenarios with large modal differences.

[0004] The development of deep learning has driven a shift in fusion methods towards a data-driven representation learning paradigm. Convolutional neural network-based fusion methods rely on local convolutions to extract features, achieving some effect in detail preservation. However, limited by local receptive fields, they cannot effectively model long-range dependencies in images and struggle to integrate cross-regional structural information. Transformer-based fusion methods utilize self-attention mechanisms to model global long-range dependencies. Some methods optimize computational efficiency through hierarchical structures, but the computational complexity of global attention increases quadratically with input size, resulting in significant computational and storage overhead in high-resolution medical images or 3D scenes, thus limiting practical application efficiency.

[0005] In recent years, the Structured State Space Model (SSM) has been applied to visual and medical image analysis tasks. This model achieves long-range dependency capture with linear complexity through a selective scanning mechanism, and some fusion methods combine this model to optimize feature representation. However, existing multimodal fusion methods based on the state space model only use it as a single-modal feature extraction module and do not build a cross-modal dependency characterization mechanism during the state space modeling process.

[0006] In summary, existing multimodal medical image fusion methods generally adopt a processing mode of first extracting features of each modality independently and then fusing them. Cross-modal correlations cannot be fully modeled in the feature representation stage. At the same time, there is a problem that it is difficult to balance long-range dependency modeling ability and computational efficiency. Furthermore, the fusion process lacks a unified constraint mechanism, resulting in poor consistency and information integrity of the fusion results. These methods cannot meet the needs of efficient and stable multimodal medical image fusion in clinical scenarios. Summary of the Invention

[0007] This application provides a cross-modal parameter coupling medical image fusion method, which takes multimodal medical images as input and constructs a cross-modal state space fusion framework to achieve high-quality multimodal medical image fusion while ensuring computational efficiency.

[0008] This application provides a cross-modal parameter coupling medical image fusion system, which can efficiently complete multimodal medical image fusion end-to-end through the collaborative work of multiple modules, ensuring the performance of the fusion results in terms of structural consistency, preservation of significant information and expression of edge details.

[0009] To achieve these objectives and other advantages according to this application, a cross-modal parameter coupling medical image fusion method is provided, comprising: Acquire medical images from the first modality and the second modality; The first modal medical image and the second modal medical image are respectively subjected to size normalization, pixel value normalization and dimension rearrangement to obtain the preprocessed first modal medical image and the preprocessed second modal medical image; Multi-level feature extraction is performed on the two images through an encoding network. The encoding network contains N encoding levels set from high to low resolution, generating initial features of the first modality and initial features of the second modality for each level, and retaining the initial features of at least one encoding level as shallow features. Performing cross-modal state-space modeling on initial features at at least one coding level includes: generating first content-aware parameters from the second modal initial features to modulate the input modulation parameters, output modulation parameters, and time step parameters of the first state-space model, and performing a four-way scan and fusion on the state-space evolution to obtain updated first modal features; generating second content-aware parameters from the first modal initial features to modulate the input modulation parameters, output modulation parameters, and time step parameters of the second state-space model, and performing a four-way scan and fusion on the state-space evolution to obtain updated second modal features; For the encoding level of the execution model, the updated first modality features and the updated second modality features are locally convolutionally enhanced, then multiplied element-wise and added element-wise to themselves, then refined by the state space model, and finally added element-wise to the linear mapping result of the shallow features of this level to obtain the fused feature map of this level. The deepest encoding layer's fused feature map is decoded and reconstructed step by step by a decoding network. The decoding network contains N decoding layers set from low to high resolution. The N decoding layers correspond one-to-one with the N encoding layers according to resolution. The decoding input of each level includes the upsampled result of the previous level's decoding output and the fused feature map of the corresponding encoding layer. In each decoding level, the resolution is doubled and the number of channels is halved through reversible convolution upsampling. The reconstructed image output by the decoding network is added element-wise to the preprocessed first modality medical image and the preprocessed second modality medical image to obtain the final fused image.

[0010] Preferably, the four-directional scanning and fusion includes: constructing sequences of four directions: original spatial order forward, transposed spatial order forward, original spatial order reverse, and transposed spatial order reverse; performing state space evolution on each direction sequence; and then restoring the outputs of each direction to the original spatial order and adding them element by element.

[0011] Preferably, the local convolution enhancement is a 3×3 depthwise convolution with SiLU activation; the state space model refinement adopts a single-modal state space model, whose parameters are generated from its own features.

[0012] Preferably, the reversible convolutional upsampling is achieved through frequency domain inversion, including: performing zero-interpolation upsampling and nearest-neighbor upsampling on the input features, transforming the learnable convolutional kernel to the frequency domain, constructing and solving the frequency domain equation, and then inversely transforming it back to the spatial domain.

[0013] Cross-modal parameter coupling medical image fusion systems include: The input and preprocessing module is used to acquire the first modality medical image and the second modality medical image, and perform size normalization, pixel value normalization and dimension rearrangement respectively, and output the preprocessed first modality medical image and the preprocessed second modality medical image. A dual-branch cross-modal representation learning module, connected to the input and preprocessing module, is used to perform multi-level feature extraction. The multi-level feature extraction includes N encoding levels set from high to low resolution, generating initial features of the first modality and initial features of the second modality at each level, and retaining the initial features of at least one encoding level as shallow features. The dual-branch cross-modal representation learning module is also used to perform cross-modal state space modeling on the initial features of at least one encoding level, including generating input modulation parameters, output modulation parameters, and time step parameters of the first state space model from the initial features of the second modality, generating corresponding parameters of the second state space model from the initial features of the first modality, and performing four-directional scanning and fusion on the state space evolution, outputting updated first and second modal features. The consensus fusion representation construction module, connected to the dual-branch cross-modal representation learning module, is used to perform local convolution enhancement on the updated first modal features and second modal features, then perform element-wise multiplication and add them element-wise to themselves, then refine them through the state space model, and finally add them element-wise to the linear mapping result of the shallow features of this level to output the fusion feature map of this level. The decoding and reconstruction module, connected to the consensus fusion representation construction module, is used to decode and reconstruct the fusion feature map of the deepest encoding level step by step. It includes N decoding levels set from low to high resolution. The N decoding levels correspond one-to-one with the N encoding levels according to resolution. The decoding input of each level includes the upsampling result of the decoding output of the previous level and the fusion feature map of the corresponding encoding level. In each decoding level, the resolution is doubled and the number of channels is halved through reversible convolution upsampling. The decoding and reconstruction module is also used to add its output reconstructed image element-wise with the preprocessed first modality medical image and the preprocessed second modality medical image to output the final fusion image.

[0014] Preferably, the dual-branch cross-modal representation learning module includes: a data embedding submodule, used to divide the input tensor into non-overlapping local regions of a preset fixed size and convert them into embedding vectors through convolution mapping; a state-space-based cross-modal representation learning submodule, used to perform the cross-modal state space modeling, including constructing sequences in four directions: original space order forward, transposed space order forward, original space order reverse, and transposed space order reverse, and performing state space evolution and fusion; and a resolution transformation and feature transfer submodule, used to perform two-dimensional convolution downsampling with a stride of 2 on the updated features to double the number of channels and perform layer normalization.

[0015] Preferably, in the consensus fusion representation construction module, the local convolution enhancement is a 3×3 depthwise convolution with SiLU activation, the state space model refinement adopts a single-modal state space model and its parameters are generated by its own features, and the linear mapping is a linear transformation that keeps the number of channels unchanged; the decoding and reconstruction module includes a state space decoder submodule, a resolution restoration and feature reconstruction submodule, and an end resolution restoration submodule; the state space decoder submodule is used to add the current decoded feature and the fusion feature of the same coding level element by element and then divide it into two branches along the channel dimension, perform local convolution processing and state space modeling processing respectively, and then perform channel recombination after splicing; the resolution restoration and feature reconstruction submodule is used to double the resolution and halve the number of channels by reversible convolution upsampling with a kernel size of 2 and padding of 1; the end resolution restoration submodule is used to perform two reversible convolution upsamplings to the resolution after size normalization, and map the number of channels to 1 by 1×1 convolution.

[0016] An electronic device includes: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the cross-modal parameter coupling medical image fusion method.

[0017] A storage medium on which a computer program is stored, which, when executed by a processor, implements the aforementioned cross-modal parameter coupling medical image fusion method.

[0018] A computer program product comprising a computer program that, when executed by a processor, implements the aforementioned cross-modal parameter coupling medical image fusion method.

[0019] This application includes at least the following beneficial effects: First, this invention introduces intermodal dependencies in the feature representation stage by using state-space modeling with cross-modal parameter coupling, fundamentally avoiding the loss of key information caused by the fusion of independent features in traditional methods. At the same time, it adopts a four-directional scanning and fusion strategy to overcome the anisotropy problem of single-directional scanning, achieves near-isotropic long-range dependency capture, and ensures the fusion quality of high-resolution medical images.

[0020] Secondly, this invention constructs an end-to-end medical image fusion system consisting of an input and preprocessing module, a dual-branch cross-modal representation learning module, a consensus fusion representation construction module, and a decoding and reconstruction module. The modules work together to achieve deep interaction of cross-modal information in the encoding stage and to achieve accurate recovery of high-frequency details in the decoding stage through reversible convolution upsampling and channel recombination, thus ensuring the overall clarity and structural consistency of the fused image.

[0021] Third, this invention uses a state-space model for sequence modeling, and the computational complexity is linearly related to the sequence length. Compared with the quadratic complexity of the Transformer method, it is more suitable for high-resolution medical image processing scenarios. At the same time, the multiply-accumulate operation and single-modal state-space refinement in consensus fusion can effectively integrate complementary information from different modalities and suppress redundancy, thereby improving the integrity of the fused information.

[0022] Fourth, this invention achieves optimal or comparable objective indicators in three typical medical image fusion tasks: CT-MRI, PET-MRI, and SPECT-MRI. In the downstream segmentation task of BRaTS, the Dice coefficients of the overall tumor region, the tumor core region, and the enhanced region reach 94.94%, 94.51%, and 91.40%, respectively, which significantly improves the discrimination ability of the downstream task and has good clinical application prospects.

[0023] Other advantages, objectives and features of this application will be partly apparent from the description below, and partly understood by those skilled in the art through study and practice of this application. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process architecture of the cross-modal parameter coupling medical image fusion system of the present invention; Figure 2 This is a schematic diagram illustrating the internal structure and cross-modal information interaction of the dual-branch cross-modal representation learning module of the present invention; Figure 3 This is a schematic diagram illustrating the feature enhancement, multiply-accumulate fusion, and state space refinement process of the consensus fusion representation construction module of the present invention; Figure 4 This is a schematic diagram of the dual-branch parallel processing flow of the state space decoder submodule in the decoding and reconstruction module of the present invention; Figure 5 This is a comparison of the visualization effects of the present invention and the comparative method on three fusion tasks: CT-MRI, PET-MRI, and SPECT-MRI. Figure 6 This is a comparison of the brain tumor segmentation performance of this invention on the BRaTS dataset. Detailed Implementation

[0025] The present application will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description. It should be noted that, unless otherwise specified, the experimental methods described in the following embodiments are conventional methods and should not be construed as limitations on the present application.

[0026] In existing technologies, different modal images are encoded using independent feature extraction networks, resulting in a lack of interaction between the generated modal features. This necessitates additional alignment and weighting operations in the subsequent fusion stage, which can easily lead to the loss of key information or the superposition of redundant information. To address this issue, this application provides a cross-modal parameter coupling medical image fusion method, such as... Figure 1 As shown, it includes: Acquiring first and second modal medical images: Any two modal medical images from clinical CT, MRI, PET, and SPECT can be acquired as the first and second modal medical images, ensuring that the two images are spatially registered, that is, the image pixels of the same anatomical position of the same patient correspond one-to-one. The first and second modal medical images are subjected to size normalization, pixel value normalization, and dimension rearrangement. Size normalization adjusts the spatial size of the original image to a preset fixed size H×W (H is the image height and W is the image width), for example, scaling the image to 256 pixels in height and 256 pixels in width. Pixel value normalization converts the pixel values ​​in the image to a specified range, for example, normalizing the size of the two images to 256×256 pixels and normalizing each pixel value to the range [0,1] by dividing it by 255. Dimension rearrangement adjusts the dimensional arrangement order of the image from the original height, width, channel (HWC) order to the channel, height, width (CHW) order, resulting in the first modal tensor and the second modal tensor, i.e., the preprocessed first modal medical image and the preprocessed second modal medical image. This eliminates the format differences in resolution, pixel value range, and dimensional arrangement of the original image, completing the standardization conversion of the original data. Multi-level feature extraction is performed on two images using an encoding network. The encoding network adopts a dual-branch structure, with two branches processing image data from the first and second modalities respectively. The two branches have identical structures and do not share parameters. The encoding network contains N encoding levels arranged from high to low resolution, where N is a preset positive integer, for example, N=4. In each encoding level, the spatial resolution of the input features gradually decreases, while the number of channels gradually increases, thereby progressively extracting hierarchical features from details to semantics. Specifically, the first encoding level uses a preprocessed tensor as input and converts the image into embedded features through a data embedding operation. The number of channels for the embedded features is set to a preset value d. model Then, the feature transformation and downsampling of the subsequent coding levels are carried out in sequence. During the coding process, each coding level generates the first modality initial feature and the second modality initial feature of each level, and retains the initial feature of at least one coding level as shallow features. The shallow features carry the low-level detail information of the image and retain high spatial resolution and rich detail information. Cross-modal state space modeling is performed on the initial features of at least one coding level. This means that during the evolution of the state space model, the state update of one modality depends not only on its own input sequence but also on the features of another modality. The evolution process of the current modality is modulated by state space model parameters dynamically generated from the features of the other modality. Specifically, this includes: for the initial features of the first modality, generating a first content-aware parameter from the initial features of the second modality to modulate the input modulation parameter B1, output modulation parameter C1, and time step parameter dt1 of the first state space model, and performing a four-way scan and fusion on the state space evolution to obtain the updated first modal features; for the initial features of the second modality, generating a second content-aware parameter from the initial features of the first modality to modulate the input modulation parameter B2, output modulation parameter C2, and time step parameter dt2 of the second state space model, and performing a four-way scan and fusion on the state space evolution to obtain the updated second modal features. The features of the two modalities achieve collaborative updating and interactive fusion of information during the state space evolution process. Specifically, taking the generation of first content-aware parameters from the initial features of the second modality as an example, the specific steps are as follows: First, the initial features of the second modality are input into a linear mapping layer. The number of output channels of the linear mapping layer is pre-set according to the parameter dimension requirements of the subsequent state space model. ssm + 2×g×d state d ssm Let g be the number of main branch channels in the state-space model, g be the number of groups, and d be the number of groups. stateFor the state space model's state dimensions, a linear transformation maps the initial features of the second modality to the dimension space generated by the adaptation parameters, yielding intermediate features. Subsequently, a dimension adjustment operation is performed on these intermediate features, transforming them from the order of batch, height, width, channels (BHWC) to batch, channels, height, width (BCHW) to adapt to the input requirements of 2D deep convolution. Next, the dimension-adjusted intermediate features are fed into a 2D deep convolutional layer for processing. The kernel size of this deep convolutional layer is set to a preset value, for example, k=3×3, stride s=1, and padding p=1. Based on the kernel size setting, it is ensured that the spatial dimensions of the features remain unchanged before and after convolution processing. The number of groups in the depthwise convolution is equal to the number of channels in the input feature, meaning that each channel performs the convolution operation independently, avoiding cross-interference between features from different channels. Finally, the output result is processed by an activation function, such as the SiLU activation function, to enhance the non-linear expressive power of the features. After the above linear mapping, dimension adjustment, depthwise convolution, and activation processing, the first content-aware parameter carrying the second modality feature information is obtained. This first content-aware parameter will be decomposed into the input modulation parameter B, the output modulation parameter C, and the time step parameter dt in subsequent steps to modulate the evolution process of the first state space model. The generation process of the second content-aware parameter is exactly the same as the above steps, and will not be repeated here. Specifically, taking the generation and local modeling process of the initial features of the first modality as an example, the specific steps are as follows: First, the initial features of the first modality are input into another independent linear mapping layer, and the number of output channels of the linear mapping layer is preset to the number of channels d of the main branch of the state space model. ssmThe initial features of the first modality are mapped to a dimensional space adapted to the evolution of the state space through a linear transformation to obtain state input features, whose dimensional order still maintains the order of batch, height, width, and channel (BHWC). Subsequently, a dimensionality adjustment operation is performed on the state input features, transforming their dimensional order to batch, channel, height, and width (BCHW) to adapt to the input requirements of two-dimensional deep convolution. Then, the dimensionally adjusted state input features are fed into a two-dimensional deep convolutional layer for processing. The parameter settings of the two-dimensional deep convolutional layer are completely consistent with those of the deep convolutional layer in the content-aware parameter generation process, i.e., the kernel size is a preset value, for example, kernel size k=3×3, stride s=1, padding size p=1, and segmentation... The number of groups is equal to the number of input channels, meaning that each channel performs convolution operations independently, avoiding cross-interference between features from different channels. Through deep convolution operations, the local spatial correlation information of the state input features is captured, enhancing the model's ability to perceive details such as image edges and textures. Finally, activation functions, such as the SiLU activation function, are applied to the output results to enhance the nonlinear expressive power of the features. After the above linear mapping, dimension adjustment, deep convolution, and activation processing, the state input features after local modeling are obtained. These features will serve as the input sequence x of the first state space model and participate in subsequent state evolution calculations. The generation of the state input features and the local modeling process for the initial features of the second modality are exactly the same as the above steps, and will not be repeated here. Specifically, the content-aware parameter features and state input features obtained above are concatenated along the channel dimension to obtain joint features, with the number of channels of the joint features being d. ssm + (d ssm + 2×g×d state ) = 2×d ssm + 2×g×d stateThe joint features simultaneously contain the state input sequence required for state space evolution, as well as the input modulation parameter B, output modulation parameter C, and time step parameter dt information generated by another mode. The dimensional order of the joint features remains batch, height, width, and channel (BHWC). To apply the state space model to a two-dimensional image, the two-dimensional feature map needs to be flattened into a one-dimensional sequence. However, scanning in a single direction will destroy the spatial isotropy of the image, leading to directional deviation. Therefore, this application constructs a four-directional scanning sequence: First, the joint features are flattened into a one-dimensional sequence according to the original spatial order (i.e., row-major, from left to right and from top to bottom), with a sequence length L = H × W (H is the feature height, W is the feature width), resulting in the sequence for the first direction; then, the joint features are transposed in the height and width dimensions, resulting in H'-W' = The feature map of WH is then flattened into a one-dimensional sequence according to the transposed spatial order to obtain the sequence of the second direction. Next, the two directional sequences are flipped in reverse along the sequence length direction to obtain the sequences of the third and fourth directions. Thus, the four directional sequences correspond to: original spatial order forward, transposed spatial order forward, original spatial order reverse, and transposed spatial order reverse. The sets of these four directional sequences are stacked together to form a four-directional sequence tensor with a directional dimension. The tensor shape is B-4-LC (4 is the number of directions). Specifically, for each direction of the sequence tensor in the four-directional sequence, the joint features are decomposed along the channel dimension to obtain the various core parameters required for state space evolution. The specific decomposition method is as follows: First, the joint features are decomposed into two parts. The first part contains the joint information of the state input modulation parameter B and the output modulation parameter C, and the second part contains the time step parameter dt. The number of channels in the first part is d. ssm + 2×g×d state The second part has the number of channels as nheads; then, the first part is further divided into three parts: state input x, input modulation parameters B, and output modulation parameters C, where the number of channels for state input x is d. ssm The number of channels for both input modulation parameter B and output modulation parameter C is g×d. state We obtained four core parameters for state space evolution in each direction: state input sequence x, input modulation parameter B, output modulation parameter C, and time step parameter dt. Among them, B, C, and dt are all generated by the content-aware parameter features of another modality. Specifically, the continuous state-space model needs to be discretized. The preset state matrix parameter A is stored in logarithmic form. Combined with the time step parameter dt obtained from the decomposition, the discrete state transition matrix is ​​calculated using the discretization formula. and discrete input matrix The differential equations of the continuous state-space model are transformed into discrete recursive equations. The evolution of the discretized state space follows the following recursive logic: the feature state at the current time step depends on a linear combination of the feature state at the previous time step and the input sequence at the current time step; the output at the current time step depends on the feature state at the current time step. Specifically, for each position t in the sequence, the feature state update formula is: The output formula is: y t = C·h t Among them, h t Let h be the state characteristic at time t. t-1 Let x be the state characteristic at time t-1. t Let y be the input sequence feature at time t. t The output feature at time t; Overall, the state-space model defines a linear mapping: y = SSM(A,B,C)(x) = Mx The elements of matrix M satisfy M ji = C j T A j …A i + 1 B i Where M represents the linear mapping relationship formed by state propagation and input-output mapping; In the actual calculation process, in order to improve the computational efficiency, the position-by-position recursion method is not adopted. Instead, a parallel scanning algorithm is used to organize the entire sequence into a preset block size and then perform parallel calculations to obtain the evolution output results of the entire sequence in four directions at once. Specifically, after parallel scanning computation, the sequence in each direction yields a corresponding output result, the shape of which is batch-sequence length-number of heads-single-head dimension (BL-nheads-d). head ), where the single-head dimension headdim = d ssm / nheads: First, the output results are merged along the head count dimension to restore the batch-sequence length-feature dimension (BLd). ssmThe output sequences in the four directions are then restored to the spatial order of the original image and converted into two-dimensional feature maps. The specific restoration rules are as follows: For the output sequence in the forward direction of the original spatial order, the one-dimensional sequence is directly reshaped into a height-width (HW) two-dimensional feature map; for the output sequence in the reverse direction of the original spatial order, it is first flipped in the reverse direction along the sequence length and then reshaped into an HW two-dimensional feature map; for the output sequence in the forward direction of the transposed spatial order, it is first reshaped into a WH two-dimensional feature map and then transposed to restore the original spatial order HW; for the output sequence in the reverse direction of the transposed spatial order, it is first flipped in the reverse direction along the sequence length and then reshaped into a WH two-dimensional feature map, and finally transposed to restore the original spatial order HW; after the spatial restoration of the output sequences in all directions is completed, the outputs in all four directions are converted into two-dimensional feature maps with the same size as the joint feature space and are unified to the original spatial order; finally, the two-dimensional feature maps in these four directions are added element-wise at the corresponding spatial positions to obtain the fused state space output feature map; Specifically, a gated normalization operation is performed on the fused state space output: First, the state space output is normalized in the channel dimension to eliminate numerical distribution differences between different batches of features. Then, the state gate component obtained from the jump connection branch is used as a gate signal to modulate the normalized features. Furthermore, gate and auxiliary components are separated from the jump connection path. The gate component is processed by the SiLU activation function and then multiplied element-wise with the auxiliary component to obtain the jump connection auxiliary features. Finally, the jump connection auxiliary features and the gated normalized state space output feature map are concatenated in the channel dimension to obtain a concatenated feature map that fuses global evolution features and local detail features. An output linear mapping operation is then performed on the concatenated feature map to map its internal channel dimension back to the same dimension d as the input features. model Finally, the updated first modal features are obtained; the same cross-modal state space modeling operation is performed on the initial features of the second modality to obtain the updated second modal features; After obtaining the updated first and second modal features for the encoding level of the modeling process, a consensus fusion operation is further performed to generate a fused feature map for this level. The first step of the consensus fusion operation is local convolution enhancement processing. The updated first and second modal features are subjected to 3×3 depthwise convolution operations respectively. The stride of the depthwise convolution is set to s=1, and the padding size is set to p=1 to ensure that the spatial size of the features remains unchanged before and after the convolution processing, and that the number of convolution groups is equal to the number of channels of the input features, achieving independent convolution operations for each channel. After local convolution enhancement, the output result is processed by the SiLU activation function to enhance the nonlinear expressive power of the features, ultimately obtaining the first local enhanced feature f1 and the second local enhanced feature f2, effectively enhancing the edge, texture, and other detailed information in the feature map. Multiply-add fusion is then performed: the first and second local enhanced features are multiplied element-wise to obtain an interaction term, which is then multiplied by the first and second local enhanced features respectively, d = f1⊙f2 + f1 + f2. The aforementioned intermediate fusion result d is input into a single-modal state space model for refinement. The difference between this single-modal state space model and the aforementioned cross-modal state space model is that its input modulation parameter B, output modulation parameter C, and time step parameter dt are all generated from the intermediate fusion result itself, without depending on another modality. The internal processing flow of the single-modal state space model also includes operations such as linear mapping, depthwise convolution, four-directional scanning and fusion, gated normalization, and skip connection concatenation. The final output state space fusion result is denoted as d. ssd Then, the shallow features of this encoding level (i.e., the initial features retained at the beginning of encoding) are obtained, and a linear mapping that keeps the number of channels unchanged is performed on these shallow features to obtain the linear mapping result of the shallow features, denoted as d. 1skip and d 2skip Finally, the state-space fusion result is added element-wise to the linear mapping results of these two shallow features to obtain the fused feature map y = d at this level. ssd + d 1skip + d 2skip ; The decoding network outputs updated first and second modal features at the deepest encoding level (Nth level). The input to the decoding network is the fusion result of these two features. The fused feature map is decoded and reconstructed level by level. The updated first modal feature output from the fourth level of encoding is added element-wise at the corresponding spatial positions to obtain the decoding starting feature. The decoding network contains N decoding levels set from low to high resolution. The N decoding levels correspond one-to-one with the N encoding levels according to resolution. The input of each decoding level includes the upsampled result of the previous level's decoding output and the fused feature map of the corresponding encoding level. In each decoding level, the resolution is doubled and the number of channels is halved through reversible convolutional upsampling. Specifically, for each decoding operation, the current decoding feature is first added element-wise to the fused feature map of the same encoding level (generated and saved by the consensus fusion module). This addition operation introduces multi-scale information retained in the encoding stage. Then, the added feature is divided into two branches along the channel dimension, with the number of channels in each branch being half the number of input channels, denoted as hidden. dim / 2 The first branch performs local convolution processing, specifically batch normalization, 3×3 convolution, batch normalization, ReLU activation, 3×3 convolution, batch normalization, ReLU activation, 1×1 convolution, and ReLU activation, resulting in a local convolution output. The second branch performs state space modeling, using a single-modal state space model (parameter self-generation, four-directional scanning and fusion) for long-range dependency modeling, resulting in a state space branch output. The local convolution output and the state space branch output are then concatenated along the channel dimension. Next, a channel reassembly operation is performed, where the concatenated features are grouped and their channel order is swapped to promote full mixing of information from the two branches. The reassembled features are the decoding output for this stage. Then, the resolution is doubled through reversible convolutional upsampling, while the number of channels is halved. Reversible convolutional upsampling uses a convolution operation with a kernel size of 2 and padding of 1, and is implemented through frequency domain inversion. After N levels of decoding, the reconstructed image output by the decoding network is obtained. The spatial resolution of the reconstructed image is the same as that of the preprocessed original image (256×256 pixels), and the number of channels is 1. In order to further preserve the structural information in the original input, the reconstructed image output by the decoding network is added element by element to the preprocessed first modality medical image and the preprocessed second modality medical image to obtain the final fused image.

[0027] In the above technical solution, cross-modal dependency is introduced at the feature representation stage, avoiding information loss caused by fusion after independent feature extraction, thus improving fusion efficiency and information integrity. The state space model is used for sequence modeling, and the computational complexity is linearly related to the sequence length. Compared with the quadratic complexity of Transformer, it is more suitable for high-resolution medical image processing. The four-directional scanning and fusion strategy overcomes the directional deviation of unidirectional scanning and achieves isotropic long-range dependency capture. The multiply-add operation (f1⊙f2+ f1+ f2) in consensus fusion and the single-modal SSM refinement effectively integrate complementary information between modalities and suppress redundancy. The reversible convolution upsampling is based on frequency domain inversion, which can accurately recover high-frequency details and improve the edge sharpness of the fused image. The skip connection and residual addition mechanism in the decoding network ensure that shallow detail information can be transmitted to the final output, improving the overall quality of the fusion result and realizing cross-modal parameter coupling medical image fusion.

[0028] In another technical solution, the four-directional scanning and fusion includes: constructing sequences of four directions: original spatial order forward, transposed spatial order forward, original spatial order reverse, and transposed spatial order reverse; performing state space evolution on each direction sequence; and then restoring the outputs of each direction to the original spatial order and adding them element by element.

[0029] First, obtain the feature map of the current coding level after cross-modal state space modeling. The dimensions of this feature map are batch, height, width, and channel. Set the height of the feature map to H and the width to W. Then the length of the flattened sequence is L = H×W. The steps to construct the four-directional sequence are as follows: Flatten the feature map into a one-dimensional sequence of length L in the original spatial order (row priority, from left to right, from top to bottom) to obtain the original spatial order forward sequence. Transpose the feature map in the height and width dimensions, and then flatten it in row priority to obtain the transposed spatial order forward sequence. Reverse the above two sequences along the length direction to obtain the original spatial order reverse sequence and the transposed spatial order reverse sequence, thus obtaining the four-directional sequence. For each direction of the sequence, state space evolution is performed to obtain the output sequences in four directions. Then, the original spatial order is restored according to the following rules: the original forward sequence is directly reshaped into an H×W feature map; the original reverse sequence is first reverse-flipped and then reshaped; the transposed forward sequence is first reshaped into a W×H and then transposed back into H×W; the transposed reverse sequence is first reverse-flipped, then reshaped into a W×H, and finally transposed back into H×W. Finally, the feature maps in the four directions are added element by element to obtain the fused state space output.

[0030] In the above technical solution, by constructing scanning sequences in four directions and performing state space evolution on each direction before fusion, the spatial anisotropy problem caused by single-direction scanning is effectively overcome. In medical images, lesion areas and normal tissues often exhibit complex directional distributions. Four-direction scanning enables the state space model to capture cross-regional structural dependencies from multiple angles such as horizontal, vertical, and reverse directions, thereby achieving near-isotropic long-range modeling capabilities while maintaining linear computational complexity.

[0031] In another technical solution, the local convolution enhancement is achieved through 3×3 depthwise convolution and SiLU activation. Specifically, the updated two modal features are subjected to depthwise convolution with a kernel of 3×3, a stride of 1, padding of 1, and a group number equal to the number of channels, followed by SiLU activation to obtain locally enhanced features. This effectively extracts texture and edge information within the local spatial neighborhood while maintaining computational efficiency. The grouping design of the depthwise convolution avoids parameter redundancy across channels, and the self-gating characteristic of the SiLU activation function helps improve gradient flow. The state space model refinement adopts a single-modal state space model, whose parameters are generated from its own features. The processing flow of this model includes linear mapping, depthwise convolution, four-directional scanning and fusion, gating normalization, skip connection concatenation, etc., ultimately outputting the state space fusion result. This avoids the additional computational overhead caused by cross-modal coupling, allowing the model to focus on the global dependencies within the current fused features. Compared to simple addition or concatenation, the multiply-add fusion operation (f1⊙f2+ f1+f2) better highlights the regions of common interest to both modalities while preserving their unique information.

[0032] In another technical solution, the reversible convolution upsampling is achieved through frequency domain inversion, including: performing zero-interpolation upsampling and nearest-neighbor upsampling on the input features, transforming the learnable convolution kernel to the frequency domain, constructing and solving the frequency domain equation, and then inversely transforming it back to the spatial domain.

[0033] First, zero-interpolation upsampling and nearest-neighbor upsampling are performed on the input features: Let the spatial dimensions of the input features be height H, width W, number of channels C, and upsampling factor 2. Zero-interpolation upsampling: Create a zero tensor of size 2H×2W×C, place each pixel value in the original features into the even-numbered row and even-numbered column of the new tensor, and fill the remaining positions with zeros. Nearest-neighbor upsampling: The input features are enlarged to size 2H×2W through nearest-neighbor interpolation, that is, each original pixel value is copied into the adjacent 2×2 blocks in the target grid. Then, the learnable convolution kernel is transformed to the frequency domain: a zero tensor with the same size as the target recovery (2H×2W) is constructed, the value of the learnable convolution kernel (size 2×2) is copied to the upper left corner of the tensor, and then a cyclic shift is performed to align the center of the convolution kernel. Then a two-dimensional Fourier transform is performed to obtain the frequency domain convolution kernel representation, and the conjugate term and magnitude square term of the frequency domain convolution kernel are calculated. Next, the frequency domain equation is constructed and solved: a two-dimensional Fourier transform is performed on the zero-interpolation upsampling result, and the result is multiplied element-wise with the conjugate convolution kernel to obtain the intermediate result in the frequency domain. The bias correction term is multiplied element-wise with the nearest neighbor upsampling result and then Fourier transformed. The result is then added to the intermediate result in the frequency domain to obtain the right-hand term in the frequency domain. The frequency domain convolution kernel is multiplied element-wise with the right-hand term, and the average is calculated in blocks according to the upsampling factor (2×2). The amplitude square term is also averaged in blocks. The inversion correction amount is constructed from the block average result. The correction amount is subtracted from the right-hand term and then divided by the bias correction term to obtain the recovered frequency domain representation. Finally, the inverse transform is performed back to the spatial domain: a two-dimensional inverse Fourier transform is performed on the recovered frequency domain representation, and its real part is taken to obtain high-resolution spatial features. If there was previous boundary padding, the padding region is cropped in the same proportion. Finally, the number of channels is halved by a 1×1 convolution, and a feature map with a size of 2H×2W×(C / 2) is output.

[0034] By implementing reversible convolutional upsampling through frequency domain inversion, high-frequency details in high-resolution features can be accurately recovered, avoiding the blurring effect of traditional interpolation methods. In the SPECT-MRI fusion task, this method achieved significantly higher peak signal-to-noise ratio and structural similarity metrics than the contrasting methods, demonstrating its ability to effectively preserve edge sharpness and texture details in the fused image. Furthermore, the upsampling process is reversible, which is beneficial for gradient backpropagation and model optimization.

[0035] like Figure 1 As shown, a cross-modal parameter-coupled medical image fusion system includes: The input and preprocessing module is used to acquire the first and second modal medical images. During the training phase, it first obtains the file paths of the two input images according to the pre-defined data root directory and the folder names corresponding to the two modalities. The two images are then arranged in a uniform order. If the number of images in the two paths is inconsistent, the path with the fewer images is used, and the other path sequence is truncated to the same length. Based on the sample number, two images at the same position are extracted from the two path sequences as a set of input images. The images are read in a normal image format, using color mode. Each input image is read as three-channel image data. The channel arrangement order of the two three-channel images is adjusted to meet the standard color channel order required for subsequent network processing. The spatial size of each image is uniformly transformed to 256×256. The pixel values ​​in the image are converted to floating-point form and then scaled to between 0 and 1. Let the original pixel value be p, and the processed pixel value be p', satisfying p' = p / 255. The image data is transformed from a height, width, and channel arrangement to a channel, height, and width arrangement. The two images that have completed the dimensional rearrangement are converted into data tensors that the network can directly process. Each image corresponds to a three-dimensional tensor with 3 channels and a spatial size of 256×256. According to the batch size preset in the training phase, multiple sets of preprocessed sample pairs are combined along the batch dimension so that multiple images of the same modality form a four-dimensional tensor. The images of the other modality also form a corresponding four-dimensional tensor. The two image tensors after batching are fed into the subsequent fusion model as data input to the network. This module outputs the preprocessed first modality medical image and the preprocessed second modality medical image. A dual-branch cross-modal representation learning module is connected to the input and preprocessing module to perform multi-level feature extraction. The module adopts a dual-branch coding structure, with different modal images entering their respective coding branches for feature extraction. During the coding process, information interaction is achieved through a cross-modal state space modeling mechanism. The coding structure consists of multiple levels of cascaded representation learning units, which are connected through resolution transformation units. The representation learning units are stacked in four levels along the coding path, and downsampling operations are introduced between adjacent levels to gradually reduce spatial resolution and improve channel expressiveness. Each level of representation learning unit embeds a state space-based cross-modal representation learning sub-module to introduce cross-modal dependencies during the feature generation stage, while combining local feature modeling and residual connection mechanisms. The dual-branch cross-modal representation learning module specifically includes a data embedding submodule, a state-space-based cross-modal representation learning submodule, and a resolution transformation and feature transfer submodule; The data embedding submodule first divides the input image into fixed-size local regions and simultaneously performs channel mapping, converting the original image into embedding features that can be processed by subsequent networks. The size parameter of the local region is set to 4, i.e., a 4×4 local region division method is adopted. The number of input channels is 3 and the number of embedding channels is 128. Two-dimensional convolution mapping is performed on the input image. The size of the convolution kernel is set to 4×4, the stride is set to 4×4, the number of input channels is 3, and the number of output channels is 128. The convolution kernel slides along the height and width directions of the input image in a non-overlapping manner, covering a 4×4 local region each time, and mapping the 3-channel data in the local region to a 128-dimensional output feature. The convolution operation is repeated until all spatial regions of the input image have been traversed. The feature tensor output by convolution is rearranged in dimension, from batch, channel, height, width to batch, height, width, channel, so that each spatial position corresponds to a 128-dimensional embedding vector. Layer normalization is performed on the rearranged embedding features to obtain the data embedding output. like Figure 2 As shown in the figure This represents the activation function. The expression represents pointwise multiplication, N represents normalization, and Z is the skip connection component mentioned earlier in this module. The state-space-based cross-modal representation learning submodule receives two input features, denoted as the first modal feature and the second modal feature, respectively. Both input features are arranged in batch, height, width, and channel format. The batch size B, spatial height H, spatial width W, and number of channels C are read. The sequence length L = H × W is calculated, and the number of scanning directions is set to 4. Cross-modal combination selection is performed on the two input features using a "none" combination method. That is, the inputs used to generate content-aware parameters from the first modality directly take the original features of the second modality, and the inputs used to generate content-aware parameters from the second modality directly take the original features of the first modality. Linear mapping of skip connection branches is performed on the two input features to obtain the intermediate features of the skip connections, with an output channel count of 2 × d. inner -d ssm , where d inner =expand×d model Preferably, expand = 2, so the total number of internal channels is twice the number of input channels. The intermediate features of the two skip connections are split into three parts, and the number of channels in the first two parts after splitting is denoted as d. mlp The number of channels in the third part is d. ssm , where d mlp = (Number of intermediate feature channels in skip connections - d) ssm After splitting by 2, the subsequent state space output, together with the first two parts, constitutes the features before the final output mapping. Linear mapping of the state input branches is then performed on the two input features to obtain the state input features. The number of output channels is d.ssm Then, the dimensional order is transformed to batch, channel, height, and width, and two-dimensional depthwise convolutions are performed on the two state input features respectively, with the convolution kernel size set to d. conv ×d conv The step size is 1, and the padding is (d). conv -1) / 2, the number of convolutional groups equals the number of channels. After convolution, SiLU activation is performed. Linear mapping of content-aware parameter branches is performed on the two cross-modal content features respectively to obtain content-aware intermediate parameters. The number of output channels is 2×ngroups×d state + nheads, perform dimension adjustment, 2D deep convolution, and activation on the two content-aware intermediate parameters respectively. Concatenate the state input features of each path with the corresponding content-aware parameter features in the channel dimension to obtain joint features. Construct four-directional scanning sequences for the two joint features, including forward in the original spatial order, forward in the transposed spatial order, reverse in the original spatial order, and reverse in the transposed spatial order. Perform parameter decomposition on the two four-directional sequences to obtain state input x, parameter B, parameter C, and dt. Perform multi-head partitioning on the state input x, with nheads and a single head dimension of head. dim =d ssm / nheads flattens the time-step offset parameters, rearranging the pre-saved time-step offsets corresponding to the four directions into a one-dimensional vector, which serves as the correction term for subsequent time-step parameters, generating the jump connection parameter D. If D needs to include the internal dimension of a single head, it is rearranged into the number of heads and the dimension of a single head; if D does not include the internal dimension of a single head, it is directly kept in its original form. Preferably, the latter method is adopted, so D remains a jump connection parameter organized by head. Two sets of pre-saved logarithmic parameters are taken, their exponents are taken first, and then a negative sign is added to obtain the state matrix parameters corresponding to the first and second modes, i.e., A. s = -exp(A log ), where A log For the pre-saved logarithmic parameter, A s For the state matrix parameters used in discretization, parameters B and C are grouped and rearranged, and the state-space core computation is performed. The discrete state-space evolution can be written as: y t = C·h t Overall, the state-space model defines a linear mapping: y = SSM(A,B,C)(x) = Mx The elements of matrix M satisfy M ji = C j T A j …Ai + 1 B i Parallel scanning is used to obtain four-directional outputs. Direction reconstruction and fusion are then performed, with the results from the four directions added element-wise to obtain the fused state space output. Gating normalization is then executed, using the state-gated components as the gating inputs. If d mlp If the value is >0, SiLU activation is first performed on the gated component, and then it is multiplied element-wise with the auxiliary component to obtain the skip connection auxiliary feature. Then, the auxiliary feature is concatenated with the state space output on the channel dimension, and the output linear mapping is performed on the concatenation result. The skip connection branch is concatenated, and the output linear mapping is performed on the concatenation result to obtain the updated first modality feature and second modality feature. The resolution transformation and feature transfer submodule receives the feature tensor output from the previous stage, arranged in batch, height, width, and channel order. It adjusts the input feature dimensions to batch, channel, height, and width, then performs a 2D convolution operation with a 2×2 kernel size and stride. No bias term is set, the number of input channels is C, and the number of output channels is 2C. The convolution kernel moves 2 pixels spatially each time, without overlap calculations. This maps the input features within the corresponding 2×2 local region to the output features at the current position. The spatial height and width of the output features are halved, and the number of channels is doubled. The convolution output is then adjusted again in batch, height, width, and channel order. Layer normalization is performed on the transformed output features, which are then passed to the next stage of the network as the current stage's output features. The two-branch cross-modal representation learning module is configured to perform four-level concatenated coding, outputting the first and second modal features updated at the fourth level, as well as intermediate features at each level; like Figure 3 As shown, the consensus fusion representation construction module is connected to the output of each level of the dual-branch cross-modal representation learning module. The inputs include two stages of features after cross-modal representation learning and two shallow features without cross-modal representation learning. All four input features are arranged in batch, height, width and channel. Linear mapping is performed on the two shallow features to obtain the first and second skip connection features. The number of input and output channels is equal to the current stage channel count (dim). After processing, the spatial dimensions of the two skip connection features remain unchanged, and the number of channels is kept to dim. The dimensionality order of the two stage features after cross-modal representation learning is adjusted, transforming them into batch, channel, height, and width. Two-dimensional depthwise convolution is then performed on the two stage fusion features, with the number of input and output channels both set to dim, the number of convolution groups set to dim, the kernel size set to 3×3, and the stride set to dim. With a padding value of 1 and a bias term, SiLU activation is performed on the two convolutional results to obtain the first and second local enhancement features. These features are then restored to their post-channel form and transformed back to batch, height, width, and channel values. Element-wise fusion is then performed on the two local enhancement features: first, the first and second local enhancement features are multiplied element-wise to obtain the interaction term; then, this interaction term is added element-wise to both the first and second local enhancement features to obtain the fusion result for the current stage, calculated as d = f1⊙f2 + f1 + f2. This fusion result is then fed into a single-modal state-space modeling unit. This unit no longer performs cross-modal parameter generation but only performs state-space modeling on the currently explicitly fused features, obtaining the refined fused features, denoted as d. ssd d ssd The final output of the fusion module is obtained by adding the features of the two skip connections element-wise, and its calculation form is y = d. ssd + d 1_skip + d 2_skip Output the final fusion result, keeping the original spatial dimensions unchanged and the number of channels unchanged at the current stage's channel count (dim); like Figure 4 As shown, This indicates point-by-point multiplication. This represents point-by-point addition. The decoding and reconstruction module is connected to the fourth-level output of the dual-branch cross-modal representation learning module and the outputs of each level of the consensus fusion representation construction module. The decoding and reconstruction module is used to restore and refine the deep fusion features obtained in the encoding stage. The whole adopts a four-level decoding structure, which corresponds one-to-one with the four-level hierarchical structure of the encoding stage, and combines the jump connection features from the encoding stage in the decoding process of each level. The decoding and reconstruction module includes a state-space decoder submodule, a resolution restoration and feature reconstruction submodule, and an end resolution restoration submodule; State-space decoder submodule: Input features are arranged in batch, height, width, and channel order. The input features are divided into two parts along the last channel direction, with each part having a hidden number of channels. dim / 2, the first part serves as the input to the local convolutional branch, and the second part serves as the input to the single-modal state space branch. The features entering the single-modal state space branch are first normalized by the layer, and then fed into the single-modal state space modeling unit. In this unit, a unified linear mapping is first performed on the input features, transforming them into four intermediate quantities, corresponding to jump connection related components, state input components, state space parameter components, and time step parameter components, respectively. After the linear mapping, the output is split into four parts along the channel dimension. A dimensional transformation is performed on the joint components used for state space scanning, followed by two-dimensional depthwise convolution and SiLU activation. The result of the convolution activation is then processed... If a sequence representation with four directions is constructed, the four-directional sequence is parametrically decomposed to obtain the input sequence x, parameter B, parameter C, and dt. The remaining operations are the same as the subsequent steps of the state space-based cross-modal representation learning submodule in the two-branch cross-modal representation learning module, resulting in the output of the single-modal state space branch. Dimensional transformation is performed on the input of the local convolutional branch, transforming it into batch, channel, height, and width. Convolution, normalization, and activation are repeatedly performed on the local convolutional branch to complete three layers of local feature transformation: the first layer performs batch normalization, followed by a 3×3 convolution with a stride of 1 and padding of 1. Both the input and output channels are hidden. dim / 2, after convolution, batch normalization and ReLU activation are performed; the second layer continues with a 3×3 convolution with a stride of 1 and padding of 1, keeping the number of input and output channels unchanged, and then batch normalization and ReLU activation are performed again; the third layer performs a 1×1 convolution with a stride of 1 and no padding, keeping the number of input and output channels hidden. dim / 2, After convolution, ReLU activation is performed to obtain the output of the local convolution branch. The output of the local convolution branch is restored to the channel-post format, transformed back to batch, height, width, and channels, and then concatenated with the output of the local convolution branch and the output of the single-modal state space branch on the channel dimension. The number of output channels is restored to the original number of input channels. dim The splicing result is reorganized into channels. First, the channel dimension is divided into two groups of 2. Then, the positions of the channels within the group and the channels between the groups are swapped. Finally, the swapped result is flattened back to the original number of channels to obtain the final output of the current state space decoder. The resolution restoration and feature reconstruction submodule receives input features and reads their batch size, spatial height, spatial width, and number of channels. The input features are arranged in batch, height, width, and channel order. Let the number of input channels be *dim* and the upsampling factor be 2. The dimensionality of the input features is adjusted to batch, channel, height, and width. The transformed features are then fed into the reversible convolution restoration process. It is determined whether padding is needed. In this implementation, the convolution kernel size is 2, the upsampling factor is 2, and the padding size is 1. Therefore, boundary padding is first performed on the input features. A transformation is then performed on the learnable bias to generate a positive stable term. Specifically, the bias is first subtracted by a constant, and then a Sigmoid transformation is performed. The process involves replacing the input features with a stability constant, constructing a zero-interpolation upsampling result, and creating a higher-resolution blank feature map on each channel of the input features, doubling its spatial height and width. The original input features are then copied to positions in the enlarged feature map at fixed intervals, with the remaining new positions filled with zeros. The current input features are then synchronously enlarged to twice the resolution using a nearest-neighbor approach. The learnable convolutional kernel is transformed from the spatial domain to the frequency domain, constructing a zero tensor with the same size as the target recovery. The learnable convolutional kernel is copied to the upper left region of this tensor, followed by a cyclic shift along the spatial dimension, and then a two-dimensional Fourier transform to obtain the frequency domain convolutional kernel representation. Based on the frequency domain convolutional kernel calculation... Calculate its conjugate term and magnitude squared term. Perform a frequency domain transformation on the zero-interpolation upsampling result and multiply it with the conjugate convolution kernel to obtain the intermediate frequency domain result. Multiply the bias correction term element-wise with the amplified input features, and then perform a two-dimensional Fourier transform on the result. Subsequently, add it element-wise with the intermediate frequency domain result to obtain the right-hand term in the frequency domain. Multiply the frequency domain convolution kernel element-wise with the right-hand term in the frequency domain. Divide the result into four sub-blocks in space according to the upsampling factor of 2×2, and average them along the dimension of each sub-block to obtain the block average result. Perform the same block processing on the magnitude squared term of the frequency domain convolution kernel to obtain the block average energy term. Divide the block average result by the sum of the block average energy term and the bias correction term. The correction coefficients are obtained, and then repeatedly extended back to the high-resolution scale by the upsampling factor. Subsequently, they are multiplied element-wise with the conjugate convolution kernel to obtain the frequency domain inversion correction term. The frequency domain inversion correction term is subtracted from the right-hand side of the frequency domain term, and then divided by the bias correction term to obtain the restored frequency domain representation. A two-dimensional inverse Fourier transform is performed along the spatial dimension on the restored frequency domain representation, and its real part is taken to obtain the restored high-resolution spatial features. Since the padding size is 1 and the upsampling factor is 2, two pixels are cropped from each of the four sides of the restored result to obtain the upsampled features after reversible convolution restoration. Channel mapping is performed on the restored upsampled features, and a 1×1 convolution is performed to compress the number of channels from dim to dim / dim. scale Due to dim scale= 2, compress the number of channels from dim to dim / 2, so the number of output channels becomes half of the number of input channels. Perform dimensional order restoration on the channel-mapped features, transform back to batch, height, width, and channels. Perform layer normalization on the restored features to obtain the upsampling restoration result of the current stage, and pass the output to the next decoding stage; The end-resolution restoration submodule receives input features and reads their batch size, spatial height, spatial width, and number of channels. The input features are arranged in the order of batch, height, width, and channels. Let the number of input channels be dim, and the total end-resolution restoration factor be 4. It performs two reversible convolutional upsampling operations, the same as those in the resolution restoration and feature reconstruction submodules. Channel mapping is performed between the two restorations and after restoration, doubling the resolution and halving the number of channels each time to obtain high-resolution features consistent with the scale of the initial embedding layer. The dimensionality order of the features output by the end-reversible convolutional restoration submodule is adjusted, transforming them into batch, channels, height, and width. The final channel mapping is performed on the transformed features using a single two-dimensional convolution with a kernel size of 1×1. The number of input channels is 128, the number of channels in the initial embedding, and the number of output channels is 1. Without changing the spatial height and spatial width, the multi-channel features are compressed into single-channel output features. The final output features are then added element-wise to the two original input images to output the final fused image.

[0036] In the above technical solution, cross-modal parameter coupling medical image fusion is achieved through the organic combination of an input and preprocessing module, a bi-branch cross-modal representation learning module, a consensus fusion representation construction module, and a decoding and reconstruction module. The data embedding sub-module, cross-modal representation learning sub-module, and downsampling sub-module in the bi-branch cross-modal representation learning module work collaboratively to introduce cross-modal dependencies at the feature representation stage, avoiding information loss caused by fusion after independent feature extraction. The consensus fusion representation construction module effectively integrates complementary information between modalities through local convolution enhancement, multiply-accumulate fusion, and single-modal state space refinement. The state space decoder sub-module, resolution restoration and feature reconstruction sub-module, and terminal resolution restoration sub-module in the decoding and reconstruction module achieve high-quality reconstructed image output through reversible convolution upsampling and channel reconstruction. This system demonstrates excellent performance in various medical image fusion tasks such as CT-MRI, PET-MRI, and SPECT-MRI, and has promising clinical application prospects.

[0037] To verify the effectiveness of the cross-modal parameter coupling medical image fusion method (CSCFusion) proposed in this application, comparative experiments were conducted on three typical multimodal medical image fusion tasks: CT-MRI, PET-MRI, and SPECT-MRI.

[0038] Imaging characteristics of each modality: CT images are good at displaying high-density structures such as bones and calcifications, with sharp edges but low contrast in soft tissues; MRI images are good at displaying fine anatomical structures such as soft tissues and gray and white matter in the brain, with clear textures but weak bone signals; PET images reflect the level of glucose metabolism and can be used to detect high metabolic areas such as tumors or inflammation, but have low spatial resolution and blurred structures; SPECT images reflect blood flow or receptor distribution, with low signal-to-noise ratio and high noise, making it the functional modality with the worst imaging quality.

[0039] Comparison Methods: The current mainstream multimodal medical image fusion methods were selected, including IFT (traditional multiscale transformation method), SwinFusion (fusion method based on Swin Transformer), MATR (fusion method based on cross-modal attention Transformer), BSAFusion (CNN fusion method based on bidirectional self-attention) and FusionMamba (state space fusion method based on unimodal Mamba).

[0040] All comparative experiments used the same training / test split and image preprocessing workflow (size normalized to 256×256, pixel values ​​normalized to [0,1]). Evaluation metrics included structural similarity (SSIM), peak signal-to-noise ratio (PSNR), visual information fidelity (VIF), and feature mutual information (FMI).

[0041] Structural Similarity (SSIM): Measures the similarity between the fused image and the source image in three dimensions: brightness, contrast, and structure. The value ranges from [0,1], and the higher the value, the better the structural fidelity.

[0042] Peak Signal-to-Noise Ratio (PSNR): Measured in decibels (dB), it evaluates the pixel-level reconstruction quality of a fused image. A higher value indicates less distortion and higher reconstruction accuracy.

[0043] Visual Information Fidelity (VIF): Based on the human visual perception model, it measures the amount of information from the source image retained in the fused image. The higher the value, the better the visual quality.

[0044] Feature Mutual Information (FMI): Evaluates the total amount of mutual information retained from the two source modalities in the fused image. A higher value indicates a more complete integration of complementary information between the modalities.

[0045] like Figure 5 As shown, the first column is the first modality source image (CT / PET / SPECT), the second column is the second modality source image (MRI), the third to seventh columns are the fusion results of BSAFusion, FusionMamba, IFT, SwinFusion, and MATR, and the eighth column is the fusion result of CSCFusion of this invention.

[0046] Example 1: CT-MRI Fusion Task The registered CT and MRI images are used as the first and second modal inputs, respectively. The method of this application is used for four-level coding, cross-modal state space modeling, consensus fusion and decoding reconstruction to obtain fused images. At the same time, IFT, MATR, SwinFusion and other methods are used to fuse the same data and calculate various indicators.

[0047] This application achieves optimal results in four metrics: SSIM, PSNR, VIF, and FMI. The SSIM score reaches 0.7059, significantly higher than methods such as IFT, MATR, and SwinFusion. A SSIM score closer to 1 indicates higher structural consistency between the fused and source images. The value of 0.7059 demonstrates that this application effectively preserves the sharp edges of bones in CT images and the natural contrast of soft tissues in MRI images, resulting in anatomical structures without ghosting or distortion in the fused image. The PSNR score reaches 17.60, a significant improvement over other methods. PSNR reflects pixel-level reconstruction accuracy; a higher PSNR means higher fidelity in the fused image. With less noise and clearer details, the fused image is better suited for doctors to observe minute lesions. The VIF index reaches 0.2907, indicating that the fused image retains richer source image information under human visual perception, making it less likely to cause visual fatigue during clinical observation. The FMI index reaches 1.5715, which specifically measures the degree of complementary preservation of cross-modal information. The value of 1.5715 is significantly higher than the comparison method, proving that this application has successfully fused the skeletal information of CT and the soft tissue information of MRI at the feature level through cross-modal parameter coupling. In the visualization, the fusion result of this invention simultaneously retains the sharp edges of CT bones and the delicate texture of MRI soft tissue, without blurring or artifacts, and with the best structural fidelity.

[0048] Example 2: PET-MRI Fusion Task The registered PET and MRI images are used as the first and second modal inputs, respectively. The method of this application is used for four-level coding, cross-modal state space modeling, consensus fusion and decoding reconstruction to obtain fused images. At the same time, IFT, MATR and SwinFusion comparison methods are used to fuse the same data and calculate various indicators.

[0049] The experimental results of this application show that the PSNR reaches 20.27, which is basically on par with the current superior method SwinFusion (20.41), indicating that this method has reached an advanced level in pixel-level reconstruction accuracy. The SSIM index reaches 0.7044, the VIF index reaches 0.3525, and the FMI index reaches 2.3680, all of which are better than the comparison methods. The SSIM index of 0.7044 indicates that the fused image maintains the clarity of fine anatomical structures such as gyri and sulci in the MRI image. The VIF index of 0.3525 indicates that the human eye can clearly see both low-metabolic and high-metabolic areas at the same time, without the PET high-brightness signal obscuring the MRI structure. The FMI index of 2.3680 is the highest among all methods, which reflects the unique advantages of this application in integrating functional and structural modalities. In the visualization, this invention accurately anchors the PET functional lesions to the MRI anatomical location, with optimal synergy between functional and structural information, without obstruction or signal submersion.

[0050] Example 3: SPECT-MRI Fusion Task The registered SPECT and MRI images were used as the first and second modal inputs, respectively. The method of this application was used to perform four-level coding, cross-modal state space modeling, consensus fusion and decoding reconstruction to obtain fused images. At the same time, IFT, MATR and SwinFusion comparison methods were used to fuse the same data and calculate various indicators.

[0051] This application demonstrates the most significant advantage in the SPECT-MRI task, achieving a PSNR of 25.68, significantly higher than other methods. Given the high noise levels in SPECT images, the higher PSNR indicates that this application effectively suppresses noise propagation through cross-modal parameter coupling and consensus fusion mechanisms, resulting in cleaner fused images. The SSIM score reaches 0.7641, the VIF score reaches 0.3991, and the FMI score reaches 1.9793, all optimal results. The SSIM score of 0.7641 is the highest absolute value among the four tasks, reflecting almost no loss of MRI structural information during fusion. The VIF score of 0.3991 indicates excellent visual recovery capability even with low-quality SPECT input. The FMI score of 1.9793 proves that even with weak SPECT information, this application can still extract its functional features and align them with MRI. In the visualization, this invention effectively suppresses noise and preserves high-resolution MRI details in challenging low-quality, high-noise scenarios, demonstrating optimal robustness.

[0052] Example 4: Downstream Segmentation Task Verification To verify the clinical applicability of the fusion results, the fused image of this application was used as input and fed into the same segmentation model (e.g., 3D U-Net) for training and testing on the BRaTS brain tumor segmentation dataset. The BRaTS dataset contains MRI images and expert annotations in four modalities. In this application, the fused image of this application was used to replace the original single-modal or simple splicing input. The segmentation task was divided into three regions: the overall tumor region (WT) which includes all tumor-related tissues, the tumor core region (TC) which excludes edematous parts, and the enhanced tumor region (ET) which is the most active tumor core. The evaluation index was the Dice similarity coefficient, ranging from [0,1]. The higher the value, the better the overlap between the segmentation result and the real annotation.

[0053] like Figure 6 As shown, the first and second columns are the source modal T2 and Flair images, the third column is the fusion result of the comparison fusion method SwinFusion, the fourth column is the fusion result of the present invention (CSCFusion), the fifth to eighth columns are the segmentation results after inputting the fused images of T2, Flair, SwinFusion, and the present invention into the segmentation model (U-Net), and the last column is the expert-annotated gold standard (GT).

[0054] The segmentation model based on the fused image of this application achieves 94.94%, 94.51%, and 91.40% accuracy in WT, TC, and ET, respectively, all significantly outperforming results based on single-modal input and other fusion methods. Specifically, the 94.94% accuracy in the WT region indicates that the overall contour of the tumor in the fused image of this application is very clear, and the segmentation model can accurately define the tumor boundary without interference from surrounding normal tissue. The 94.51% accuracy in the TC region indicates extremely high distinguishability between the tumor solid and the edematous region, which is crucial for developing surgical or radiotherapy plans, allowing doctors to accurately determine the core area that needs to be resected. The 91.40% accuracy in the ET region is the most clinically challenging indicator, as enhanced regions are usually small in size and have irregular boundaries, making segmentation difficult. The 91.40% accuracy of this application means that the fused image retains the fine vascular structure and edge enhancement features of the enhanced tumor, enabling the model to accurately locate it. In summary, this application not only improves the quality of visual fusion but also significantly enhances the discriminative ability of the fusion results in downstream medical tasks, validating its potential for practical clinical application.

[0055] This application achieves the best or equivalent objective indicators in three typical medical image fusion tasks: CT-MRI, PET-MRI, and SPECT-MRI. Its advantages are particularly prominent in the most challenging SPECT-MRI task. Downstream BRaTS segmentation experiments further demonstrate the supporting value of the fusion results for clinical diagnosis. This application can achieve high-quality, cross-modal collaborative multimodal medical image fusion while ensuring linear computational complexity, and has good application prospects.

[0056] This application also provides an electronic device, including: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the cross-modal parameter coupling medical image fusion method described above. This electronic device can be any terminal device including mobile phones, laptops, desktop computers, tablets, PDAs (Personal Digital Assistants), POS (Point of Sales) terminals, in-vehicle computers, etc.

[0057] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the method described above. Through the above description of the embodiments, those skilled in the art will clearly understand that this application can be implemented using software plus necessary general-purpose hardware, or it can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented using corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for this application, software program implementation is more often a preferred implementation method. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the cross-modal parameter coupling medical image fusion method described in the various embodiments of this application.

[0058] This application also provides a computer program product comprising a computer program that, when executed by a processor, implements the aforementioned cross-modal parameter coupling medical image fusion method. The computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled languages, interpreted languages, declarative languages, or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program may, but does not necessarily, correspond to a file in a file system. The program can be stored as a portion of a file containing other programs or data, for example, stored in one or more scripts: in a markup language document; in a single file dedicated to the related program; or in multiple co-located files, for example, a file storing one or more modules, subroutines, or code portions. The computer program can be deployed to execute on one or more computers located in one place or distributed across multiple locations and interconnected via a communication network. The processing and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs by processing input data and generating output to run functions.

[0059] The number of devices and processing scale described herein are for the purpose of simplifying the description of this application. Applications, modifications, and variations of this application will be readily apparent to those skilled in the art.

[0060] Although the embodiments of this application have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for this application. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, this application is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A medical image fusion method with cross-modal parameter coupling, characterized in that, include: Acquire medical images from the first modality and the second modality; The first modal medical image and the second modal medical image are respectively subjected to size normalization, pixel value normalization and dimension rearrangement to obtain the preprocessed first modal medical image and the preprocessed second modal medical image; Multi-level feature extraction is performed on the two images through an encoding network. The encoding network contains N encoding levels set from high to low resolution, generating initial features of the first modality and initial features of the second modality for each level, and retaining the initial features of at least one encoding level as shallow features. Performing cross-modal state-space modeling on initial features at at least one coding level includes: generating first content-aware parameters from the second modal initial features to modulate the input modulation parameters, output modulation parameters, and time step parameters of the first state-space model, and performing a four-way scan and fusion on the state-space evolution to obtain updated first modal features; generating second content-aware parameters from the first modal initial features to modulate the input modulation parameters, output modulation parameters, and time step parameters of the second state-space model, and performing a four-way scan and fusion on the state-space evolution to obtain updated second modal features; For the encoding level of the execution model, the updated first modality features and the updated second modality features are locally convolutionally enhanced, then multiplied element-wise and added element-wise to themselves, then refined by the state space model, and finally added element-wise to the linear mapping result of the shallow features of this level to obtain the fused feature map of this level. The deepest encoding layer's fused feature map is decoded and reconstructed step by step by a decoding network. The decoding network contains N decoding layers set from low to high resolution. The N decoding layers correspond one-to-one with the N encoding layers according to resolution. The decoding input of each level includes the upsampled result of the previous level's decoding output and the fused feature map of the corresponding encoding layer. In each decoding level, the resolution is doubled and the number of channels is halved through reversible convolution upsampling. The reconstructed image output by the decoding network is added element-wise to the preprocessed first modality medical image and the preprocessed second modality medical image to obtain the final fused image.

2. The medical image fusion method with cross-modal parameter coupling according to claim 1, characterized in that, The four-directional scanning and fusion includes: constructing sequences in four directions: original spatial order forward, transposed spatial order forward, original spatial order reverse, and transposed spatial order reverse; performing state space evolution on each directional sequence; and then restoring the outputs of each direction to the original spatial order and adding them element by element.

3. The medical image fusion method with cross-modal parameter coupling according to claim 1, characterized in that, The local convolution enhancement is achieved through 3×3 depthwise convolution and SiLU activation; the state space model refinement adopts a single-modal state space model, whose parameters are generated from its own features.

4. The medical image fusion method with cross-modal parameter coupling according to claim 1, characterized in that, The reversible convolutional upsampling is achieved through frequency domain inversion, including: performing zero-interpolation upsampling and nearest-neighbor upsampling on the input features, transforming the learnable convolutional kernel to the frequency domain, constructing and solving the frequency domain equation, and then inversely transforming it back to the spatial domain.

5. A medical image fusion system with cross-modal parameter coupling, characterized in that, include: The input and preprocessing module is used to acquire the first modality medical image and the second modality medical image, and perform size normalization, pixel value normalization and dimension rearrangement respectively, and output the preprocessed first modality medical image and the preprocessed second modality medical image. A dual-branch cross-modal representation learning module, connected to the input and preprocessing module, is used to perform multi-level feature extraction. The multi-level feature extraction includes N encoding levels set from high to low resolution, generating initial features of the first modality and initial features of the second modality at each level, and retaining the initial features of at least one encoding level as shallow features. The dual-branch cross-modal representation learning module is also used to perform cross-modal state space modeling on the initial features of at least one encoding level, including generating input modulation parameters, output modulation parameters, and time step parameters of the first state space model from the initial features of the second modality, generating corresponding parameters of the second state space model from the initial features of the first modality, and performing four-directional scanning and fusion on the state space evolution, outputting updated first and second modal features. The consensus fusion representation construction module, connected to the dual-branch cross-modal representation learning module, is used to perform local convolution enhancement on the updated first modal features and second modal features, then perform element-wise multiplication and add them element-wise to themselves, then refine them through the state space model, and finally add them element-wise to the linear mapping result of the shallow features of this level to output the fusion feature map of this level. The decoding and reconstruction module, connected to the consensus fusion representation construction module, is used to decode and reconstruct the fusion feature map of the deepest encoding level step by step. It includes N decoding levels set from low to high resolution. The N decoding levels correspond one-to-one with the N encoding levels according to resolution. The decoding input of each level includes the upsampling result of the decoding output of the previous level and the fusion feature map of the corresponding encoding level. In each decoding level, the resolution is doubled and the number of channels is halved through reversible convolution upsampling. The decoding and reconstruction module is also used to add its output reconstructed image element-wise with the preprocessed first modality medical image and the preprocessed second modality medical image to output the final fusion image.

6. The medical image fusion system with cross-modal parameter coupling according to claim 5, characterized in that, The dual-branch cross-modal representation learning module includes: a data embedding submodule, used to divide the input tensor into non-overlapping local regions of a preset fixed size and convert them into embedding vectors through convolution mapping; a state-space-based cross-modal representation learning submodule, used to perform the cross-modal state space modeling, including constructing sequences in four directions: original space order forward, transposed space order forward, original space order reverse, and transposed space order reverse, and performing state space evolution and fusion; and a resolution transformation and feature transfer submodule, used to perform two-dimensional convolution downsampling with a stride of 2 on the updated features to double the number of channels and perform layer normalization.

7. The cross-modal parameter coupling medical image fusion system according to claim 5, characterized in that, In the consensus fusion representation construction module, the local convolution enhancement is a 3×3 depthwise convolution with SiLU activation, the state space model refinement adopts a single-modal state space model and its parameters are generated by its own features, and the linear mapping is a linear transformation that keeps the number of channels unchanged; the decoding and reconstruction module includes a state space decoder submodule, a resolution restoration and feature reconstruction submodule, and an end resolution restoration submodule; the state space decoder submodule is used to add the current decoded feature and the fusion feature of the same coding level element by element and then divide it into two branches along the channel dimension, perform local convolution processing and state space modeling processing respectively, and then perform channel recombination after splicing; the resolution restoration and feature reconstruction submodule is used to double the resolution and halve the number of channels by reversible convolution upsampling with a kernel size of 2 and padding of 1; the end resolution restoration submodule is used to perform two reversible convolution upsamplings to the size normalized resolution and map the number of channels to 1 by 1×1 convolution.

8. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the cross-modal parameter coupling medical image fusion method according to any one of claims 1-4.

9. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the medical image fusion method with cross-modal parameter coupling as described in any one of claims 1-4.

10. A computer program product comprising a computer program, characterized in that, When the program is executed by the processor, it implements the medical image fusion method with cross-modal parameter coupling as described in any one of claims 1-4.

Images