Cross-modal-based VMama medical image fusion method and system combining packet ACmix convolution and selective clustering

Abstract

The invention relates to the technical field of medical image and artificial intelligence crossing, and discloses a cross-modal-based VMama medical image fusion method and system combining grouped ACmix convolution and selective clustering, and the system comprises an input module, a preprocessing module, a feature extraction module, a multi-scale fusion module, and an output reconstruction module. According to the scheme, cross-modal attention is introduced into a visual state space model for the first time, a new multi-modal image fusion framework LMACV is obtained, the framework performs linkage optimization on ACmix and VMamba structures in a cross-modal medical image for the first time, feature reconstruction efficiency is enhanced through a selective clustering mechanism, and MSE and PSNR are remarkably superior to existing methods such as MPCT, FATFusion and MATR. By fusing a convolutional network and state space modeling, the network can effectively capture local texture details, and meanwhile, long-distance semantic association is reserved; besides, linear state updating and cross-modal dynamic alignment of attention guidance are effectively realized, and compared with the latest MPCT algorithm, the fusion speed is improved by 37.5%.

Description

Technical Field

[0001] This invention relates to the field of medical imaging and artificial intelligence, specifically to a medical image fusion method and system based on VMamba that combines cross-modal grouping ACmix convolution and selective clustering, applicable to medical image fusion and diagnosis of different modalities such as SPECT / MRI, PET / MRI, and CT / MRI. Background Technology

[0002] In clinical settings, each imaging modality has its unique advantages in diagnosis. For example, MRI excels at presenting soft tissue contrast, while CT is more effective at displaying the dense anatomical structures composed of bone. However, in complex scenarios such as tumor assessment, a single modality cannot simultaneously convey anatomical details and pathological changes. Clinicians often combine data from different imaging modalities, such as positron emission tomography (PET), single-photon emission computed tomography (SPECT), computed tomography (CT), and magnetic resonance imaging (MRI), to improve diagnostic reliability. For example, functional imaging modalities such as PET and SPECT can provide crucial information related to metabolic function, enabling the identification of functional abnormalities. However, clinicians often face significant challenges in independently interpreting multiple imaging modalities because manually integrating image data from multiple sources can reduce diagnostic efficiency and accuracy.

[0003] To overcome the limitations of single-modality imaging and manual integration of image data from multiple sources, multimodal medical image fusion (MMIF) technology has become a promising area for improvement. However, existing MMIF methods mainly fall into two categories: convolutional neural networks (CNNs) and Transformers. For example... Figure 1 As shown, CNNs possess excellent local receptive properties but struggle to capture long-range dependencies; while Transformers offer global modeling capabilities, their high computational complexity and slow inference speed make them unsuitable for real-time clinical scenarios. Visual state-space models (VMamba) achieve linear-complexity global modeling through selective state updates; however, they lack content awareness and feature alignment capabilities in cross-modal tasks, easily leading to functional hotspot suppression and blurred structural boundaries, hindering accurate diagnosis and treatment. Therefore, superior fusion methods are urgently needed to address these issues. Summary of the Invention

[0004] The present invention aims to provide a medical image fusion method and system based on VMamba that combines cross-modal grouping ACmix convolution and selective clustering, in order to solve the technical problem that existing multimodal medical image fusion methods cannot take into account both local and global features and efficient inference speed, making it difficult for the fused images to support rapid and accurate diagnosis and treatment.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a medical image fusion system based on VMamba that combines cross-modal grouping ACmix convolution and selective clustering, comprising an input module, a preprocessing module, a feature extraction module, a multi-scale fusion module, and an output reconstruction module; The input module includes a functional image submodule, a structural image submodule, and a stitching submodule. The functional image submodule is used to input functional images, the structural image submodule is used to input structural images, and the stitching submodule is used to stitch together the functional images and structural images and output them to the feature extraction module. The functional image is any one of PET, SPECT, CT, and MRI; the structural image is MRI; when the functional image is PET or SPECT, the functional image submodule is used to input the functional image and convert it into YUV space, and extract the brightness channel Y; at this time, the stitching submodule is used to stitch the brightness channel Y and MRI. The feature extraction module includes an ABR component, which is used to extract features from the stitched image output by the input module and send the extracted feature data to the multi-scale pyramid extraction submodule. The ABR component consists of an ACmix unit, batch normalization and nonlinear activation function, and is used to extract local features and contextual features of the stitched image. The multi-scale fusion module includes a pyramid sub-module and a fusion sub-module. The pyramid sub-module includes at least two pyramid structures for fusing the feature data output by the feature extraction module, processing features at different scales into features of different modalities and channels, and outputting the processed data to the fusion sub-module. Each pyramid structure includes a top branch, a middle branch, and a bottom branch. The top branch includes one cascaded ABR component and one MVSS component; the middle branch includes two cascaded ABR components and one MVSS component; and the bottom branch includes three cascaded ABR components and one MVSS component. Before the branches, the pyramid structure also includes a channel segmentation CS module for dividing the input feature map into smaller groups, facilitating independent feature extraction for each group by the branch. The MVSS component includes a multi-layered attention modulation state space module, Vision-Mamba, which introduces a cross-modal attention context a_t into the state update equation h_t=Δ·h_{t-1}+B·x_t in the Mamba structure, making Δ and B functions of a_t: h_t=Δ(a_t)·h_{t-1}+B(a_t)·x_t, thereby realizing content-based dynamic memory. The fusion submodule is used to use a gating mechanism to weightedly combine features from different modalities and channels obtained from different pyramid layers, and outputs the fused data to the output reconstruction module. The output reconstruction module is used to output the fused image; when the functional image is PET or SPECT, the output reconstruction module is used to synthesize the fused image and the original U / V components into a color fused image and output it.

[0006] Preferably, as an improvement, this solution also provides a medical image fusion method based on cross-modal VMamba combining grouped ACmix convolution and selective clustering, which is implemented using the above-mentioned system and includes the following steps: S1. Input preprocessing stage: The input functional image and structural image channels are concatenated and then input into the fusion network; S2, Feature Extraction Stage: Local and contextual features are extracted using the ABR component; S3, Multi-scale fusion stage: Features of different scales are fused through at least two pyramid structures, and the output is then weighted and combined from different modalities and channels using a gating mechanism. S4. Output Reconstruction Stage: Upsampling and convolutional fusion to output fused brightness map. Combined with the original U / V components to form a color fused image .

[0007] Preferably, as an improvement, it also includes S5, the loss function verification stage: using composite unsupervised loss. L whole = μ 1 L RMI + μ 2 L SSIM + μ 3 L Intensity This enables joint optimization of structure, region, and brightness consistency.

[0008] This solution utilizes a lightweight, multi-scale ACmix-VMamba (LMACV) framework specifically designed for efficient multimodal medical image fusion. Its principles and advantages are as follows: 1. Innovation: This solution is the first to introduce cross-modal attention into a state-space model, resulting in a novel multimodal image fusion framework, LMACV. This framework is the first to perform joint optimization of ACmix and VMamba structures in cross-modal medical images, and enhances feature reconstruction efficiency through selective clustering. MSE and PSNR are significantly better than existing methods such as MPCT, FATFasion, and MATR. By fusing convolutional networks and state-space modeling, this network can effectively capture local texture details while preserving long-distance semantic relationships. Furthermore, it effectively achieves attention-guided linear state updates and cross-modal dynamic alignment, improving fusion speed by 37.5% compared to the latest MPCT algorithm.

[0009] 2. Efficiency: This solution effectively maintains linear complexity and improves inference speed by approximately 40% compared to Transformer-based methods.

[0010] 3. High Accuracy: This approach demonstrates LMACV's superior performance and good generalization ability through comprehensive experiments on three representative multimodal datasets (SPECT / MRI, PET / MRI, CT / MRI) and two downstream tasks (lesion segmentation and object detection). Specifically, for example... Figure 2 As shown, the PSNR, MS-SSIM, QMI and other indicators of the fused images obtained by this solution are significantly improved, which improves the fusion quality and texture accuracy of the images, making the boundaries of the fused images clearer and the details richer, and effectively improving the accuracy of diagnosis and treatment.

[0011] 4. Generalization ability: The model parameters of this solution are only about 25K, which can be transferred to various medical imaging tasks such as segmentation and detection.

[0012] 5. Compared to traditional deep learning methods that typically input images into a single, uniform network, making it difficult to collect local information of different sizes and potentially losing important global contextual information, the grouped multi-scale pyramid structure designed in this approach can extract hierarchical features from different receptive fields while maintaining low computational cost. More specifically, this approach extracts hierarchical features from fine to coarse semantic levels by combining grouped ACmix operations with multi-stage VMamba modules, effectively reducing the loss of detail and texture.

[0013] 6. By employing grouped ACmix operations in multi-resolution branches, the inventors' architecture effectively expands the receptive field and encodes hierarchical semantic information while maintaining low parameter complexity. This trade-off is crucial for real-time, resource-constrained clinical applications.

[0014] 7. This solution proposes a novel Multi-Stage Visual State Space (MVSS) module built on the VMamba framework. Through directional scanning and data-dependent memory dynamics, it achieves efficient global modeling with linear complexity. To the inventors' knowledge, this is the first time VMamba has been applied to the field of multimodal medical image fusion. Attached Figure Description

[0015] Figure 1 This is a comparison chart of existing local convolutional neural networks (CNNs), global self-attention (Transformer), and selective scanning (VMamba) used in this embodiment.

[0016] Figure 2The output of the LMACV model in this embodiment of the invention is obtained through multimodal fusion (including the fusion of SPECT, PET, CT and MRI; each row represents a multimodal fusion combination (such as PET / MRI or CT / MRI), and each column represents the original image, reference image and fused image respectively).

[0017] Figure 3 This is an overall architecture diagram of LMACV when the functional image is PET or SPECT in an embodiment of the present invention.

[0018] Figure 4 This is an overall architecture diagram of LMACV when the functional image is CT or MRI in an embodiment of the present invention.

[0019] Figure 5 The architecture of MVSS (Multi-Scale Selection Module) and the design of its internal VSS module in Experimental Example 1 of this invention are shown in ((a) MVSS composed of multiple VSS modules; (b) VSS module with SS2D as its core).

[0020] Figure 6 This refers to the SS2D operations ((a) directional expansion; (b) merging) in Experimental Example 1 of this invention.

[0021] Figure 7 This is a qualitative comparison of the various model fusion results of SPECT / MRI image pairs in Experiment Example 1 of this invention (each row shows a representative set of images).

[0022] Figure 8 This is a qualitative comparison of the fusion results of various models of PET / MRI image pairs in Experiment Example 1 of this invention (each row shows a representative set of images).

[0023] Figure 9 This is a qualitative comparison of the fusion results of various models for CT / MRI image pairs in Experiment Example 1 of this invention (each row shows a representative set of images).

[0024] Figure 10 The two architectural variants used in the ablation experiment in Experiment Example 2 of this invention are ((a) a single-layer multi-scale pyramid structure configuration and (b) a hierarchical structure consisting of three stacked multi-scale pyramid layers).

[0025] Figure 11 This is an evaluation of the effectiveness of the time-series feature extractor for fused SPECT-MRI images at multiple pyramid levels in Experimental Example 2 of this invention.

[0026] Figure 12 This is an evaluation of the effectiveness of the time-series feature extractor for fused PET-MRI images at multiple pyramid levels in Experiment Example 2 of this invention.

[0027] Figure 13 This is the fusion result of SPECT-MRI image pairs with and without the MVSS module in Experiment Example 2 of the present invention.

[0028] Figure 14 This is the fusion result of PET-MRI image pairs with and without the MVSS module in Experiment Example 2 of the present invention.

[0029] Figure 15 The results of the ablation experiment (SPECT-MRI) of the loss function in Experiment Example 2 of this invention are shown.

[0030] Figure 16 The results of the ablation experiment (PET-MRI) of the loss function in Experiment Example 2 of this invention are shown.

[0031] Figure 17 This is the quantitative result of brain tumor segmentation using different modalities in Experiment Example 2 of the present invention.

[0032] Figure 18 This is the first result of target detection using different fusion methods on the MSRS dataset in Experiment Example 2 of this invention.

[0033] Figure 19 This is the second example of target detection results using different fusion methods on the MSRS dataset in Experiment Example 2 of this invention. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to embodiments, but the implementation of the present invention is not limited thereto. Unless otherwise specified, the technical means used in the following embodiments and experimental examples are conventional means well known to those skilled in the art, and the materials and reagents used can all be obtained commercially.

[0035] Example This solution provides a medical image fusion system based on cross-modal VMamba that combines grouped ACmix convolution with selective clustering, referred to as the LMACV model. The overall architecture is as follows: Figure 3 and Figure 4 As shown, the LMACV model employs a multi-scale architecture to process input images: this architecture consists of parallel ABR and MVSS modules, and uses a pyramid structure design with at least two levels. Specifically, it includes an input module, a preprocessing module, a feature extraction module, a multi-scale fusion module, and an output reconstruction module. The input module includes a functional image submodule, a structural image submodule, and a stitching submodule. The functional image submodule is used to input functional images, the structural image submodule is used to input structural images, and the stitching submodule is used to stitch together the functional images and structural images and output them to the feature extraction module. The functional image can be any one of PET, SPECT, CT, or MRI; the structural image is MRI. When the functional image is PET or SPECT, the functional image submodule is used to input the functional image and convert it into YUV space, extracting the brightness channel Y. At this time, the stitching submodule is used to stitch the brightness channel Y and the MRI. The feature extraction module includes an ABR component, which is used to extract features from the stitched image output by the input module and send the extracted feature data to the multi-scale pyramid extraction submodule. The ABR component consists of an ACmix unit, batch normalization and non-linear activation function, and is used to extract local features and contextual features of the stitched image. The multi-scale fusion module includes a pyramid submodule and a fusion submodule. The pyramid submodule includes at least two pyramid structures, used to fuse the feature data output by the feature extraction module, processing features at different scales into features of different modalities and channels, and outputting the processed data to the fusion submodule. Each pyramid structure includes a top branch, a middle branch, and a bottom branch. The top branch includes one ABR component and one MVSS component connected in series; the middle branch includes two ABR components and one MVSS component connected in series; and the bottom branch includes three ABR components and one MVSS component connected in series. The MVSS component includes the attention modulation state space module AC-Mamba, which introduces a cross-modal attention context a_t into the state update equation h_t=Δ·h_{t-1}+B·x_t in the Mamba structure, making Δ and B functions of a_t: h_t=Δ(a_t)·h_{t-1}+B(a_t)·x_t, thereby realizing content-based dynamic memory; The fusion submodule is used to use a gating mechanism to weightedly combine features from different modalities and channels obtained from different pyramid layers, and outputs the fused data to the output reconstruction module. The output reconstruction module is used to output the fused image; when the functional image is PET or SPECT, the output reconstruction module is used to synthesize the fused image with the original U / V components to form a color fused image and output it.

[0036] The operation process of the medical image fusion system in this solution is as follows: First, a given pair of registered multimodal medical images is fused. This includes one functional image (such as a PET or SPECT image with RGB channels, or a single-channel CT or MRI image, denoted as ). I T ∈R H×W×3 ) and one structural image (such as a single-channel MRI image, denoted as ... I MRI ∈R H×W×1This process resolves channel mismatch and color inconsistency issues by converting the functional image from the RGB color space to the YUV color space. The decomposition process yields the luminance component. With chromaticity components , : Equation (1); Then, the luminance component With structural images By splicing along the channel dimension, the input to the fusion network is obtained. : Equation (2); Let the LMACV network be denoted as LMACV(·), and this network takes the input as... , and Generate the luminance component of the fused image : Equation (3).

[0037] Finally, the final RGB fused image is obtained through inverse transformation. : Equation (4).

[0038] This solution also provides a medical image fusion method based on cross-modal VMamba that combines grouped ACmix convolution with selective clustering, including the following steps: S1. Preprocessing Input Stage: The input functional image and structural image channel are stitched together and then input into the fusion network; the functional image can be any one of PET, SPECT, CT, or MRI; when the functional image is PET or SPECT, the brightness channel Y needs to be extracted and stitched together with the structural image channel; when the functional image is CT or MRI, it is directly stitched together with the structural image channel. S2, Feature Extraction Stage: Local and contextual features are extracted using the ABR component; The ABR (ACmix-BatchNorm-ReLU) component consists of ACmix units, batch normalization (BN), and a non-linear activation function (ReLU). The process by which the ABR component extracts local and contextual features is as follows: Given input Feature maps are generated through three 1×1 convolutions. And grouped by channel.

[0039] (1) Local feature extraction: The three feature groups are reshaped and spliced ​​together. Then obtained through a fully connected layer ;Utilizing depthwise separable convolution pairs Processing is performed to obtain local features. ; (2) Contextual feature extraction: extracting feature groups The query (Q), key (K), and value (V) are used as flattened inputs, respectively, and a multi-head self-attention mechanism is applied. Contextual features are captured by concatenating the outputs of all attention heads. ; Local features With context features Adaptive fusion is achieved using learnable parameters α and β: Equation (5); This hybrid design strikes a balance between feature representation capability and computational efficiency. The output of the ABR module is: Equation (6); in, This indicates the output characteristics after processing by the ABR module. This indicates the ACmix operation. Therefore, the ABR module can learn features from the source image more efficiently while effectively suppressing redundant noise.

[0040] S3, Multi-scale fusion stage: Features at different scales are fused through at least two pyramid structures, and the output is then weighted and combined using a gating mechanism after the features from different modalities and channels are weighted. Specifically, this stage includes the following: To balance preserving spatial resolution and efficiently extracting hierarchical features at different semantic levels, this scheme employs a three-branch multi-scale architecture. However, directly adding multiple parallel branches would significantly increase computational costs. To address this issue, the inventors first utilize a channel segmentation (CS) module to divide the input feature map into smaller groups, and then extract features independently from each group, ensuring representational power while maintaining computational efficiency. Equation (7).

[0041] Specifically, the three-branch multi-scale architecture includes the following: the top branch contains one ABR component and one subsequent MVSS component; the middle branch contains two ABR components and one MVSS component; and the bottom branch contains three ABR components and one MVSS component. These branches generate corresponding feature representations. From the top to the bottom branch, the number of ABR components gradually increases, enabling the network to progressively capture more refined and semantically richer features. Finally, features at different scales are aggregated through an additive mechanism to enhance the understanding of context and the preservation of local details. The formal definition of the multi-scale feature representation is as follows: Equation (8); Equation (9); Equation (10).

[0042] in, , , These represent the outputs of the three parallel branches. This represents the multi-stage visual state space model (SSM) module used in the network architecture.

[0043] Furthermore, to construct the second level of the pyramid, spatial downsampling of the input features is performed according to the principles of feature pyramid networks. This process can be represented as: Equation (11); in, This indicates the downsampling (DS) process. This indicates the channel splitting (CS) process.

[0044] Equation (12); Equation (13); Equation (14); in, , , This represents the output of the three branches. Then, the outputs of these three branches are concatenated with the output of the second level to integrate the features obtained from each path. This process can be quantitatively represented as: Equation (15); Here, "+" represents the addition operation. Indicates an upsampling operation. Conv This represents the convolution operation. Concat This indicates a splicing operation. Tanh This represents the hyperbolic tangent activation function.

[0045] The deep feature extraction based on MVSS includes the following: like Figure 5 As shown, the VSS block, based on the improved VMamba architecture, is the basic unit of LMACV in this scheme. First, the input is normalized (LN) and then divided into two parallel branches: the first branch performs a linear transformation followed by a non-linear activation function; the second branch processes the input through a linear layer, depthwise separable convolution, and activation function, then inputs the result into a 2D selective scan (SS2D) module to enhance feature extraction. Afterward, the extracted features are normalized again using LN and fused with the output of the first branch through element-wise multiplication, effectively integrating the two paths. Finally, a linear mixing layer processes the fused features and establishes a residual connection with the original input to obtain the final output of the VSS block. In SS2D, the default is to use... SiLU Activation function.

[0046] The SS2D module consists of three key components: a scan expansion unit, an S6 block, and a scan merging operation. Scan expansion refers to the process of unfolding the input image into a sequence along the four diagonal directions, such as... Figure 6 As shown, the S6 block processes the directional sequences and extracts features from multiple perspectives, improving the model's ability to capture diverse representations. Subsequently, a scan-merge operation aggregates and sums the sequences in the four directions to reconstruct an output image that preserves the original spatial dimensions.

[0047] Based on the S4 block and Mamba, the S6 block employs a selective method to adaptively adjust the parameters of the state-space model. Therefore, in practical applications, the network can retain relevant features and systematically eliminate redundant data.

[0048] S4. Output Reconstruction Stage: Upsampling and convolutional fusion to output fused brightness map. Combined with the original U / V components to form a color fused image .

[0049] This also includes S5, the loss function evaluation stage: the composite loss function specifically includes Region Mutual Information (RMI) loss, Structural Similarity Index (SSIM) loss, and intensity loss. The total loss is defined as follows: Equation (16); In the formula, the weights µ 1. µ 2 and µ 3 is used for adjustment L RMI , L SSIM and L Intensity The trade-off between them.

[0050] first, L RMI The loss is due to the increased integrity of information retention in the fusion results. L RMI The calculation formula is as follows: Equation (17); Secondly L SSIM The loss is used to ensure that the fusion result retains the structural authenticity of the source images. It is defined as: Equation (18); SSIM(·) is used to measure the structural similarity between two images, and the weighting factor η affects the emphasis on structure preservation. To balance the preservation of source image structure, the inventors chose η=0.5, so that the source images have the same impact on the fusion result.

[0051] Finally, the intensity loss is used to ensure that the fused image retains appropriate intensity features, and it is defined as: Equation (19); in, I f Indicates a fused image. I 1 and I 2 represents the input source image. H and W These represent the height and width of the image, respectively. `max(·)` represents the operation of taking the maximum value element-wise from both images. Euclid ( The norm. This formula ensures that the fused image retains the most salient intensity values ​​from the input modalities.

[0052] This evaluation method uses a pseudo-reference for full-reference measurement of the luminance channel, which corresponds to the intensity term in equation (19). In the calculation of mean square error / peak signal-to-noise ratio (MSE / PSNR), the following is defined: This is the element-wise maximum value of the two source brightness images. This choice reflects the clinical goal: to preserve the most significant intensity in either modality while avoiding bias towards a single source image.

[0053] While the intensity term prompts the fused brightness to follow the significant values ​​of the source image, it does not enforce a strict per-pixel maximum value constraint. Its effect is modulated through a composite objective function: (Default settings) μ 3=1), where L SSIM Used to punish structural distortion L RMI Used to ensure regional consistency. Combining batch normalization with the Tanh output layer, "maximum selection" is a soft preference designed to preserve diagnostically significant salient regions (such as hotspots in PET / SPECT and high-density structures in CT) rather than hard rules that lead to image saturation.

[0054] Experimental Example 1: Comparison of the fusion method in this scheme with existing technologies Dataset (Processing Object): This approach uses the publicly available Harvard Medical Image Dataset and references previous research to fuse and evaluate three image combinations: SPECT / MRI (539 pairs), PET / MRI (329 pairs), and CT / MRI (166 pairs), all with a resolution of 256×256. Pre-registered modal image pairs are provided in this dataset.

[0055] Data Preprocessing: In this scheme, 20 pairs of test images are used for each modality (CT / MRI, PET / MRI, and SPECT / MRI). The remaining image pairs are divided into training and validation sets at a 9:1 ratio. Data augmentation is performed only on the training set (overlapping cropping using a 120×120 patch size and a stride of 20; rotation; horizontal / vertical flipping) to avoid data leakage. For PET and SPECT images, RGB images are converted to the YUV color space; the Y (luminance) channel is fused with the MRI image while retaining the U / V components for color reconstruction. During the validation and testing phases, full-size images are input into the model without cropping. Before training and evaluation, all source and fused images are normalized to the [0,1] floating-point range to ensure numerical stability. Quantitative metrics such as Peak Signal-to-Noise Ratio (PSNR) and Mean Squared Error (MSE) are calculated on the normalized images.

[0056] The training strategy is as follows: All tests were conducted using the PyTorch framework and run on an NVIDIA GPU workstation equipped with CUDA. The Adam optimizer was used (learning rate = 10). -3 Batch size = 16, weight decay = 10 -4 The training process was run for 100 epochs, with an early stopping strategy based on validation loss. Unless otherwise specified, all layers used 3×3 kernels, with the output layer using 1×1 kernels to reduce dimensionality. Each experiment was repeated three times using a different random seed, and the results were averaged on the test set.

[0057] To verify the effectiveness of the proposed LMACV framework, the inventors benchmarked it against eight state-of-the-art multimodal medical image fusion algorithms. These methods fall into three categories: (a) multi-scale transform-based methods—Non-subsampled shear wave transform (NSST) and Local Direction Pattern (LDP); (b) deep learning-based networks—DDcGAN (Generative Adversarial Network based on dense connections), DFENet (Dual-branch Feature Enhancement Network), and PPTFusion; and (c) Transformer-based methods—FATFusion, MACTFusion, and MPCT.

[0058] Existing benchmark models (such as MATR, MACTFusion, MPCT, PPTFusion, DFENet, and DDcGAN) were evaluated using their official code and pre-trained weights, or, where authors provided code and recommended settings, these were used to retrain the models. The retrained algorithms employed the same data splitting, data augmentation, preprocessing, and training settings as the main experiments. To ensure the comparability and validity of all benchmark results, all models were tested on the same hardware (NVIDIA 3090 GPU and Intel i9-9900K CPU) with an input size of 256×256. Each experiment was repeated three times using a different seed, and the results were averaged.

[0059] To objectively and comprehensively evaluate the fusion methods proposed by the inventors, each method was evaluated using qualitative visual inspection and seven quantitative metrics (entropy (EN), mutual information (MI), mean squared error (MSE), peak signal-to-noise ratio (PSNR), multi-scale structural similarity index (MS-SSIM), quantitative mutual information (QMI), and visual information fidelity (VIF)) to comprehensively measure the fusion performance. All metrics are based on the fused image. I Y Calculations were performed on the luminance channel to ensure consistency in cross-modal assessment. Higher values ​​for entropy (EN), mutual information (MI), peak signal-to-noise ratio (PSNR), multi-scale structural similarity index (MS-SSIM), quantitative mutual information (QMI), and visual information fidelity (VIF) indicate better performance; lower values ​​for mean squared error (MSE) indicate smaller reconstruction errors and higher accuracy. Red boxes mark clinically critical areas such as lesion margins and functional hotspots for detailed examination and comparison between different methods. These comparisons reveal the advantages and disadvantages of each strategy, further highlighting the strengths of the LMACV framework presented in this protocol.

[0060] (1) Results and analysis of SPECT and MRI image fusion Figure 7 Three pairs of representative SPECT-MRI images used for evaluation are presented. Each pair includes the source SPECT image, the source MRI image, and the fusion results of eight contrast methods (NSST, LDP, DDcGAN, DFENet, PPTFusion, FATFusion, MACTFusion, MPCT) and the LMACV of this protocol.

[0061] The eight comparison methods are detailed in the following literature: 1. SST:W. Tan, P. Tiwari, H. M. Pandey, C. Moreira, A. K. Jaiswal,Multimodal medical image fusion algorithm in the era of big data, NeuralComputing and Applications (2020) 1–21. 2. LDP:Liu, S. Liu, Z. Wang, A general framework for image fusionbased on multi-scale transform and sparse representation, Information Fusion24 (2015) 147–164. 3. DDCGAN:J. Ma, H. Xu, J. Jiang, X. Mei, X.-P. Zhang, Ddcgan: Adual-discriminator conditional generative adversarial network for multi-resolution image fusion, IEEE Transactions on Image Processing 29 (2020)4980–4995 4. DFENet: W. Li, Y. Zhang, G. Wang, Y. Huang, R. Li, Dfenet: A dual-branch feature enhanced network integrating transformers and convolutionalfeature learning for multimodal medical image fusion,Biomed. Signal Process.Control. 80 (Part) (2023) 104402. 5. PPTFusion:Y. Fu, T. Xu, X. Wu, J. Kittler, Ppt fusion: Pyramidpatch transformerfor a case study in image fusion, arXiv preprint arXiv:2107.13967 (2021) 6. FATFusion:W. Tang, F. He, Fatfusion: A functional–anatomicaltransformer for medical image fusion,Information Processing Management 61 (4)(2024) 103687 –103699. 7. MACTFusion:X. Xie, X. Zhang, X. Tang, J. Zhao, D. Xiong, L.Ouyang, B. Yang, H. Zhou, B. W.-K. Ling, K. L. Teo, Mactfusion: Lightweightcross transformer for adaptive multimodal medical image fusion, IEEE Journalof Biomedical and Health Informatics (2024). 8. MPCT:Y. Xu, Z. Wang, S. Wu, X. Zhan, Mpct: A medical image fusionmethod based on multi-scale pyramid convolution and transformer, BiomedicalSignal Processing and Control 101 (2025) 107129 – 107135. Compared to traditional fusion methods, transform-based methods such as NSST and LDP often oversmooth low-intensity functional regions in SPECT and blur fine anatomical details in MRI. DDcGAN, based on generative adversarial networks, introduces structural artifacts and has limited ability to preserve metabolic signals. Autoencoder-based methods such as PPTFusion and DFENet typically overemphasize structural details in MRI while suppressing functional features in SPECT, leading to modal imbalance. Transformer-based methods (FATFusion, MACTFusion, MPCT) perform better in preserving local structure and fine texture.

[0062] In comparison, the LMACV scheme consistently outperformed all benchmark methods in both qualitative and quantitative assessments. It effectively preserves subtle functional signals in SPECT while maintaining clear structural boundaries and soft tissue texture in MRI. The images generated by LMACV exhibit balanced brightness, natural contrast, and higher visual clarity, facilitating accurate clinical interpretation. Table 1 shows that LMACV outperforms other methods in key metrics such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), and multi-scale structural similarity index (MS-SSIM), further demonstrating its effectiveness. LMACV achieves a PSNR of 69.912 dB and a mean squared error of 0.008, highlighting its superior signal reconstruction fidelity. Furthermore, it consistently leads in structural similarity (MS-SSIM=0.689), mutual information (QMI=3.155), and visual fidelity (VIF=0.601), indicating a good balance between preserving structural and perceptual features. These results demonstrate that the proposed method can effectively preserve structural features and functional information in multimodal medical image fusion (MMIF).

[0063] Table 1. Quantitative comparison of nine fusion methods on the SPECT / MRI task (↑ indicates higher value is better; ↓ indicates lower value is better; results are averages).

[0064] (2) Results and analysis of PET and MRI image fusion Figure 8Three representative PET-MRI fusion cases are presented, with two regions marked by red boxes in each case magnified for detailed analysis. For each case, the inventors compared the results of eight contrast methods (NSST, LDP, DDcGAN, PPTFusion, DFENet, FATFusion, MACTFusion, MPCT) with their proposed LMACV scheme. The regions marked by red boxes are magnified to highlight local structural features and metabolic details. As shown in the figure, traditional transform-based methods such as NSST and LDP often oversmooth functional signals in PET and fail to preserve fine anatomical structures in MRI. DDcGAN, based on generative adversarial networks, introduces significant artifacts and has poor preservation of metabolic information. Autoencoder-based methods such as PPTFusion and DFENet overemphasize PET or MRI content, leading to loss of detail and inconsistencies in intensity. Transformer-based methods (FATFusion, MACTFusion, MPCT) perform better in preserving global anatomical texture, but these models still suffer from local contrast imbalances and residual background artifacts.

[0065] In contrast, the LMACV approach consistently generates high-quality fusion results, effectively preserving metabolic activity information from PET while maintaining clear tissue boundaries and detailed textures from MRI. As shown in Table 2, the average quantitative results from 20 PET-MRI cases support the conclusions drawn from visual observations. LMACV achieved the highest scores in mutual information (4.656), peak signal-to-noise ratio (61.462 dB), and quantitative mutual information (4.695), and also ranked among the top three in entropy, mean squared error, multi-scale structural similarity index, and visual information fidelity. These results demonstrate that LMACV can compactly fuse structural and functional data, making it particularly suitable for PET-MRI fusion tasks.

[0066] Table 2. Quantitative comparison of nine fusion methods on PET / MRI tasks (↑ indicates higher values ​​are better; ↓ indicates lower values ​​are better; the table shows the mean).

[0067] (3) Results and analysis of CT and MRI image fusion Figure 9Qualitative results from three representative CT-MRI image pairs are presented. Each row includes the source CT image, the source MRI image, and the fusion results from NSST, LDP, DDcGAN, PPTFusion, DFENet, FATFusion, MACTFusion, MPCT, and the LMACV scheme presented here. The areas marked with red boxes are magnified to facilitate close observation of structural details. Ideally, the fusion result should simultaneously preserve the high-density anatomical structures from CT and the rich soft tissue texture from MRI. Figure 9 As shown, traditional methods such as NSST and LDP often over-smooth bone structures on CT and blur fine tissue boundaries on MRI. DDcGAN, based on generative adversarial networks, produces significant artifacts and fails to maintain structural consistency. Autoencoder-based methods such as PPTFusion and DFENet can preserve some structural features, but generally suffer from deficiencies in intensity fidelity and spatial resolution. Transformer-based models (FATFusion, MACTFusion, MPCT) improve detail preservation, but still exhibit contrast imbalance and slight information redundancy in low-contrast regions.

[0068] In comparison, the LMACV method in this scheme produces visually superior fusion results, preserving the dense bone structure in CT and the detailed soft tissue contrast in MRI. Magnified areas clearly show that LMACV more effectively preserves subtle low-intensity features in CT and fine textures in MRI. The data in Table 3 further support these observations, indicating that LMACV outperforms other methods in key metrics such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), and multi-scale structural similarity index (MS-SSIM). LMACV performs best in mean squared error (0.072), PSNR (61.784 dB), and multi-scale structural similarity index (0.701), indicating strong structural similarity and low reconstruction error. Furthermore, it also performs well in other metrics such as quantitative mutual information (2.542, second highest) and visual information fidelity (0.457). Therefore, LMACV not only preserves important information from each modality but also improves the accuracy of the fusion results.

[0069] Table 3. Quantitative comparison of ten fusion methods on CT / MRI tasks (↑ indicates higher value is better; ↓ indicates lower value is better; results are averages).

[0070] Note: MSAFusion: He R, Xu Y, Zheng Y, et al. Multiscale Self-AttentionConvolution and Adaptive Fusion for Enhanced Multimodal Medical Image Fusion[J]. Expert Systems with Applications, 2025: 129967. Analysis of the quantitative results (Tables 1-3) shows that LMACV did not achieve the highest values ​​across all metrics. The metrics where LMACV performed exceptionally well (such as peak signal-to-noise ratio, mean squared error, and multi-scale structural similarity index) primarily assess pixel-level fidelity and structural preservation. The superior performance on these metrics highlights LMACV's ability to preserve anatomical integrity and intensity realism—crucial for diagnostic interpretation. In contrast, metrics such as entropy (EN) and sometimes visual information fidelity (VIF) reflect global information richness or perceptual contrast. Some generative adversarial network-based methods or texture enhancement methods may achieve slightly higher values ​​on these metrics by artificially enhancing contrast or texture, but sometimes at the expense of true structural accuracy. Instead, LMACV prioritizes faithful representation and balanced performance across metrics, avoiding perceptual exaggeration that could distort anatomical realism. This trade-off results in a clinically reliable fusion output, where diagnostic accuracy takes precedence over visual enhancement.

[0071] Furthermore, no whitening of image saturation or highlight areas was observed in any of the cases presented in this scheme; the bright areas were always confined to specific locations and were anatomically plausible, indicating that the intensity term was only a soft bias rather than a strict per-pixel maximum value constraint.

[0072] Experimental Example 2: Ablation Analysis of Key Components and Their Clinical Relevance (1) Study on the ablation of pyramid structure layers The inventors studied the impact of varying layer numbers on performance during feature extraction, aiming to determine the optimal number of layers for multi-scale pyramid networks. Figure 10 (a) and (b) illustrate single-level multi-scale pyramid networks and three-level stacked multi-scale pyramid networks, respectively. Specifically, three pyramid depths—1, 2, and 3 consecutive ABR+MVSS stacks—are compared to determine the optimal feature extractor for LMACV. Figure 11 and Figure 12Qualitative fusion results of these models on SPECT / MRI and PET / MRI image pairs are presented separately. The single-layer model can only capture coarse contextual information, smoothing small lesions and blurring brain sulcus boundaries; while the three-layer model can recover more texture, it amplifies high-frequency noise and introduces artifacts around blood vessels and cortical folds. Visually, the two-level structure achieves an optimal balance, accurately depicting CT bone edges, MRI tissue boundaries, and SPECT / PET hotspots, while avoiding over-smoothing or noise amplification.

[0073] Table 4 provides quantitative validation of these observations. The results show that on the SPECT / MRI dataset, the two-level model achieves the highest values ​​in entropy (4.782), mutual information (3.158), peak signal-to-noise ratio (69.912 dB), multi-scale structural similarity index (0.687), quantitative mutual information (3.168), and visual information fidelity (0.582), while having the lowest mean squared error (0.0066). Similarly, on the PET / MRI dataset, the two-level model performs best in entropy (5.772), mutual information (4.663), peak signal-to-noise ratio (61.588 dB), multi-scale structural similarity index (0.677), quantitative mutual information (4.772), and visual information fidelity (0.573), while having the lowest mean squared error (0.0403). Both single-layer and three-layer models have limitations: the single-layer model over-smooths subtle features, while the three-layer model introduces redundant textures, reducing perceptual accuracy.

[0074] Table 4 Performance Comparison of Different Pyramid Levels

[0075] These results demonstrate that the two-level pyramid extractor possesses sufficient hierarchical depth to simultaneously capture global context and fine local features, avoiding the drawbacks of insufficient or excessive parameters. The two-level design achieves a balance, maximizing information retention and visual clarity—crucial for clinical applications, as accurate depiction of anatomical structures and metabolic activities directly impacts diagnostic and treatment decisions.

[0076] (2) Ablation experiment of MVSS module To evaluate the role of the MVSS module (which integrates different VMamba networks to enhance long-range semantic modeling), the inventors conducted ablation experiments by replacing the MVSS module with a Transformer in the network.

[0077] Table 5 shows the quantitative evaluation metrics (entropy, mutual information, mean squared error, peak signal-to-noise ratio, multiscale structural similarity index, quantitative mutual information, and visual information fidelity) for the PET / MRI and SPECT / MRI datasets. Figure 13 and 13A visual comparison of the fusion results with and without the MVSS module is provided, among which... Figure 13 The results of the SPECT-MRI image pair are shown. Figure 14 Results for PET-MRI image pairs are presented. Removing the MVSS module significantly degraded the performance of all evaluation metrics. For example, on the SPECT / MRI dataset, the multiscale structural similarity index decreased from 0.691 to 0.369, and visual information fidelity decreased from 0.583 to 0.435; on the PET / MRI dataset, quantitative mutual information decreased from 4.772 to 4.127, and visual information fidelity decreased from 0.573 to 0.558. The peak signal-to-noise ratio (PSNR) without the MVSS module was 60.144, lower than that with the MVSS module, consistent with the improvements reported below. These results indicate that without the MVSS module, the structural boundary sharpness of the fused images decreases, more artifacts are introduced, and structural consistency and visual fidelity are significantly reduced.

[0078] Table 5 Ablation Study of MVSS Module

[0079] Figure 13 This study presents the results of an ablation study using the MVSS module in SPECT / MRI fusion of ischemic areas. In the bottom red area, the fusion output without the MVSS module shows blurred ischemic boundaries, inconsistent color mapping, reduced lesion-background contrast, and loss of subtle surrounding structures—reflecting the mismatch between SPECT functional deficiencies and MRI anatomy. With the MVSS module, ischemic lesions are more clearly defined: clearer boundaries, higher lesion-background contrast, and fewer fusion artifacts. By enhancing multi-scale semantic consistency, the MVSS module aligns SPECT functional deficiencies with MRI structural information, thereby mitigating the edge blurring and contrast loss observed in ablation experiments. These effects directly improve the clinical interpretability of ischemic lesions, particularly in accurate localization and boundary delineation.

[0080] Figure 14 The results of PET / MRI fusion with and without the MVSS module are presented. The inclusion of the MVSS module significantly improved boundary sharpness and structural continuity, especially in the cortical and subcortical regions marked with red boxes. In the fused images generated by LMACV (with the MVSS module), anatomical contours were well preserved, and functional signals were well aligned with MRI structures, enabling clinicians to more accurately delineate lesion locations and assess metabolic activity. In contrast, results without the MVSS module showed blurred edges, reduced edge contrast, and partial loss of cortical details, which could potentially mask subtle pathological changes. These observations suggest that the MVSS module is crucial for maintaining multi-scale structural fidelity and improving the clinical interpretability of PET / MRI fusion images.

[0081] Therefore, these observations demonstrate the importance of the MVSS module in ensuring accurate feature alignment across different modalities and preserving key semantic details. The presence of the MVSS module not only improves the visual quality of the fused images by enhancing edge sharpness and contrast, but also significantly improves quantitative performance metrics, confirming its crucial role in achieving high-quality fusion of multimodal image structure and function.

[0082] (3) Investigation of the impact of loss function and parameters To evaluate the impact of each component in the composite loss function, the inventors removed the structural similarity (SSIM) loss, regional mutual information (RMI) loss, and intensity loss respectively, and conducted quantitative and qualitative ablation experiments. Figure 15 and Figure 16 The fusion output results are shown in Table 6, which presents the quantitative results of the SPECT / MRI and PET / MRI datasets.

[0083] Table 6 Component Removal Analysis of Loss Function

[0084] Removing the structural similarity loss reduced the multi-scale structural similarity index and visual information fidelity (e.g., on the SPECT / MRI dataset, the multi-scale structural similarity index decreased from 0.699 to 0.489, and visual information fidelity decreased from 0.588 to 0.442), highlighting its importance in preserving structural similarity. Removing the region mutual information component significantly reduced mutual information and quantitative mutual information, indicating a decrease in complementary information. Removing the intensity loss reduced peak signal-to-noise ratio (PSNR) and entropy (e.g., on the SPECT / MRI dataset, PNR decreased from 69.992 to 66.914, and entropy decreased from 4.782 to 3.256), indicating weakened detail preservation and brightness consistency. Importantly, removing the intensity loss term did not improve structure-related metrics (MS-SSIM, QMI, VIF). This suggests that the selected intensity loss weight does not lead to overexposure or artificial saturation; instead, removing the weight makes the fused image darker and less information-rich, without any gain at the perceptual or structural level. These findings are consistent across SPECT / MRI and PET / MRI tasks (Table 6). Figure 15 As further shown, the absence of SSIM loss blurs anatomical boundaries, while removing RMI loss or intensity loss leads to color distortion or decreased contrast. Figure 16 The results show that only by combining these three types of loss can optimal structural clarity and functional information identification be achieved, and no saturation or highlight artifacts were observed in any of the reported results.

[0085] In the aforementioned loss function, Weighting coefficient μ 1. μ 2. μ 3 is treated as an independent parameter, controlling the trade-off between region information preservation, structural similarity, and intensity consistency. To systematically evaluate the contribution of each item, the inventors adopted a control variable strategy: for each parameter under study, the other two parameters are fixed at default values. This approach avoids redundancy and allows performance variations to be clearly attributed to each weight. Table 7 summarizes representative results on the PET / MRI validation set, demonstrating the impact of each loss weight on standard fusion metrics (EN, MI, MSE, PSNR, MS-SSIM, QMI, VIF).

[0086] Table 7. The effect of different loss weight parameters on PET / MRI fusion (↑ indicates higher values ​​are better; ↓ indicates lower values ​​are better).

[0087] The results showed that increasing μ 1 can improve regional mutual information (higher MI and VIF), but excessively large values ​​may compromise global structural consistency; adjustment μ μ2 can enhance structural similarity (MS-SSIM), but excessively large μ2 may over-smooth details; increasing μ2 can further enhance structural similarity. μ 3 helps maintain intensity alignment, but overemphasis can reduce structural diversity and visual naturalness. In all test configurations, ( μ 1: μ 2: μ 3) = (2.5:1:1) achieves the best overall balance across all evaluation indicators.

[0088] (4) Computational complexity analysis To evaluate the effectiveness of LMACV compared to the lightweight multi-scale grouping ACmix and VMamba architectures of our proposed scheme, the inventors compared its parameter count and inference speed with five representative deep learning fusion networks (PPTFusion, DFENet, FATFausion, MACTFusion, and MPCT). Complexity calculations were based on 256×256 input images, a batch size of 12, and execution on an RTX 3090 GPU (CUDA 12.2). Table 8 shows that PPTFusion and DFENet require over 78 million and 22 million parameters, respectively, while FATFausion and MACTFusion use 6.2 million and 620,000 parameters, respectively. The MPCT method uses 76,000 parameters, while LMACV requires only 23,000 parameters, reducing the model size by 60% compared to MPCT while maintaining similar fusion accuracy.

[0089] Table 8 Comparison of Model Parameters and Inference Time

[0090] LMACV achieves the fastest inference speed (0.05 seconds per image on an NVIDIA RTX 3090), outperforming MPCT (0.11 seconds) and MACTFusion (1.25 seconds). On a CPU (Intel Core i9-9900K), LMACV processes 256×256 images in 0.6 seconds, demonstrating its practical inference speed even without GPU acceleration in offline or semi-automated clinical scenarios. In many clinical deployments, model inference is performed on central servers or hospital workstations with higher computing resources, with only the fusion results displayed at the clinical front end. This architecture enables near real-time interaction in practical applications, unrestricted by the computing power of individual user devices. Our contributions include: grouped ACmix operations for efficient extraction of diverse multi-scale features; and a VMamba-based MVSS module that replaces the quadratic complexity self-attention mechanism with linear complexity state-space modeling, resulting in significant improvements. These design choices collectively lay a solid foundation for potential real-time optimizations in future deployments and clinical applications, while retaining the generalization capabilities and performance demonstrated above.

[0091] (5) Application in medical image segmentation We evaluated our proposed LMACV method on the BraTS_2019 dataset 8, specifically for the downstream task of brain tumor segmentation. This dataset is a widely used public benchmark that includes labeled volumetric MRI scan images with three tumor sub-regions annotated: enhanced tumor (ET), tumor core (TC), and whole tumor (WT). The dataset contains 285 training samples and 49 test samples, both with T2-weighted and FLAIR modalities.

[0092] All segmentation experiments were implemented using the official PyTorch implementation of U-Net++. We used the same training / test split as the original BraTS_2019 benchmark, employed the Adam optimizer (learning rate 0.001, weight decay 1e-4), a batch size of 8, an input patch size of 160×160, and trained each model for 500 epochs with an early stopping strategy. Data augmentation methods included random rotation, flipping, and intensity shifting.

[0093] For each fusion method, the same network architecture and training procedure were used; all experiments were repeated three times, and no additional hyperparameter tuning was performed on LMACV. All benchmark and fusion methods were evaluated under the same settings to ensure fairness and comparability.

[0094] In our experiment, the proposed LMACV method was used to fuse MR-FLAIR and MR-T2 images and compared with the PPTFusion benchmark method. The experiment used U-Net++

[74] network as the segmentation backbone and constructed four segmentation processes: U-Net++ was directly applied to FLAIR images, T2 images, and fused images obtained by PPTFusion and LMACV methods, respectively.

[0095] Performance evaluation employed Dice coefficient and sensitivity, with quantitative analysis performed in whole tumor (WT), tumor core (TC), and enhanced tumor (ET) regions. The Dice coefficient was used to calculate the spatial overlap between the model's predicted and actual segmentation results, while sensitivity measured the model's ability to correctly label positive tumor regions. Related visualization results can be found... Figure 17 Detailed numerical results are shown in Table 9. Among them, Dice coefficient is used to measure the overlap of segmentation results, and sensitivity is used to measure the model's ability to identify positive regions; ET represents enhanced tumor, TC represents tumor core, and WT represents whole tumor.

[0096] Table 9. Segmentation results using different fusion inputs on the BraTS_2019 dataset.

[0097] The combination of LMACV and U-Net++ achieved Dice scores of 50.29%, 86.05%, and 78.58% for enhanced tumor (ET), tumor core (TC), and whole tumor (WT), respectively, which are 1.61, 0.20, and 0.71 percentage points higher than the PPTFusion+U-Net++ framework. The corresponding sensitivity values ​​increased to 49.81%, 83.16%, and 83.61%, with improvements of 0.98, 1.68, and 5.38 percentage points, respectively. Figure 17 As shown, the image segmentation map fused by the LMACV model is more helpful in improving lesion detection and tumor core identification. More importantly, the model can significantly reduce false positive results, especially in areas where it is difficult to distinguish between tissues. This not only facilitates lesion resection for doctors, but also demonstrates the model's ability to maintain the integrity of anatomical structures and enhance intermodal contrast during fusion.

[0098] (6) Application in target detection To verify the cross-domain scalability of the proposed LMACV framework beyond medical image fusion, we conducted object detection experiments on the MSRS dataset using infrared-visible (IR-VIS) image fusion. The MSRS benchmark dataset contains 1444 pairs of registered infrared and visible image pairs, all acquired in complex real-world scenes (including low-light and occluded environments), making it ideal for evaluating fusion performance in practical detection scenarios. We processed each pair of infrared-visible images using LMACV and several state-of-the-art fusion methods (DDcGAN, PPTFusion, DFENet, FATFusion, MACTFusion, MPCT), and input the resulting fused images into a pre-trained YOLOv8s object detector to evaluate the impact of fusion quality on downstream object detection (especially pedestrian and vehicle detection).

[0099] All object detection experiments were implemented using PyTorch with YOLOv8s, following the official split of the MSRS dataset (70% for training, 30% for testing). We used the Adam optimizer (learning rate 0.001), a batch size of 16, and an input resolution of 640×640 pixels. Data augmentations included random scaling, cropping, flipping, and color jittering. The model was trained for 800 epochs with an early stopping strategy based on the validation loss. To ensure statistical robustness, each fusion method was evaluated through three independent runs, and we report the mean and standard deviation of key metrics (mAP, precision, recall). No additional hyperparameter tuning was performed on the fused output of LMACV; all baseline and fusion methods were evaluated under the same experimental settings to ensure fairness and comparability. Complete object detection scripts and configurations for reproducing the experiments are available in our GitHub repository.

[0100] Each pair of infrared-visible images is processed using LMACV and other innovative fusion techniques (including DDcGAN, PPTFusion, DFENet, FATFusion, MACTFusion, and MPCT). The resulting fused images are then fed into a pre-trained YOLOv8s object detector to evaluate the impact of fusion quality on downstream object detection, particularly pedestrian and vehicle detection. Figure 18 and Figure 19 The detection results were visualized and compared in three representative scenarios. Although most baseline methods can locate known targets well, they often miss small or partially occluded objects, especially in cluttered or dimly lit environments.

[0101] In contrast, LMACV continuously optimizes texture and thermal features, achieving more accurate bounding box localization and fewer false negatives. For example, in cluttered backgrounds, LMACV avoids oversegmentation and false positives common in FATFasion and MACTFFasion; in low-light scenes, it preserves pedestrian contours more clearly than PPTFFasion or DDcGAN.

[0102] Table 10 presents the quantitative performance of four standard metrics: precision (PR), recall (RE), MAP@0.5, and MAP@0.5:0.95. LMACV+YOLOv8s exhibits the best overall performance, with a high MAP@0.5:0.95 of 0.721 and a maximum MAP@0.5 of 0.958. Simultaneously, its precision (0.928) is competitive, and its recall (0.901) is the highest, indicating that LMACV achieves a good balance between accuracy and completeness in detection. These results outperform all other fusion methods, such as MPCT (MAP@0.5:0.95=0.699) and DFENet (recall=0.899).

[0103] Table 10 Detection performance of different methods in infrared-visible (IR-VIS) fusion (including precision (PR), recall (RE), and mean precision (MAP)).

[0104] These experimental results demonstrate that LMACV's design, which combines fine-scale spatial details with global semantic modeling, is applicable not only to medical applications but also to other computer vision tasks. Because it can generate high-fidelity, information-rich fused representations, it holds promise as a powerful tool for real-time perception in safety-critical scenarios such as surveillance, autonomous navigation, and intelligent transportation.

[0105] In summary, this paper proposes a lightweight and efficient multimodal fusion framework, LMACV, which combines grouping ACmix operations with a VMamba-based visual state space module. It can be executed on CPU / GPU devices, and the workflow can be implemented using deep learning frameworks such as PyTorch. LMACV employs a dual-branch, multi-scale pyramid structure to jointly perform local detail extraction and global semantic modeling, aiming to improve clinical applicability and computational efficiency. This algorithm significantly reduces the number of parameters while improving fusion results across different imaging modalities. Experimental results on three representative datasets demonstrate that LMACV successfully preserves complementary functional and anatomical information. Notably, LMACV's performance generalization ability in two tasks (brain tumor segmentation and object detection) is further confirmed: without retraining, LMACV continuously improves the performance of YOLOv8s (object detection model) and U-Net++ (segmentation model), highlighting its applicability in both clinical and non-clinical scenarios. This model is lightweight, has fast inference speed, and is highly adaptable, making it ideal for seamless integration into clinical workflows.

[0106] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A medical image fusion system based on cross-modal VMamba combining grouped ACmix convolution and selective clustering, characterized in that: It includes an input module, a preprocessing module, a feature extraction module, a multi-scale fusion module, and an output reconstruction module; The input module includes a functional image submodule, a structural image submodule, and a stitching submodule. The functional image submodule is used to input functional images, the structural image submodule is used to input structural images, and the stitching submodule is used to stitch together the functional images and structural images and output them to the feature extraction module. The feature extraction module includes an ABR component, which is used to extract features from the stitched image output by the input module and send the extracted feature data to the multi-scale pyramid extraction submodule. The ABR component consists of an ACmix unit, batch normalization and nonlinear activation function, and is used to extract local features and contextual features of the stitched image. The multi-scale fusion module includes a pyramid submodule and a fusion submodule. The pyramid submodule includes at least two pyramid structures for fusing the feature data output by the feature extraction module, processing features at different scales into features of different modalities and channels, and outputting the processed data to the fusion submodule. Each pyramid structure includes a top branch, a middle branch, and a bottom branch. The top branch includes one ABR component and one MVSS component connected in series. The middle branch includes two ABR components and one MVSS component connected in series. The bottom branch includes three ABR components and one MVSS component connected in series. The fusion submodule is used to use a gating mechanism to weightedly combine features from different modalities and channels obtained from different pyramid layers, and outputs the fused data to the output reconstruction module. The output reconstruction module is used to output the fused image.

2. The medical image fusion system based on VMamba with cross-modal combined grouped ACmix convolution and selective clustering as described in claim 1, characterized in that: The functional image is any one of PET, SPECT, CT, and MRI; the structural image is MRI; when the functional image is PET or SPECT, the functional image submodule is used to input the functional image and convert it into YUV space, and extract the luminance channel Y; at this time, the stitching submodule is used to stitch the luminance channel Y and MRI; the output reconstruction module is used to synthesize the fused image with the original U / V components to form a color fused image and output it.

3. The medical image fusion system based on VMamba with cross-modal combined grouped ACmix convolution and selective clustering as described in claim 2, characterized in that: The ABR component consists of an ACmix unit, batch normalization, and a nonlinear activation function, and is used to extract local and contextual features of the stitched image.

4. The medical image fusion system based on VMamba with cross-modal combined grouped ACmix convolution and selective clustering as described in claim 3, characterized in that: Before the branch, the pyramid structure is equipped with a channel segmentation CS module, which is used to divide the input feature map into smaller groups, so that the branch can extract features independently for each group.

5. A medical image fusion system based on VMamba with cross-modal combined grouped ACmix convolution and selective clustering as described in claim 4, characterized in that: The MVSS component includes a multi-layered attention modulation state space module, Vision-Mamba, which comprises a state space module and an attention modulation module. The state space module introduces a cross-modal attention context a_t according to the formula h_t=Δ·h_{t-1}+B·x_t, making Δ and B functions of a_t: h_t=Δ(a_t)·h_{t-1}+B(a_t)·x_t, thereby achieving content-based dynamic updates.

6. A medical image fusion method based on cross-modal VMamba combining grouped ACmix convolution and selective clustering, characterized in that: The system described in any one of claims 1 to 5 is used to complete the task, which includes the following steps: S1. Input preprocessing stage: The input functional image and structural image channels are concatenated and then input into the fusion network; S2, Feature Extraction Stage: Local and contextual features are extracted using the ABR component; S3, Multi-scale fusion stage: Features of different scales are fused through at least two pyramid structures, and the output is then weighted and combined from different modalities and channels using a gating mechanism. S4. Output Reconstruction Stage: Upsampling and convolutional fusion to output fused brightness map. Combined with the original U / V components to form a color fused image .

7. A medical image fusion method based on cross-modal combined grouped ACmix convolution and selective clustering VMamba according to claim 6, characterized in that: In S2, the ABR component extracts local and contextual features as follows: Given input Feature maps are generated through three 1×1 convolutions. And grouped by channel; (1) Local feature extraction: The three feature groups are reshaped and spliced ​​together. Then obtained through a fully connected layer ; Utilizing depthwise separable convolution pairs Processing is performed to obtain local features. ; (2) Contextual feature extraction: extracting feature groups The query Q, key K, and value V, respectively, are used as flattening inputs. A multi-head self-attention mechanism is applied; contextual features are captured by concatenating the outputs of all attention heads. ; Local features With context features Adaptive fusion is achieved using learnable parameters α and β: ; The output of the ABR module is: ; in, This indicates the output characteristics after processing by the ABR module. This indicates an ACmix operation.

8. The medical image fusion method based on cross-modal combined grouped ACmix convolution and selective clustering VMamba according to claim 7, characterized in that: In S3, the MVSS component in the pyramid structure is a VSS block based on an improved VMamba architecture, including the following: First, the input is normalized using LN and then divided into two parallel branches: the first branch performs a linear transformation followed by a non-linear activation function; the second branch processes the input through a linear layer, depthwise separable convolution, and activation function. The result is then input into a 2D selective scanning (SS2D) module to enhance feature extraction. Next, the extracted features are normalized again using LN and fused with the output of the first branch through element-wise multiplication, effectively integrating the two paths. Finally, the fused features are processed through a linear mixing layer and a residual connection is established with the original input to obtain the final output of the VSS block.

9. A medical image fusion method based on cross-modal combined grouped ACmix convolution and selective clustering of VMamba according to claim 8, characterized in that: In S3, the formal definition of multi-scale feature representation is as follows: ； ； ； in, , , These represent the outputs of the three parallel branches. This represents the SSM module, which represents the multi-stage visual state space model used in the network architecture. When constructing the second layer of the pyramid, spatial downsampling of the input features is performed according to the principles of feature pyramid networks; this process can be represented as: ； in, This indicates the downsampling (DS) process. This represents the channel splitting CS process; ； ； ； in, , , This represents the output of the three branches; subsequently, the outputs of these three branches are concatenated with the output of the second level to integrate the features obtained from each path; this process can be quantitatively represented as: ； Here, "+" represents the addition operation. Indicates an upsampling operation. Conv This represents the convolution operation. Concat This indicates a splicing operation. Tanh This represents the hyperbolic tangent activation function.

10. A medical image fusion method based on cross-modal combined grouped ACmix convolution and selective clustering of VMamba according to claim 9, characterized in that: It also includes S5, the loss function evaluation stage: the composite loss function specifically includes regional mutual information (RMI) loss, structural similarity index (SSIM) loss, and intensity loss; the total loss is defined as follows: ； In the formula, the weights µ 1. µ 2 and µ 3 is used for adjustment L RMI , L SSIM and L Intensity The trade-off between them.

Images