A segmentation-aware multimodal medical image fusion method and system
By employing a segmentation-aware multimodal medical image fusion method, a closed-loop optimization of fusion-segmentation-quality enhancement is formed, which solves the problem of inaccurate recovery of lesion boundaries and weak contrast regions in existing technologies, and achieves higher-precision lesion segmentation and cross-dataset adaptability.
Patent Information
- Application Number
- CN202611438031.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing medical image fusion methods struggle to accurately restore lesion boundaries and weak contrast areas while maintaining overall image clarity. Furthermore, they lack generalization ability across institutions and datasets, failing to effectively support clinical diagnosis and segmentation tasks.
A segmentation-aware multimodal medical image fusion method is adopted. By forming a closed-loop optimization of fusion-segmentation-quality enhancement, and utilizing reversible feature encoding, gated memory fusion mechanism and stepwise region expansion, the lesion boundary and key structure are strengthened, thereby improving the downstream segmentation accuracy.
It effectively improves the recovery of lesion boundaries and fine textures, reduces the impact of noise and artifacts, reduces the dependence on complete pixel-level annotation, enhances the model's cross-device and cross-modal generalization ability, and provides traceable clinical deployment assurance.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, specifically to a segmentation-aware multimodal medical image fusion method and system. Background Technology
[0002] In clinical diagnosis, treatment planning, and preoperative navigation, a single medical imaging modality typically provides only partial information about tissues and organs. For example, CT images clearly show bony structures, MRI provides better visualization of soft tissues and lesion boundaries, while PET reflects metabolic activity. Multimodal medical image fusion, by integrating complementary information from different modalities into the same image space, can reduce the workload of physicians repeatedly comparing multiple images and provide a unified input for downstream tasks such as lesion detection and target delineation.
[0003] Existing medical image fusion methods mainly include multi-scale transformation, sparse representation, convolutional neural networks, and Transformer-based fusion architectures. These methods typically use general metrics such as information entropy, average gradient, or structural similarity as optimization targets, which can improve the visual quality of the fused image to some extent. However, the core requirements for clinical auxiliary diagnosis are not only the overall image clarity, but also the continuity of lesion boundaries, the discernibility of weak contrast regions, and whether the fusion result is beneficial to subsequent segmentation tasks. Simply optimizing general fusion metrics is insufficient to directly guarantee that the fused image has a positive impact on downstream semantic tasks.
[0004] In multi-scale feature extraction and fusion, networks typically need to perform downsampling, channel compression, and cross-modal weighting. While these operations are beneficial for expanding the receptive field and integrating global structure, they may also weaken high-frequency information such as local edges, subtle textures, and weak-contrast lesions. Especially for brain tumors, micronodules, and irregular soft tissue lesions, once local structural information is lost in the upstream fusion stage, the downstream segmentation network struggles to recover accurate boundaries solely based on high-level semantic features. Some solutions attempt to add enhancement or super-resolution modules after fusion, but such enhancements usually aim at pixel reconstruction of the entire image without leveraging the segmentation results to locate the key regions that truly need enhancement. This can easily lead to over-enhancing of non-lesion areas or amplifying noise and artifacts.
[0005] On the other hand, in existing schemes that connect fusion and segmentation networks, segmentation is usually only used as a post-processing module, and its output cannot be used to influence the fusion features, thus failing to create a closed-loop constraint between the semantic task and the upstream visual representation. Furthermore, real-world clinical data exhibits differences in equipment models, grayscale ranges, scarce annotations, and incomplete modal pairings. Existing joint training methods often rely on complete pixel-level annotations, limiting their generalization capabilities across institutions and datasets. Additionally, in the engineering deployment of PACS and medical imaging workstations, mechanisms for handling anomalies such as registration failures and insufficient confidence are still inadequate. Summary of the Invention
[0006] The purpose of this invention is to propose a segmentation-aware multimodal medical image fusion method and system, forming a closed-loop optimization of fusion-segmentation-quality enhancement, strengthening lesion boundaries and key structures, and improving downstream segmentation accuracy.
[0007] According to a first aspect of the present disclosure, a segmentation-aware multimodal medical image fusion method is provided, comprising the following steps: Acquire at least two spatially registered source modal medical images of the same object to be analyzed, and perform preprocessing. The preprocessed source modal medical images are input into the fusion sub-network, and reversible feature encoding and cross-modal fusion at multiple resolution scales are performed to obtain fusion features at each scale and decode them to generate the initial fusion image. The initial fused image is input into the segmentation sub-network to generate an initial segmentation probability map. Pixel-level confidence maps and boundary response maps are calculated based on the initial segmentation probability map. Then, stepwise region expansion is performed according to the initial segmentation probability map, the confidence map, and the boundary response map to obtain soft interest region priors representing the target region, target boundary, and target neighborhood. According to the preset quality evaluation rules, the local image patches of the initial fused image are divided into low-quality and high-quality regions. The statistical quality prototype of the high-quality region is extracted, and the enhanced features guided by the region of interest are generated based on the content features of the low-quality region image patches, the statistical quality prototype, and the soft region of interest prior. The enhanced features guided by the region of interest are fed back to the fusion sub-network and decoded together with the fusion features at each scale to generate the final fused image; The final fused image is input into the segmentation sub-network, which shares the same network parameters as the initial segmentation stage, to obtain the final segmentation result.
[0008] In one embodiment, the reversible feature encoding includes: in the first... First modal features are obtained at each resolution scale. Second mode features The reversible transformation characteristics are obtained according to the following formula. and : in, , and For feature mapping function, For the Sigmoid function, This is element-wise multiplication; The same-scale cross-modal fusion is as follows: in, For the first A scale fusion module for each resolution scale.
[0009] In one embodiment, the segmentation subnetwork adopts an encoder-decoder structure and includes a KAN feature transformation module, a state space feature modeling module, and an attention module; The KAN feature transformation module is used to model the local nonlinear structure of the input features, and its transformation form is as follows: in, As input features, These are learnable nonlinear basis functions; Dimension of input features The state-space feature modeling module is used to model long-range dependencies on the two-dimensional feature expansion sequence, and its discrete form is as follows: in, and Sequence position Input and output, In hidden state, , , and These are learnable parameters; The segmented subnetworks share network parameters in the initial segmentation phase and the final segmentation phase. ,satisfy: in, For parameters The partitioning mapping function, For the initial fused image, This is the initial segmentation probability map. For the final segmentation result, Final merged image.
[0010] In one embodiment, the stepwise region expansion includes: normalizing the probability of the initial segmentation probability map, and performing a differentiable morphological dilation operation according to an increasing expansion scale, while retaining the probability weights, to obtain the... Prior knowledge of soft regions of interest: in, For the first Level-wise region expansion function, For extended series; Normalized segmentation probability map; The first-level soft region of interest prior is obtained by smoothing the difference between the initial segmentation probability map and the segmentation probability threshold. Subsequent levels of region of interest priors are calculated by performing morphological expansion on the previous level prior and superimposing the boundary response map. in, For the Sigmoid function, For smoothing coefficients, The segmentation probability threshold, The initial segmentation probability map after normalization In pixels The probability value at that location; For the expansion radius is The micro-expansion operation, For boundary weights, Used to limit the result to between 0 and 1; Boundary response map at pixels The value at that location, Confidence map at pixels The value at that location, No. Level soft interest region prior at the pixel The weight value at that point, ≥2. By making Follow The soft region of interest is increased to progressively cover the boundary and adjacent anatomical context from the lesion core, rather than using the initial segmentation results as a hard mask that completely determines the extent of enhancement.
[0011] In one embodiment, the statistical quality prototype By quality encoder For high-quality domains Image patches in The mean value after encoding is obtained as follows: in, For the number of high-quality image patches; The enhanced features guided by the region of interest are generated through a content encoder. Extracting low-quality image patches Content features Enhanced image patches are generated by the enhancement generator. : The high-quality image patch is used to provide a priori quality distribution and is not used as the spatial corresponding pixel-level reconstruction target of the low-quality image patch, thereby achieving unpaired quality transfer enhancement.
[0012] In one embodiment, a gated memory fusion unit is used for decoding, the gated memory fusion unit receiving the current scale fusion features. Previous scale memory state The soft interest region prior and the enhanced features The update process is as follows: in, This represents the convolution operation. For upsampling operations, the gate coefficient It is used to select between current-scale details, deep structural memory, and quality enhancement feedback, thereby improving cross-scale structural continuity; , This is a learnable bias term.
[0013] In one embodiment, the fusion subnetwork, segmentation subnetwork, and quality enhancement subnetwork are models that have been pre-trained jointly, and the objective function used in the joint training is: in, For the first Level of interest area supervision tags, , and These are, respectively, quality distribution alignment constraints, content consistency constraints, and soft interest region weighted structural similarity constraints. , , , , , , , , and To balance the parameters, The total segmentation loss of the segmented subnetwork is given by the following statement: For the focus of loss, For Dice's loss, For the loss of fusion, For gradient loss, For structural similarity loss, For monitoring loss in the region of interest, Expand the total number of levels, Binary Dice loss function, No. Priors for soft regions of interest; For quality enhancement loss, ; in, and These are the pixel prediction value and the true label, respectively. For category weights, For focusing parameters; For samples without segmentation labels, only the fusion path and the quality encoding path that can be supervised by unsupervised quality rules are updated; for samples with segmentation labels, the fusion, segmentation, region of interest, and quality enhancement paths are updated together. This allows for the utilization of complementary supervision signals from different datasets and medical institutions without requiring all data to have complete pixel-level annotations simultaneously.
[0014] According to a second aspect of the present disclosure, a segmentation-aware multimodal medical image fusion system is provided, comprising: The image acquisition and preprocessing module acquires at least two spatially registered source modal medical images of the same object to be analyzed and performs preprocessing. The multi-scale reversible fusion module inputs the preprocessed source modal medical image into the fusion sub-network, performs reversible feature encoding and same-scale cross-modal fusion at multiple resolution scales, obtains fusion features at each scale, and decodes them to generate the initial fusion image. The semantic prior generation module inputs the initial fused image into the segmentation sub-network to generate an initial segmentation probability map, and calculates a pixel-level confidence map and a boundary response map based on the initial segmentation probability map. Then, it performs stepwise region expansion based on the initial segmentation probability map, the confidence map, and the boundary response map to obtain soft interest region priors representing the target region, target boundary, and target neighborhood. The region of interest-guided quality enhancement module divides local image patches of the initial fused image into low-quality and high-quality regions according to preset quality evaluation rules, extracts the statistical quality prototype of the high-quality region, and generates enhanced features for region of interest guidance based on the content features of the low-quality region image patches, the statistical quality prototype, and the soft region of interest prior. The enhanced feature feedback fusion module feeds back the enhanced features guided by the region of interest to the fusion sub-network, and decodes them together with the fusion features at each scale to generate the final fused image; The shared parameter segmentation output module inputs the final fused image into the segmentation sub-network that shares the same network parameters as the initial segmentation stage, and obtains the final segmentation result.
[0015] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the segmentation-aware multimodal medical image fusion method described above.
[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the segmentation-aware multimodal medical image fusion method described above.
[0017] The advantages of the above technical solutions adopted in this invention compared with the prior art are as follows: This invention incorporates multimodal fusion, lesion segmentation, and quality enhancement into a unified closed-loop architecture. Segmentation semantics is not only used for the final output, but also actively locates and enhances lesion boundaries, subtle textures, and weak contrast structures that are easily lost during the upstream fusion process. This breaks through the limitation of traditional fusion methods that only optimize general visual indicators, and enables the fused image to truly serve the downstream diagnostic task.
[0018] This invention employs a reversible multi-scale coding and gated memory fusion mechanism to preserve source modal information at different resolution levels. It also compensates for information loss caused by continuous downsampling and feature compression through enhanced feature feedback, effectively improving cross-scale structural continuity and detail restoration capabilities. Furthermore, based on a progressively expanding soft interest region prior, it considers the lesion core, boundaries, and adjacent context, preserving surrounding anatomical information even when initial segmentation is uncertain, significantly reducing the risk of false enhancement by binary hard masks.
[0019] Furthermore, this invention achieves local enhancement through high-quality domain statistical prototyping and unpaired quality transfer, eliminating the need to prepare spatially corresponding high-quality image patches for each low-quality image patch. This significantly reduces reliance on complete pixel-level labeled data, facilitating training with real clinical data. The defined joint training strategy supports hybrid optimization of labeled and unlabeled samples, further enhancing the model's generalization ability across devices and modalities, and providing traceable and verifiable security for clinical deployment. Attached Figure Description
[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.
[0021] Figure 1 This is a flowchart of the segmentation-sensing multimodal medical image fusion method of the present invention; Figure 2 This is a diagram illustrating the overall architecture of the fusion subnetwork, segmentation subnetwork, and region of interest-guided quality enhancement subnetwork of the present invention. Figure 3 This is a flowchart illustrating the operation of the reversible encoding unit of the present invention. Figure 4 This is a flowchart illustrating the operation of the gated memory fusion unit of the present invention. Figure 5 This is a flowchart illustrating the workflow of the KAN-U-Net segmentation subnetwork of the present invention. Figure 6 This is a flowchart illustrating the state space or Mamba module of the present invention. Figure 7 This is a flowchart of the self-supervised quality domain learning process of the region of interest guided quality enhancement module of the present invention; Figure 8 This is a functional block diagram of the medical image fusion system of the present invention; Figure 9 This is a flowchart illustrating the deployment of the present invention in hospital PACS and medical imaging workstations. Figure 10 This is a schematic diagram illustrating the progressive expansion of the soft interest region prior in this invention. Detailed Implementation
[0022] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.
[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0025] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.
[0026] Example 1: like Figure 1-2 As shown, this embodiment provides a segmentation-aware multimodal medical image fusion method. Its inputs are a T1-enhanced sequence and a FLAIR sequence from the same patient and the same examination. The outputs are a fused image, a mask of the entire tumor region, a pixel-level confidence map, and quality control status. Specifically, it includes the following steps: S1. Acquire at least two spatially registered source modal medical images of the same object to be analyzed, and perform preprocessing; The source modal medical images in this application may include CT, MRI, PET, SPECT, ultrasound, digital subtraction angiography, and other spatially registrable medical images. The first and second source modalities are only used to distinguish the inputs and do not limit the specific modal combinations. Optional combinations include CT and MRI, T1ce and FLAIR, T1 and T2, PET and MRI, low-dose CT and conventional CT, etc.
[0027] Specifically, T1-enhanced and FLAIR sequences in DICOM or NIfTI format are read from PACS, image cloud platforms, or local directories. The data can originate from MRI equipment from Siemens Healthineers, United Imaging Healthcare, GE Healthcare, Philips Healthcare, or other manufacturers; these equipment names are merely examples for compatibility and do not constitute a limitation on the input device. When reading DICOM data, pixel spacing, slice thickness, orientation cosine, and patient coordinate system information are preserved, and patient name, hospital number, and other identity fields are anonymized.
[0028] If the two modes are not yet registered, the transformation matrix can be obtained by rigid registration or affine registration. Furthermore, non-rigid fine-tuning based on mutual information can be employed. After registration, normalized mutual information or structural similarity is calculated as the registration quality index. When the quality index is lower than a preset threshold, the system stops automatic joint inference and returns to the "registration needs to be reviewed" state. After successful registration, the two modalities are resampled to the same voxel spacing, and foreground clipping and Z-score normalization are performed on each volumetric data. in, and These represent the mean and standard deviation of the current modal foreground region. For two-dimensional inference, volume data can be cropped or scaled layer by layer. Pixels; for 3D reasoning, one can use... Or other image blocks that can be supported by video memory.
[0029] In this embodiment, the first source modal image is set as... The second source modal image is [image value], and its size is [image size]. The fusion subnetwork, the segmentation subnetwork, and the quality enhancement subnetwork are respectively denoted as... , and .
[0030] S2. Input the preprocessed source modal medical image into the fusion sub-network, perform reversible feature encoding and same-scale cross-modal fusion at multiple resolution scales, obtain fusion features at each scale, and decode to generate the initial fusion image; In this embodiment, the fusion subnetwork is set to three scales. The number of feature channels for the first, second, and third scales are set to 32, 64, and 128, respectively, and the convolutional kernels are... The downsampling step size is 2. Reversible encoding is performed on the T1 enhancement and FLAIR features at each scale to obtain... and .like Figure 3 As shown, the reversible coding unit realizes forward mapping and reverse information recovery; the first The source modal features at each scale are represented as follows: Its reverse recovery is: in, , and For learnable mappings, and These are element-wise multiplication and division, respectively. For the first Scale-reversible coding units. The aforementioned reversible structure preserves input information during feature propagation and reduces irreversible losses caused by continuous downsampling and channel compression.
[0031] Each scale has a scale fusion unit. The scale fusion unit first calculates the channel weights for the two modalities, then performs weighted summation and concatenated convolution. For significantly enhanced tumor regions in T1-enhanced sequences, the network increases the weights of T1-enhanced features; for significant edema and overall tumor extent in FLAIR sequences, the network increases the weights of FLAIR features. These weights are adaptively learned from the training data, without manually specifying lesion locations. The scale fusion unit adaptively combines the first and second modal features using modal weights. in, The weight generation function is composed of global pooling and fully connected mappings. Indicates channel splicing. This is a convolutional mapping.
[0032] Same-scale fusion features are represented as: in, This is a scale fusion module.
[0033] The deepest fused features are upsampled and then enter the gated memory fusion unit. The gated memory fusion unit is as follows: Figure 4 As shown, during the decoding process, the current scale fused features, the previous scale memory features, and subsequent quality enhancement feedback features are received. Before the first forward computation generates quality enhancement feedback, the current scale fused features, the previous scale memory features, and subsequent quality enhancement feedback features are received. Initialize to zero tensor to generate initial fused image .
[0034] S3. Input the initial fused image into the segmentation sub-network to generate an initial segmentation probability map, and calculate a pixel-level confidence map and a boundary response map based on the initial segmentation probability map. Then, perform stepwise region expansion according to the initial segmentation probability map, the confidence map, and the boundary response map to obtain soft interest region priors representing the target region, target boundary, and target neighborhood. Initial fused image Input segmentation subnetwork. The segmentation subnetwork adopts a U-Net encoder-decoder structure, with KAN convolutional blocks at the three encoding scales, state space or Mamba modules at the first two low-resolution scales, and attention gates at the decoding skip connections. The KAN-U-Net structure is as follows: Figure 5 As shown, the state space module is as follows Figure 6 As shown.
[0035] In the binary classification of the entire tumor segmentation, the segmentation subnetwork outputs a lesion probability map. After calculating the confidence map, the boundary response is obtained using Sobel gradient or a learnable edge operator. For multi-class segmentation, the output channels correspond to the background, enhanced tumor, tumor core, and edema region, respectively. Multiple tumor categories can be merged into a single tumor probability map to construct a prior for the region of interest.
[0036] in, The number of categories to be divided. For pixels Belongs to the Predicted probability of class To prevent smoothing terms with zero values from appearing in logarithmic operations, This is the gradient operator. For binary lesion segmentation, This is a probability map of lesions.
[0037] This embodiment is set Segmentation probability threshold Smoothing coefficient Three extended radii , and The regions of interest (ROIs) are 2, 4, and 8 pixels respectively. The first-level ROI primarily covers the core of high-confidence lesions; the second-level ROI covers the lesion boundaries; and the third-level ROI further covers the adjacent anatomical context. Each ROI is not a 0 or 1 hard mask, but rather a continuous weighted map. Figure 10 This illustrates the process of the region of interest expanding step by step.
[0038] When the initial segmentation probability is low but the boundary response is high, the equation The boundary term in the algorithm can retain the enhancement weight of the region; when the initial segmentation confidence is low, the confidence term reduces its weight, thereby avoiding over-enhancing of erroneously segmented regions. For isolated regions with an area smaller than a preset threshold, connected component filtering can be used to reduce noise response, but satellite regions closer to the main lesion are retained for subsequent review.
[0039] S4. According to the preset quality evaluation rules, the local image patches of the initial fused image are divided into low-quality domains and high-quality domains. The statistical quality prototype of the high-quality domain is extracted. Based on the content features of the low-quality domain image patches, the statistical quality prototype, and the soft interest region prior, interest region-guided enhancement features are generated. Specifically, the initial fused image is divided into... Image patches are slidably segmented with a 16-pixel step size; local quality scores are obtained based on entropy values. In this embodiment, the weights for gradient strength, local contrast, noise estimation, and structural consistency can be set to 0.35, 0.25, 0.15, and 0.25, respectively. Image patches with quality scores in the top 30% of the current batch are designated as the high-quality region, the bottom 30% as the low-quality region, and the remaining image patches as content-preserving samples.
[0040] The self-supervised quality domain learning process of the quality enhancement module is as follows: Figure 7 As shown. The high-quality domain is only used to calculate the statistical quality prototype. It is not required that the low-quality image patch spatially correspond to the low-quality image patch. The low-quality image patch is processed by a content encoder to extract content features, which are then combined with the high-quality prototype and soft region of interest (ROI) prior input to enhance the generator. The generator uses residual convolution and pixel rearrangement or interpolation convolution to output enhanced features. For image patches with high ROI weights, the enhancer focuses more on edge continuity and local structure; for image patches with low ROI weights, the enhancer mainly performs mild denoising and content preservation.
[0041] S5. The enhanced features guided by the region of interest are fed back to the fusion sub-network and decoded together with the fusion features at each scale to generate the final fused image; The enhanced image patches are overlaid and backfilled in their original positions. Weighted average fusion is used in the overlapping areas to avoid patch boundaries. The feature transfer unit maps the enhancement results to three scales to obtain... , and The gated memory fusion unit selectively aggregates the initial fusion features and enhancement feedback, and the fusion decoder outputs the final fused image. .
[0042] in, For feature transmission unit, For enhanced local image patches or their feature sets, For fusion decoder.
[0043] S6. Input the final fused image into the segmentation sub-network that shares the same network parameters as the initial segmentation stage to obtain the final segmentation result.
[0044] The final fused image is then input again into the segmentation sub-network with shared parameters to obtain the final segmentation result S. Thresholding, connected component filtering, and hole filling are performed on the probability map to obtain the lesion mask. Simultaneously, the unprocessed probability map is retained for clinical workstations to display uncertain regions. The output should include at least: T1-enhanced original image, FLAIR original image, initial fused image, final fused image, initial segmentation probability map, final segmentation mask, confidence map, region of interest map, model version, inference time, and quality control status.
[0045] The final segmentation result is represented as follows: The initial and final segmentations share network parameters. The first stage is used to construct semantic location priors, and the second stage is used to segment the final fused image after feedback enhancement, thus forming a closed-loop processing link of "fusion-segmentation-quality enhancement-fusion-segmentation".
[0046] The preferred training parameters for this embodiment are shown in Table 1.
[0047] Table 1. One feasible configuration of training and inference parameters. This embodiment can be implemented using PyTorch, TensorFlow, MindSpore, or other deep learning frameworks. Training devices may include NVIDIA GeForce RTX 3070Ti, professional GPU servers, domestically produced AI accelerator cards, or edge computing devices that support tensor operations. After model training is complete, it can be exported to ONNX, TensorRT, TorchScript, or vendor-specific inference formats, and connected to the clinical system via REST, gRPC, DICOM C-STORE, or file listening services.
[0048] Furthermore, when unlabeled fusion data and labeled segmentation data coexist, the unlabeled dataset can include registered CT-MRI image pairs, while the labeled dataset can include brain tumor data with T1 enhancement, FLAIR, and tumor masks. Two data loaders are established during training. The first data loader outputs multimodal image pairs without pixel-level segmentation labels, calculating only gradient loss, structural similarity loss, and quality domain loss that can be constructed from quality rules. The second data loader outputs multimodal image pairs with segmentation labels, simultaneously calculating fusion loss, segmentation loss, region of interest supervision loss, and quality enhancement loss. The two data loaders sample alternately at a 1:1, 2:1, or other ratios and jointly update the shared fusion network.
[0049] For unlabeled batches, the label indicates the quantity. Only the fusion path and unsupervised quality path are updated. For labeled batches, The model forms a complete closed loop. To avoid training instability caused by differences in grayscale distribution between the two data domains, instance normalization or domain-specific normalization parameters can be set before the encoder, and the same fusion and segmentation parameters can be used on shared high-level features.
[0050] In a specific experimental configuration, the unlabeled fusion data consisted of 120 pairs of training images, 50 pairs of validation images, and 30 pairs of test images; the labeled brain tumor data was divided into training, validation, and test sets at 70%, 20%, and 10%, respectively. Alternating sampling was used during the training phase, and fusion and segmentation metrics were calculated separately during the validation phase. Fusion metrics included mean gradient, differential correlation sum, spatial frequency, standard deviation, edge strength, entropy, NIQE, and BRISQUE; segmentation metrics included Dice and HD95.
[0051] In one training result of the above configuration, the final fusion method achieved an average gradient of 16.5973, a spatial frequency of 39.4631, a standard deviation of 87.1975, a marginal strength of 98.3999, an NIQE of 3.7404, and an entropy of 5.5548 on the independent fusion test set; and a Dice of 92.0977% and an HD95 of 5 on the independent brain tumor test set. These values are only used to illustrate that the scheme can actually train and output quantifiable results, and do not constitute a limitation on the performance range of the invention. The metrics may vary under different data cleaning, case segmentation, and 3D or 2D implementation methods.
[0052] Furthermore, on the validation subset used to verify the synergistic effect of fusion, quality enhancement, and segmentation, the full joint model achieved a Dice of 96.9226% and an HD95 reduced to 3.1623; when using only the segmentation network, the Dice was 89.8922% and the HD95 was 12.0000. These results indicate that region-of-interest guided quality enhancement can serve as a task bridging path between fusion and segmentation, improving the representation of lesion regions and boundaries.
[0053] To avoid misleading clinical use with experimental results, the statistical indicators of the training set or validation set are not directly displayed as the basis for diagnosis of a single case during deployment. Instead, the confidence level, registration quality, and model version of the current case are displayed, and physicians are required to review them.
[0054] like Figure 8-9 As shown, this embodiment deploys the above method as a medical image fusion service within a hospital intranet. The system includes a DICOM receiving service, a preprocessing service, a federated inference service, a result encapsulation service, a PACS interface, a model management service, and an auditing service.
[0055] The DICOM receiving service receives specified examination sequences via C-STORE, DICOMweb, or hospital message queues. Sequence matching rules can identify T1-enhanced, FLAIR, CT, PET, and other sequences based on SeriesDescription, ProtocolName, Modality, and image spatial information. If multiple candidate sequences exist for the same examination, the input is selected based on slice integrity, image orientation consistency, acquisition time proximity, and quality score, and the selection rules are recorded in the log.
[0056] The preprocessing service performs desensitization, registration, resampling, and normalization. For sequences that have already been registered by the hospital, the registration object or shared coordinate information can be read to skip duplicate registration; for images from other hospitals or historical images, automatic registration is performed and the transformation matrix is retained. All intermediate files are managed using case anonymization tags, and access permissions and retention periods are set.
[0057] The joint inference service loads validated model versions. Each model version includes network weights, normalization methods, input modality requirements, threshold parameters, training data summaries, validation metrics, and software dependencies. The inference service uses the GPU when available and can switch to the CPU or a backup accelerator card in case of GPU failure, but the actual running device and execution time must be recorded in the results.
[0058] The result encapsulation service encapsulates the final fused image into a DICOM secondary capture object or an enhanced DICOM object, encapsulates the segmentation mask into a DICOM-SEG or RTSTRUCT, and writes the confidence level, model version, quality control status, and key parameters into a structured report. For radiotherapy scenarios, the contours reviewed by physicians can be sent to the treatment planning system; for general diagnostic scenarios, the lesion contours can be displayed as an overlay in the PACS image viewing interface.
[0059] The following exception handling rules are set in this embodiment: If any input sequence is missing, the number of layers is inconsistent, or the spatial information is invalid, fusion is not performed, and the input is returned as incomplete. When the registration quality is below the threshold, only the source image and registration preview are displayed, and the automatic segmentation results are not written back. Cases with low confidence are marked as having an average confidence level below the threshold, abnormal lesion volume, or a large difference between the initial and final segments. When the structural similarity of non-interest regions before and after enhancement decreases beyond a threshold, the image is reverted to the initial fused image to avoid altering non-critical structures. When inference program, model validation values, or dependent environment are inconsistent, stop processing and write to the audit log; All automated results are marked as "diagnostic aids" and must be reviewed by a qualified physician before being included in the formal report or treatment plan.
[0060] Example 2: This embodiment provides a segmentation-aware multimodal medical image fusion system, including: The image acquisition and preprocessing module acquires at least two spatially registered source modal medical images of the same object to be analyzed and performs preprocessing. The multi-scale reversible fusion module inputs the preprocessed source modal medical image into the fusion sub-network, performs reversible feature encoding and same-scale cross-modal fusion at multiple resolution scales, obtains fusion features at each scale, and decodes them to generate the initial fusion image. The semantic prior generation module inputs the initial fused image into the segmentation sub-network to generate an initial segmentation probability map, and calculates a pixel-level confidence map and a boundary response map based on the initial segmentation probability map. Then, it performs stepwise region expansion based on the initial segmentation probability map, the confidence map, and the boundary response map to obtain soft interest region priors representing the target region, target boundary, and target neighborhood. The region of interest-guided quality enhancement module divides local image patches of the initial fused image into low-quality and high-quality regions according to preset quality evaluation rules, extracts the statistical quality prototype of the high-quality region, and generates enhanced features for region of interest guidance based on the content features of the low-quality region image patches, the statistical quality prototype, and the soft region of interest prior. The enhanced feature feedback fusion module feeds back the enhanced features guided by the region of interest to the fusion sub-network, and decodes them together with the fusion features at each scale to generate the final fused image; The shared parameter segmentation output module inputs the final fused image into the segmentation sub-network that shares the same network parameters as the initial segmentation stage, and obtains the final segmentation result.
[0061] The above modules can be deployed on the same device or distributed devices; the division of modules is only a functional logic description and does not limit the specific physical boundaries or implementation order.
[0062] This system can be deployed on local hospital servers, radiology workstations, private clouds, or regional medical imaging platforms. For scenarios where data cannot leave the hospital's intranet, both model inference and log services are deployed within the hospital. For multi-center collaborative scenarios, federated learning or model parameter exchange can be used to update the model without transmitting the original patient images.
[0063] Example 3: An electronic device is provided for running the aforementioned "a segmentation-aware multimodal medical image fusion method". The electronic device includes a processor, a memory, and optional communication interfaces / display devices / input devices, etc.; the memory stores a computer program that can run on the processor, and when the processor executes the program, it implements steps S1 to S6 of the method described in Embodiment 1, specifically including but not limited to: S1. Acquire at least two spatially registered source modal medical images of the same object to be analyzed, and perform preprocessing; S2. Input the preprocessed source modal medical image into the fusion sub-network, perform reversible feature encoding and same-scale cross-modal fusion at multiple resolution scales, obtain fusion features at each scale, and decode to generate the initial fusion image; S3. Input the initial fused image into the segmentation sub-network to generate an initial segmentation probability map, and calculate a pixel-level confidence map and a boundary response map based on the initial segmentation probability map. Then, perform stepwise region expansion according to the initial segmentation probability map, the confidence map, and the boundary response map to obtain soft interest region priors representing the target region, target boundary, and target neighborhood. S4. According to the preset quality evaluation rules, the local image patches of the initial fused image are divided into low-quality domains and high-quality domains. The statistical quality prototype of the high-quality domain is extracted. Based on the content features of the low-quality domain image patches, the statistical quality prototype, and the soft interest region prior, interest region-guided enhancement features are generated. S5. The enhanced features guided by the region of interest are fed back to the fusion sub-network and decoded together with the fusion features at each scale to generate the final fused image; S6. Input the final fused image into the segmentation sub-network that shares the same network parameters as the initial segmentation stage to obtain the final segmentation result.
[0064] The electronic device hardware can be one of a server, personal computer, workstation, industrial controller, edge computing device, or mobile terminal; the processor can be a general-purpose CPU, GPU, NPU, FPGA, or a combination thereof; the memory can be RAM, ROM, flash memory, or disk array. The device can interact with local / remote data storage via a communication interface. The above hardware configuration does not constitute a limitation of the present invention.
[0065] Example 4: A computer-readable storage medium storing a computer program, which, when run on a processor of an electronic device, causes the program to execute the method steps S1 to S7 described in Embodiment 1; the storage medium may be a disk, optical disk, flash memory, solid-state drive, read-only memory, random access memory, or any combination of the above media.
[0066] For 3D medical images, 2D convolutions can be replaced with 3D convolutions, 2D image patches can be replaced with 3D image patches, and sliding window inference can be used when video memory is insufficient. For inputs with more than two modalities, the scale fusion unit can be extended to multi-branch weight normalization. in, To input the number of modes, For the first The first mode in the 1st The scale characteristics. Therefore, this invention can be extended to applications such as trimodal or quadmodal MRI, PET-CT-MRI, etc.
[0067] Those skilled in the art will understand that the modules or steps described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, which can then be stored in a storage device for execution by a computer device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. This disclosure is not limited to any particular combination of hardware and software.
[0068] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0069] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.
Claims
1. A segmentation-aware multimodal medical image fusion method, characterized in that, Includes the following steps: Acquire at least two spatially registered source modal medical images of the same object to be analyzed, and perform preprocessing. The preprocessed source modal medical images are input into the fusion sub-network, and reversible feature encoding and cross-modal fusion at multiple resolution scales are performed to obtain fusion features at each scale and decode them to generate the initial fusion image. The initial fused image is input into the segmentation sub-network to generate an initial segmentation probability map. Pixel-level confidence maps and boundary response maps are calculated based on the initial segmentation probability map. Then, stepwise region expansion is performed according to the initial segmentation probability map, the confidence map, and the boundary response map to obtain soft interest region priors representing the target region, target boundary, and target neighborhood. According to the preset quality evaluation rules, the local image patches of the initial fused image are divided into low-quality and high-quality regions. The statistical quality prototype of the high-quality region is extracted, and the enhanced features guided by the region of interest are generated based on the content features of the low-quality region image patches, the statistical quality prototype, and the soft region of interest prior. The enhanced features guided by the region of interest are fed back to the fusion sub-network and decoded together with the fusion features at each scale to generate the final fused image; The final fused image is input into the segmentation sub-network, which shares the same network parameters as the initial segmentation stage, to obtain the final segmentation result.
2. The segmentation-aware multimodal medical image fusion method according to claim 1, characterized in that, The reversible feature encoding includes: in the first First modal features are obtained at each resolution scale. Second mode features The reversible transformation characteristics are obtained according to the following formula. and : in, , and For feature mapping function, For the Sigmoid function, This is element-wise multiplication; The same-scale cross-modal fusion is as follows: in, For the first A scale fusion module for each resolution scale.
3. The segmentation-aware multimodal medical image fusion method according to claim 1, characterized in that, The segmentation sub-network adopts an encoder-decoder structure and includes a KAN feature transformation module, a state space feature modeling module, and an attention module; The KAN feature transformation module is used to perform local nonlinear structure modeling on the input features, and its transformation form is as follows: in, As input features, These are learnable nonlinear basis functions; Dimension of input features The state-space feature modeling module is used to model long-range dependencies on the two-dimensional feature expansion sequence, and its discrete form is as follows: in, and Sequence position Input and output, In hidden state, , , and These are learnable parameters; The segmented subnetworks share network parameters in the initial segmentation phase and the final segmentation phase. ,satisfy: in, For parameters The partitioning mapping function, For the initial fused image, This is the initial segmentation probability map. For the final segmentation result, Final merged image.
4. The segmentation-aware multimodal medical image fusion method according to claim 1, characterized in that, The progressive region expansion includes: normalizing the probability of the initial segmentation probability map, performing differentiable morphological dilation operations according to an increasing expansion scale, while retaining the probability weights, to obtain the... Prior knowledge of soft regions of interest: in, For the first Level-wise region expansion function, For extended series; Normalized segmentation probability map; First-level soft interest area The prior is obtained by smoothing the difference between the initial segmentation probability map and the segmentation probability threshold. The prior for the region of interest in subsequent levels is also obtained by this smoothing process. Based on the prior layer, morphological extension is performed and the boundary response map is superimposed. The calculation method is as follows: in, For the Sigmoid function, For smoothing coefficients, The segmentation probability threshold, The initial segmentation probability map after normalization In pixels The probability value at that location; For the expansion radius is The micro-expansion operation, For boundary weights, Used to limit the result to between 0 and 1; Boundary response map at pixel The value at that location, Confidence map at pixels The value at that location, No. Level soft interest region prior at the pixel The weight value at that location, ≥2.
5. The segmentation-aware multimodal medical image fusion method according to claim 1, characterized in that, The statistical quality prototype By quality encoder For high-quality domains Image patches in The mean value after encoding is obtained as follows: in, For the number of high-quality image patches; The enhanced features guided by the region of interest are generated through a content encoder. Extracting low-quality image patches Content features And by the enhanced generator Generate enhanced image patches : The high-quality image patch is used to provide a priori quality distribution and is not used as the spatial corresponding pixel-level reconstruction target of the low-quality image patch, thereby achieving unpaired quality transfer enhancement.
6. The segmentation-aware multimodal medical image fusion method according to claim 1, characterized in that, Decoding is performed using a gated memory fusion unit, which receives the fusion features at the current scale. Previous scale memory state The soft interest region prior and the enhanced features The update process is as follows: in, This represents the convolution operation. For upsampling operations, the gate coefficient It is used to select between current-scale details, deep structural memory, and quality enhancement feedback, thereby improving cross-scale structural continuity; , This is a learnable bias term.
7. The segmentation-aware multimodal medical image fusion method according to claim 1, characterized in that, The fusion subnetwork, segmentation subnetwork, and quality enhancement subnetwork are models that have been pre-trained jointly. The objective function used in the joint training is: in, For the first Level of interest area supervision tags, , and These are, respectively, quality distribution alignment constraints, content consistency constraints, and soft interest region weighted structural similarity constraints. , , , , , , , , and To balance the parameters, The total segmentation loss of the segmented subnetwork is... For the focus of loss, For Dice's loss, For the loss of fusion, For gradient loss, For structural similarity loss, For the region of interest, monitoring loss, Expand the total number of levels, Binary Dice loss function, No. Priors for soft regions of interest; Losses due to quality enhancement.
8. A segmentation-aware multimodal medical image fusion system, characterized in that, include: The image acquisition and preprocessing module acquires at least two spatially registered source modal medical images of the same object to be analyzed and performs preprocessing. The multi-scale reversible fusion module inputs the preprocessed source modal medical image into the fusion sub-network, performs reversible feature encoding and same-scale cross-modal fusion at multiple resolution scales, obtains fusion features at each scale, and decodes them to generate the initial fusion image. The semantic prior generation module inputs the initial fused image into the segmentation sub-network to generate an initial segmentation probability map, and calculates a pixel-level confidence map and a boundary response map based on the initial segmentation probability map. Then, it performs stepwise region expansion based on the initial segmentation probability map, the confidence map, and the boundary response map to obtain soft interest region priors representing the target region, target boundary, and target neighborhood. The region of interest-guided quality enhancement module divides local image patches of the initial fused image into low-quality and high-quality regions according to preset quality evaluation rules, extracts the statistical quality prototype of the high-quality region, and generates enhanced features for region of interest guidance based on the content features of the low-quality region image patches, the statistical quality prototype, and the soft region of interest prior. The enhanced feature feedback fusion module feeds back the enhanced features guided by the region of interest to the fusion sub-network, and decodes them together with the fusion features at each scale to generate the final fused image; The shared parameter segmentation output module inputs the final fused image into the segmentation sub-network that shares the same network parameters as the initial segmentation stage, and obtains the final segmentation result.
9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements a segmentation-aware multimodal medical image fusion method according to any one of claims 1-8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements a segmentation-aware multimodal medical image fusion method as described in any one of claims 1-8.
