A medical image segmentation system based on multi-architecture fusion and diffusion model
By combining dual-channel encoding, cross-scale feature fusion, and diffusion model post-processing, the limitations of existing medical image segmentation methods in capturing global context and local details are overcome. This achieves efficient multi-scale feature fusion and fine-grained correction, improving segmentation accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2025-08-27
- Publication Date
- 2026-07-24
AI Technical Summary
Existing medical image segmentation methods have limitations in capturing the global context and long-range dependencies of images, especially when dealing with large targets or complex scenes that require cross-regional information association. Furthermore, they lack effective post-processing mechanisms to refine the segmentation mask.
Employing a dual-channel encoding module, a cross-scale feature fusion module, and a diffusion model post-processing module, this approach achieves efficient fusion of local and global information through dynamic deformable convolution and frequency domain information interaction, combined with CNN and Transformer architectures. It also performs iterative denoising optimization through a diffusion model.
It improves the accuracy and robustness of medical image segmentation, effectively handles complex structures and non-rigid deformations, generates high-quality segmentation results that are more consistent with anatomical structures, and significantly improves indicators such as Dice coefficient and MIoU.
Smart Images
Figure CN121053147B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical fields of computer vision and artificial intelligence, and particularly relates to a medical image segmentation system based on multi-architecture fusion and diffusion models. Background Art
[0002] Medical image segmentation is a key technology in precision medicine, whose goal is to accurately identify and outline the boundaries of pathological structures (such as tumors, polyps) or organs from medical images, providing a basis for quantitative analysis of diseases, surgical planning, and efficacy evaluation.
[0003] Existing deep learning segmentation methods mainly rely on a single network architecture. For example, the method based on convolutional neural network (CNN) represented by U-Net, through the encoder-decoder structure and skip connections, performs well in capturing local texture and detail features. However, due to the inherent locality of convolutional operations, pure CNN models have limitations in capturing the global context and long-range dependencies of images, especially in dealing with large-size targets or complex scenarios that require cross-region information association (such as multi-organ segmentation).
[0004] On the other hand, the Vision Transformer (ViT) architecture uses the self-attention mechanism and can effectively model global context information, showing great potential in natural image segmentation tasks. However, when applied to high-resolution medical images, it has problems of high computational complexity and low efficiency, and is less sensitive to local detail features (such as texture, edges) than CNN.
[0005] In order to combine the advantages of both, some hybrid CNN-Transformer architectures have emerged. However, these hybrid models still face challenges in the feature fusion stage. For example, the simple skip connections (such as concatenation or addition) in U-Net directly fuse the shallow detail features and deep semantic features of the encoder, which may lead to semantic conflicts and noise interference. Especially in medical images with blurred object boundaries, this direct fusion will reduce the segmentation accuracy.
[0006] In addition, the output results of existing segmentation models often have problems such as blurred boundaries, internal holes, or small artifacts, and lack an effective post-processing mechanism to refine and correct the segmentation masks.
[0007] Therefore, how to design a medical image segmentation method that can efficiently fuse local and global information, dynamically adapt to different-scale targets, and has the ability of refined post-processing is a difficult problem亟待解决 in the current technical field. Summary of the Invention
[0008] Objective of the Invention: The objective of the present invention is to overcome the deficiencies of the prior art and provide a medical image segmentation system based on a multi-architecture fusion and diffusion model, aiming to solve the limitations of a single network architecture in long-range dependence modeling and local detail capture, optimize the multi-scale feature fusion mechanism, and introduce a post-processing module to enhance the fineness and robustness of the segmentation results.
[0009] Technical Solution: A medical image segmentation system based on a multi-architecture fusion and diffusion model according to the present invention includes a dual-channel encoding module, a cross-scale feature fusion module, a decoding module, and a diffusion model post-processing module; the dual-channel encoding module uses a dual-channel encoder to perform multi-scale feature extraction on the input medical image; the dual-channel encoder includes multiple scale levels, and each level includes a parallel first channel and a second channel; the cross-scale feature fusion module uses a cross-scale feature fusion module based on frequency domain information to replace the traditional skip connection, enhance and fuse the multi-scale features output by the encoder; the decoding module receives the enhanced features through a decoder and generates a preliminary segmentation mask; the diffusion model post-processing module uses the preliminary segmentation mask as a noisy input and performs iterative denoising and optimization using a pre-trained denoising diffusion probability model.
[0010] Further, the first channel uses standard convolutional layers and downsampling operations to extract local textures and basic features of the image.
[0011] Further, the second channel uses a deformable convolutional module DCDA guided by dynamic attention to extract features; the module DCDA dynamically generates the offset of sampling points according to the input features, enabling the receptive field to adaptively focus on key regions and capture non-rigid deformations and long-range dependence relationships.
[0012] Further, the specific implementation process of the cross-scale feature fusion module is as follows:
[0013] For the shallow feature maps output by the encoder, their channels are divided into a high-frequency branch and a low-frequency branch; the high-frequency branch uses dynamic filter pooling and depthwise separable convolutions to enhance local details; the low-frequency branch uses a multi-head dilated window attention mechanism to model long-range semantic information in a sparse sampling manner;
[0014] Integrate the shallow features processed by high and low frequencies with the original deep feature maps, and use a global multi-head self-attention mechanism to dynamically establish associations and assign weights between features at different scales, generating enhanced features with sufficient information interaction as the input to the decoder.
[0015] Further, the decoder restores the feature maps to the original resolution through progressive upsampling and convolutional operations, generating a preliminary segmentation mask.
[0016] Furthermore, the diffusion model post - processing module is a pre - trained denoising neural network conditioned on the original medical image, denoted as ε θ ; it is based on a forward process of gradually adding Gaussian noise to a clean target segmentation mask So; this process lasts for T steps. At each step t, noise is added according to a predefined variance sequence βt, and its single - step transformation process is described by the following probability distribution:
[0017]
[0018] where S t is the noisy mask at the t - th step, N represents the normal distribution, and I is the identity matrix;
[0019] Train the neural network ε θ to learn the inverse process of the above - mentioned noise - adding process, that is, starting from a pure - noise image S T and gradually removing the noise to finally recover the clean segmentation mask So; the single - step transformation of this reverse process is expressed as:
[0020] p θ (S t-1 |S t ) = N(S t-1 ; μ θ (S t ,t),∑ θ (S t ,t))
[0021] where μ θ and ∑ θ are the mean and variance predicted by the neural network ε θ .
[0022] Furthermore, the training of the denoising neural network ε θ is conditional, that is, the network ε θ not only receives the current noisy mask S t and the time step t as inputs, but also receives the original medical image I as conditional information; the network ε θ is trained to predict the noise ε added to the mask at the t - th step; the objective function of the training is to minimize the difference between the predicted noise and the true noise, which is:
[0023]
[0024] where E represents feature embedding, ε is the true noise sampled from the standard normal distribution; through training, the network ε θ can use the context information of the original image to guide the prediction and removal of noise.
[0025] Furthermore, the iterative denoising and optimization process using a pre-trained denoising diffusion probabilistic model is as follows:
[0026] S1: Consider the preliminary segmentation mask S c as the initial input of this module, start the reverse denoising process, apply noise to S c for T steps to obtain the noisy feature S T ;
[0027] S2: Starting from time step t = T, iterate reversely to t = 1; at each time step t, perform the following operations:<3. Higher segmentation fineness: Innovatively introduce a diffusion model as a post-processing module to iteratively optimize the preliminary segmentation results, which can effectively remove noise, fill holes, and smooth boundaries, generating a final segmentation result with higher quality and more conforming to anatomical structures.
[0037] 4. Excellent performance and generalization: Incorporating the advantages of three architectures, namely CNN, Transformer, and diffusion model, the experimental results on multiple public datasets show that the method of the present invention outperforms existing mainstream methods in key metrics such as Dice coefficient and MIoU, and demonstrates good robustness and generalization ability for medical image segmentation tasks with different modalities and different targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 is a schematic diagram of the overall architecture of the present invention;
[0039] Figure 2 is a schematic diagram of the working principle of the deformable convolutional (DCDA) guided by dynamic attention in the present invention;
[0040] Figure 3 is a schematic diagram of the structure of the high-low frequency domain perception module (HLFPM) in the present invention;
[0041] Figure 4 is a schematic diagram of the working principle of the multi-head attention mechanism (MHSA) in the present invention;
[0042] Figure 5 is a schematic diagram of the working principle of the multi-head dilated window attention (MDWA) in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0043] The present invention will be further described in detail below with reference to the accompanying drawings:
[0044] The present invention proposes a medical image segmentation system based on multi-architecture fusion and diffusion model, and its overall architecture is as Figure 1 shown, mainly including a dual-channel feature encoding module, a cross-scale feature fusion module, a decoding and preliminary segmentation module, and a diffusion model refinement post-processing module.
[0045] As Figure 1 shown, the left area thereof shows the dual-channel feature encoding module of the system. The system receives a medical image and performs multi-scale feature extraction on the medical image through a dual-channel encoder to generate multi-scale feature maps. The dual-channel encoder contains multiple scale levels, and each level includes:
[0046] The first channel 101 uses a standard 3x3 convolutional layer for feature extraction, and then performs downsampling through 2x2 max pooling to reduce the spatial resolution and increase the receptive field.
[0047] The second channel 102 uses the dynamically deformable convolution (DCDA) module designed in the present invention. As Figure 2 shown, for an input feature map, the DCDA module first predicts a two-dimensional offset (Δp) for each convolutional kernel sampling point through a lightweight offset prediction network 201. According to the output offset, the dynamic attention 202 samples features at the positions of the deformation points of the input feature map to obtain keys and values, making the feature map pay more attention to the important regions in the image. Subsequently, the convolution operation samples and aggregates at these sparse positions with offsets, thereby achieving adaptability to the target shape. This design is more flexible than the standard convolution and has a lower computational cost than the global self-attention.
[0048] At the end of the last scale, the feature maps extracted from the first channel and the second channel are concatenated or added in the channel dimension to form the fused feature output 103 at this scale, and are passed to the decoder module and the cross-scale feature fusion module.
[0049] The cross-scale feature fusion uses a cross-scale feature fusion module based on frequency domain information to replace the traditional skip connection and enhance and fuse the multi-scale features output by the encoder.
[0050] The four different-scale feature maps (X1, X2, X3, X4) output by the encoder enter the cross-scale feature fusion module 104. The core of this module is to refine the shallow features X1 and X2 to better fuse with the deep features X3 and X4.
[0051] For the feature maps X1 and X2, they first enter the high-low frequency domain perception module (HLFPM) 301 as Figure 3 shown. In this module, the feature channels are divided into a high-frequency branch 302 and a low-frequency branch 303.
[0052] In the high-frequency branch 302, operations such as depthwise separable convolution are used to focus on extracting and retaining high-frequency detail information such as edges and textures of the image.
[0053] In the low-frequency branch 303, a multi-head dilated window attention (MDWA) module is used. As Figure 5As shown, MDWA performs sparse sampling within a fixed-size window around the query patch at different dilation rates (r = 1, 2, 3...), and then calculates self-attention. This method can capture long-range dependencies while avoiding the high computational cost of global attention. The processed high- and low-frequency features are re-fused to obtain enhanced shallow features X1' and X2'.
[0054] Finally, the features of all four scales (X1', X2', X3, X4) are flattened into a sequence and fed into a standard multi-head self-attention (MHSA) layer as shown in Figure 4 This layer calculates the dependencies between features of all scales, enables global information interaction, and the finally output features will be used as dynamic, content-aware skip connections and passed to the decoder.
[0055] As Figure 1 shown, the right region shows the decoding module of the system. The decoding module inputs the enhanced features processed by the cross-scale feature fusion module into a decoder network. The decoder restores the feature map to the original resolution through successive upsampling and convolution operations, generating a preliminary segmentation mask.
[0056] The diffusion model refinement post-processing module 105 introduces a post-processing module based on the denoising diffusion probabilistic model (DDPM) after the decoder generates the preliminary segmentation result, which is used to refine and correct the preliminary segmentation mask. The core of this post-processing module is a pre-trained denoising neural network conditioned on the original medical image, denoted as ε θ . Its working principle is based on a symmetric noise addition-denoising process.
[0057] The denoising diffusion probabilistic model is based on a forward process of gradually adding Gaussian noise to a clean target segmentation mask So. This process lasts for T steps, and at each step t, noise is added according to a predefined variance sequence βt. Its single-step conversion process can be described by the following probability distribution:
[0058]
[0059] where S t is the noisy mask at the t-th step, N represents the normal distribution, and I is the identity matrix.
[0060] The goal of the post-processing module in this invention is to train a neural network ε θ to learn the inverse process of the above noise addition process, that is, starting from a pure noise image S T , gradually removing the noise and finally restoring the clean segmentation mask So. The single-step conversion of this reverse process can be expressed as:
[0061] pθ (S t-1 |S t ) = N(S t-1 ; μ θ (S t , t), ∑ θ (S t , t))
[0062] where μ θ and ∑ θ are the mean and variance predicted by the neural network ε θ .
[0063] To achieve precise segmentation result correction, the denoising neural network ε θ is trained conditionally. Specifically, the network ε θ not only receives the current noisy mask S t and the time step t as inputs, but also receives the original medical image I as conditional information. The network is trained to predict the noise ε added to the mask at the t-th step. The objective function of the training is to minimize the difference between the predicted noise and the true noise, which can be expressed as:
[0064]
[0065] where E represents the feature embedding, and ε is the true noise sampled from the standard normal distribution. Through this training, the network ε θ learns to use the context information of the original image to guide the prediction and removal of noise.
[0066] In the inference stage, when the decoder generates a preliminary segmentation mask (denoted as the desired clean result S c ), the post-processing module performs the following iterative optimization steps:
[0067] a) Consider the preliminary segmentation mask S c as the initial input of the module. To initiate the reverse denoising process, first apply noise to S c for T steps to obtain the noisy feature S T . In a preferred embodiment, T is a preset total number of iterative steps.
[0068] b) Starting from the time step t = T, iterate backward to t = 1. At each time step t, perform the following operations:
[0069] i. Input the current noisy mask S t and the original medical image I into the trained denoising neural network ε θ to predict the noise ε θ (S t , I, t).
[0070] ii. Based on the predicted noise, calculate the previous, cleaner mask S using the following formula t-1 :
[0071]
[0072] where is the cumulative product of α t .
[0073] c) Repeat step b) until t = 1 to obtain the final denoised result So, which is the final segmentation mask after refinement.
[0074] Through the above iterative denoising process, the present invention can effectively utilize the rich details and structural information of the original image to correct the boundary blurring, internal holes, and small artifacts in the preliminary segmentation result, thereby significantly improving the accuracy and robustness of the final segmentation result.
[0075] To verify the effectiveness of the present invention, the method of the present invention was compared with various existing technology models in terms of performance on the publicly available Kvasir-SEG and Synapse datasets. The evaluation metrics include Dice similarity coefficient (Dice), precision (Precision), recall (Recall), and mean intersection over union (MIoU); the comparison results are shown in Tables 1 and 2.
[0076] Table 1 Performance comparison results on the Kvasir-SEG dataset
[0077]
[0078] [[ID=3 (重复)]] t-1 [[ID=4 (重复)]]:
[0079] Table 2 Performance comparison results on the Synapse dataset
[0080] [[ID=3 (重复)]] t-1 [[ID=4 (重复)]]:
[0081] [[ID=4 (重复)]]: As shown in Table 1, in the Kvasir-SEG polyp segmentation task, the present invention achieves the best performance in the four key metrics of Dice similarity coefficient, precision, recall, and mean intersection over union (MIoU). Specifically, its Dice coefficient (94.67%) is 5.6 percentage points higher than that of the classic UNet (89.07%), and the MIoU (89.86%) is even increased by 9.55 percentage points. Even compared with the sub-optimal advanced hybrid model UCTransNet (Dice 93.26%), it still maintains a significant advantage of 1.41 percentage points.
[0082] As shown in Table 2, in the more complex Synapse multi-organ segmentation task, the Dice coefficient (90.46%) and MIoU (83.45%) of the present invention reached the optimal level again. Compared with the models with sub-optimal performance in this task, such as Deeplabv3-plus (Dice 88.67%) and FusionUnet (MIoU 79.81%), the present invention achieved significant performance improvements of 1.79 and 3.64 percentage points respectively.
[0083] The above experimental results across datasets objectively prove that the multi-architecture fusion scheme proposed by the present invention, by organically combining dual-channel encoding, frequency-domain-based feature fusion, and diffusion model post-processing, shows excellent accuracy and strong generalization ability for both polyp segmentation with relatively simple morphology and multi-organ segmentation with complex structures and numerous categories. This fully demonstrates the creativity, advancement of the present invention and its great application value in various medical image segmentation scenarios.
[0084] It should be understood that the above examples and descriptions in the specification are only the principles, main features and advantages of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the protection scope of the present invention.
Claims
1. A medical image segmentation system based on a multi-architecture fusion and diffusion model, characterized in that, The system includes a dual-channel encoding module, a cross-scale feature fusion module, a decoding module, and a diffusion model post-processing module. The dual-channel encoding module uses a dual-channel encoder to extract multi-scale features from the input medical image. The dual-channel encoder includes multiple scale levels, each level comprising a parallel first channel and a second channel. The cross-scale feature fusion module, based on frequency domain information, replaces traditional skip connections to enhance and fuse the multi-scale features output by the encoder. The decoding module receives the enhanced features through a decoder and generates a preliminary segmentation mask. The diffusion model post-processing module uses the preliminary segmentation mask as a noisy input and performs iterative denoising and optimization using a pre-trained denoising diffusion probability model. The specific implementation process of the cross-scale feature fusion module is as follows: For the shallow feature map output by the encoder, its channels are divided into high-frequency branches and low-frequency branches; the high-frequency branches use dynamic filtering pooling and depthwise separable convolution to enhance local details; the low-frequency branches use a multi-head dilated window attention mechanism to model long-range semantic information in a sparse sampling manner. The shallow features processed by high and low frequencies are integrated with the original deep feature map. A global multi-head self-attention mechanism is used to dynamically establish associations and assign weights between features at different scales, generating enhanced features with sufficient information exchange, which serve as the input to the decoder. The diffusion model post-processing module is a pre-trained denoising neural network conditioned on the original medical image, denoised as follows: The noisy diffusion stage of the diffusion model is based on a positive, progressive approach towards a clean target segmentation mask. The process of adding Gaussian noise; this process continues Step by step Based on a predefined variance sequence With noise added, the single-step conversion process is described by the following probability distribution: in, It is in the The noisy mask of the step, Represents a normal distribution. It is the identity matrix; Denoising Neural Network Learning the inverse process of adding noise, that is, from a purely noisy image Initially, noise is gradually removed, eventually resulting in a clean segmentation mask. The single-step transformation of this inverse process is represented as: in, and It is made by neural networks The predicted mean and variance; The denoising neural network The training is conditional, meaning the network... Not only receive the current noisy mask and time step It also receives raw medical images as input. As conditional information; network Trained to predict the first Noise added to the mask The objective function for training is to minimize the difference between the predicted noise and the actual noise, as follows: in, , Represents feature embedding, It is real noise sampled from a standard normal distribution; through training, the network... It can utilize the contextual information of the original image to guide noise prediction and removal; The iterative denoising and optimization process using a pre-trained denoising diffusion probability model is as follows: S1: Initial segmentation mask This is taken as the initial input to the model, initiating the inverse denoising process. Applying T steps of noise yields noisy features. ; S2: From time step Start by iterating backwards to... ; at each time step Perform the following operations: change the current noisy mask. and original medical images Input into the trained denoising neural network In the process, the noise of the current step is predicted. Based on the predicted noise, calculate the mask from the previous step using the following formula. : in, yes The cumulative product; S3: Repeat step S2 until... This yields the final denoising result, which is the final segmentation mask after refined processing. .
2. The medical image segmentation system based on a multi-architecture fusion and diffusion model according to claim 1, characterized in that, The first channel uses standard convolutional layers and downsampling operations to extract local textures and basic features of the image.
3. The medical image segmentation system based on a multi-architecture fusion and diffusion model according to claim 1, characterized in that, The second channel uses a deformable convolutional module DCDA based on dynamic attention guidance for feature extraction; the DCDA module dynamically generates the offset of the sampling points according to the input features, so that the receptive field adaptively focuses on the key region and captures non-rigid deformation and long-range dependencies.
4. The medical image segmentation system based on a multi-architecture fusion and diffusion model according to claim 1, characterized in that, The decoder restores the feature map to its original resolution through progressive upsampling and convolution operations, generating a preliminary segmentation mask.