Evidence learning based on information bottleneck and sam3 for trustworthy medical image segmentation
By combining the SAM3 encoder with the information bottleneck evidence learning method of the U-shaped structure of medical images, efficient multi-level feature fusion and uncertainty quantification are achieved in medical image segmentation tasks. This solves the problems of insufficient generalization ability and credibility of the SAM large model in medical image segmentation tasks, improves segmentation accuracy and credibility, and is suitable for high-risk clinical applications.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-05-14
- Publication Date
- 2026-08-04
AI Technical Summary
Existing large SAM models suffer from insufficient generalization ability, high computational cost, low segmentation accuracy, and insufficient reliability in medical image segmentation tasks. In particular, they perform poorly in segmenting small target areas and complex lesion areas. Furthermore, existing uncertainty quantification methods are computationally expensive or overconfident, making it difficult to meet the needs of high-risk clinical applications.
Combining the SAM3 encoder with the U-shaped structure of medical images, this paper employs an evidence-based learning method with information bottlenecks to perform multi-level feature fusion and uncertainty estimation. The HCF module is used to achieve cross-layer fusion of deep semantic features and shallow spatial detail features. The dual-head output structure completes segmentation prediction and uncertainty quantification in a single forward propagation. Information bottleneck constraint regularization is introduced to improve the reliability of the model.
It significantly improves the model's ability to perceive the small boundaries of lesions, reduces computational overhead, provides more accurate and robust uncertainty estimation, improves segmentation accuracy and reliability, and is suitable for high-risk medical application scenarios.
Smart Images
Figure CN122223340B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image segmentation technology, specifically to a reliable medical image segmentation method based on evidence learning and SAM3, which addresses information bottlenecks. Background Technology
[0002] Current mainstream medical image segmentation methods are typically deep learning-based methods using convolutional neural networks or U-shaped encoder-decoder structures. These methods utilize trained segmentation models to identify target regions in medical images and segment the corresponding organs, tissues, lesions, or anatomical structures. While these methods enable end-to-end learning and generally exhibit good segmentation performance under specific organ, modality, or data distribution conditions, their performance tends to degrade and their generalization ability is limited when test data distribution shifts, imaging equipment changes, or lesion morphology varies significantly.
[0003] In recent years, with the development of the SAM series models, some methods have attempted to leverage the general representation capabilities learned by large-scale pre-trained models on massive datasets and transfer them to medical image segmentation tasks to improve the model's generalization ability in medical imaging scenarios. Among them, the SAM3 model was the first to introduce prompt-based concept segmentation and was pre-trained on the multi-class dataset SA-co, demonstrating excellent performance in downstream segmentation tasks and showing potential for transfer to the medical imaging field. These methods typically employ efficient parameter fine-tuning techniques such as LoRA and Adapter to adapt the model and introduce automatic prompting modules to reduce the burden of manual interaction and manual annotation by doctors.
[0004] However, while existing large-scale SAM model transfer methods possess strong general representation capabilities, their pre-training data primarily originates from natural images, which differ significantly from medical images in terms of imaging mechanisms, tissue structures, texture patterns, and boundary features. Therefore, when transferred to specific medical image segmentation tasks, they still tend to suffer from insufficient segmentation accuracy for small target regions, weak boundary regions, and complex lesion regions. Furthermore, since the original structure of the large-scale SAM model was not specifically designed for medical scenarios, its segmentation capabilities in challenging scenarios such as weak boundaries and small lesions remain limited. In addition, the internal decision-making mechanism of the large model is relatively complex, resulting in insufficient interpretability and reliability of prediction results, often requiring further manual verification in high-risk medical applications.
[0005] To improve the segmentation performance of SAM models in medical images, some studies have attempted to concatenate SAM-like pre-trained encoders with a basic U-Net structure. However, limited by the decoding method of the basic U-shaped structure, this simple structural concatenation struggles to bridge the semantic gap between pre-trained features from natural images and low-level features from medical images. Existing methods still fall short in deep fusion between features at different levels, particularly in their limited ability to effectively transfer deep global semantic information to the shallow decoding stage, resulting in room for improvement in segmentation performance in complex boundary regions and fine-grained structural regions.
[0006] To further enhance the reliability of model predictions, existing methods typically quantify the uncertainty of segmentation results to improve the model's usability in high-risk medical scenarios. Common uncertainty quantification methods include Monte Carlo Dropout, ensemble networks, and evidence-based deep learning. However, existing uncertainty quantification methods still suffer from the following problems: (1) Uncertainty estimation methods based on Monte Carlo Dropout and ensemble networks usually require multiple forward propagations, resulting in long inference times and high computational costs, making it difficult to balance the accuracy of uncertainty estimation with the real-time requirements of clinical practice. (2) Although the uncertainty modeling method based on evidence deep learning can output prediction results and uncertainty information in a single forward propagation, the existing methods often fail to effectively limit redundant information and domain noise when extracting features, resulting in problems such as excessive concentration of target category evidence, overconfidence, and insufficient calibration performance, which in turn affects the credibility of the segmentation results. (3) Existing uncertainty quantification methods often focus more on the output of the uncertainty result itself, while not giving enough consideration to how uncertainty information can guide the optimization of the segmentation model. At the same time, there is a lack of effective identification and constraint of task-related and irrelevant information in the feature representation, which makes it difficult to achieve effective coordination between segmentation performance and credibility.
[0007] In summary, existing SAM large model transfer methods still struggle to balance generalization ability, computational cost, segmentation accuracy, and result reliability in medical image segmentation tasks. In particular, the lack of effective constraints on redundant features and the absence of uncertainty-based back guidance result in limited generalization ability and segmentation accuracy of commonly used segmentation models, making it difficult to meet the reliability requirements of high-risk clinical medical scenarios. Summary of the Invention
[0008] This invention provides a reliable medical image segmentation method based on information bottleneck-based evidence learning and SAM3. By combining the generalization ability of SAM3 with the U-shaped structure of medical images, the segmentation accuracy is improved. At the same time, an evidence-based deep learning method based on information bottleneck is introduced to enable the model to have uncertainty perception capabilities, thereby adapting it to high-risk application scenarios such as medical imaging and solving the problems in the background technology.
[0009] To achieve the above objectives, the technical solution of the present invention is as follows: A reliable medical image segmentation method based on information bottleneck-driven evidence learning and SAM3 performs the following steps using a computer device: S1. Acquire medical image data and preprocess the medical image data; S2. Input the preprocessed medical image into the SAM3 encoder containing the Adapter module, fine-tune the parameters under the condition of freezing the SAM3 encoder backbone parameters, and extract multi-level features. S3. Using the HCF multi-scale feature fusion module, the deep semantic features and shallow spatial detail features in the multi-level features are fused across layers to obtain fused features. S4. Input the fused features into the decoder for step-by-step reconstruction, and obtain the mean and variance of the pre-evidence through a dual-head output structure; S5. Based on the pre-evidence mean and variance, construct the evidence distribution and Dirichlet parameters through the uncertainty estimation module, and perform joint loss optimization to finally obtain the medical image segmentation result and uncertainty estimation result; S6. Based on the segmentation results and uncertainty estimation results obtained in S5, the segmentation accuracy and confidence level of the model are measured by the distribution of mDice, mIoU, ECE, and NLL indices.
[0010] Preferably, the specific steps of S2 include: S21. Obtain the training data of the two-dimensional medical images to be processed from the preprocessed medical images, and construct the corresponding input features; S22. Input the input features into the Transformer module of the SAM3 encoder for feature encoding; wherein, the Transformer module includes an Adapter module, a layer normalization module, a multi-head self-attention module, and a multilayer perceptron module; The input features are sequentially processed by the Adapter module, including feature transformation and residual connection, layer normalization, multi-head self-attention processing and residual connection, further layer normalization, and multilayer perceptron processing, to output the transformed feature representation; wherein, the input features of the current layer are denoted as... The output features are denoted as ; The Adapter module adopts a bottleneck structure, including a downsampling linear layer, a GELU activation function, an upsampling linear layer, and residual connections; The feature dimension is Where B represents the batch size. and These represent the height and width of the feature map, respectively. The SAM3 encoder includes 32 Transformer modules. S23. During the encoding process of the 32-layer Transformer module, four levels of multi-level features are extracted from layers 8, 16, 24, and 32, respectively. These features include shallow detail features for representing local textures and deep semantic features for representing global context. The features are denoted from shallow to deep as follows: .
[0011] Preferably, the specific steps of S3 include: S31, the shallow detail features extracted in S23 Respectively with deep semantic features Input is fed into the HCF module for cross-level feature fusion; for any shallow detail feature ,in The shallow detail features With the deep semantic features By splicing along the channel dimension, the splicing feature is obtained; S32. The stitched features are sequentially input into a 1×1 convolutional layer, an interpolation upsampling layer, and a 3×3 convolutional layer to achieve channel adjustment, spatial scale alignment, and local context fusion, generating corresponding multi-scale context enhancement features. The processing procedure is as follows:
[0012] in, This indicates a splicing operation along the channel dimension. This represents a 1×1 convolution operation. This indicates a 3×3 convolution operation, and the 3×3 convolution operation does not change the feature map size; S33. For deep semantic features Separate interpolation upsampling is performed to obtain spatial scale features that match the multi-scale context enhancement features. The multi-scale context enhancement features and the upsampled deep semantic features are then sent to the decoder for subsequent segmentation.
[0013] Preferably, the specific steps of S4 include: S41. The decoder is composed of the following structure: The input features are first transformed by a 1×1 convolutional layer, and then split into several groups along the channel dimension by a split operation. Each group of features is extracted by a depthwise separable convolution, and then all grouped features are concatenated and fused. Finally, a 1×1 convolutional layer is used to restore the number of channels, and the decoded feature map is output. S42, Decoder Receive Spatial resolution is gradually restored through upsampling and convolution operations, outputting pixel-level feature maps. Where C is the number of categories, , and Pixel-level feature mapping The height, width, and number of categories of the feature map; S43. The decoder output is a dual-head output structure, receiving the pixel-level feature mapping. The pre-evidence mean was obtained. Pre-evidence variance ,use Activation function to ensure ,in, It is a mapping function for outputting the mean. , It is a mapping function that outputs the unactivated variance.
[0014] Preferably, the specific steps of S5 include: S51, Based on the pre-evidence mean variance of pre-evidence Pre-evidence Establish a Gaussian random distribution, expressed as:
[0015] in, For the input medical images, Pre-evidence The probability distribution, Indicated by Each component is a diagonal covariance matrix composed of diagonal elements. The modeling method transforms the model output from a deterministic logits distribution into a probability distribution, enabling the model to explicitly express the uncertainty of pre-evidence. S52. To ensure that the gradient can be backpropagated, a reparameterization technique is used to sample the pre-evidence, as shown below:
[0016] in, For sampling index, For the number of samples, This represents the pre-evidence obtained from the k-th sampling. Standard Gaussian noise, For the first The standard Gaussian noise obtained from the second sampling Represents element-wise product. It is the identity matrix; during the training phase, for The loss obtained from the second sampling is averaged and optimized; during the inference phase, the loss is... The pre-evidence obtained from the second sampling is taken as the mean, and is expressed as:
[0017] S53, the pre-evidence pass Function mapping as nonnegative evidence , is represented as:
[0018] in, This represents the amount of evidence for each category of each pixel. It always holds true; 'c' represents the index of the category, and the amount of evidence. The current pixel is the first pixel. The support strength of a category; a higher value indicates that the model is more certain in its judgment of that category. S54, Based on the aforementioned nonnegative evidence Constructing Dirichlet distribution parameters , is represented as:
[0019] in, For the first The Dirichlet parameter corresponding to the category is incremented by 1 to ensure... The parameter constraints satisfy the Dirichlet distribution; the corresponding Dirichlet distribution is expressed as:
[0020] in, Let T be the class probability vector, and let T denote the transpose of the matrix. The sum of Dirichlet parameters. It is the Gamma function; S55, Based on the Dirichlet distribution parameters Calculate the predicted probabilities for each category, taking the expected value of the Dirichlet distribution, expressed as:
[0021] in, The current pixel belongs to the first... Predicted probability of the category The sum of Dirichlet parameters for all categories; Final segmentation result for: ; Uncertainty estimation results for: ; S56. Introduce information bottleneck regularization constraints into the pre-evidence space to affect the pre-evidence probability distribution. With standard Gaussian prior To minimize the KL divergence between them, the information bottleneck constraint term is expressed as:
[0022] in, This is an information bottleneck constraint. This represents the expectation operation. Let KL divergence be the denoted KL divergence. Analytical expansion of the above KL divergence yields:
[0023] in, These are the trainable parameters of the model; S57, to The pre-evidence obtained from the sub-sampling is used to calculate the MSE Bayesian risk loss based on the Dirichlet distribution, and the mean is taken as the segmentation supervision loss, expressed as:
[0024] in, Let b be the segmentation supervision loss, i be the spatial index in the height direction (i = 1, 2, ..., H), and j be the spatial index in the width direction (j = 1, 2, ..., W). Let be the label for the b-th batch, the c-th category, and the position (i,j) pixel. For the b-th batch, c-th category, and pixel (i,j); This represents the sum of all category Dirichlet parameters at pixel (i,j) in the b-th batch. S58. Jointly optimize the segmentation supervision loss and the information bottleneck constraint term to construct the overall loss function of the uncertainty estimation module, expressed as:
[0025] Where L is the overall loss function of the uncertainty estimation module. The weight hyperparameter is used to constrain information bottlenecks and balance segmentation accuracy with uncertainty and calibration performance.
[0026] Preferably, in S5, the model is trained and optimized by minimizing the overall loss function; the overall loss function It is obtained by weighted summation of segmentation loss, Dirichlet distribution fitting loss, and information bottleneck constraint term, and is expressed as:
[0027] in, This indicates Dice's loss. The weights represent the Dice loss. This represents the binary cross-entropy loss. The weights represent the binary cross-entropy loss. This represents the fitting loss for the Dirichlet distribution. The weights represent the loss for fitting the Dirichlet distribution; The Dice loss and binary cross-entropy loss together constitute the segmentation loss. , is represented as:
[0028] Wherein, the Dice loss Represented as:
[0029] The binary cross-entropy loss Represented as:
[0030] in, Let be the label of the b-th batch, at position (i,j) pixel. Let f(x) represent the probability distribution of the b-th batch at position (i,j) pixel, which follows a Gaussian distribution.
[0031] As can be seen from the above technical solution compared with the prior art, the present invention has the following beneficial effects: This invention introduces the information bottleneck evidence learning mechanism into medical image segmentation tasks by applying regularization constraints to evidence-based deep learning based on information bottleneck theory. Through a dual-head output structure, segmentation prediction and uncertainty quantification are simultaneously achieved in a single forward propagation, effectively overcoming the overconfidence problem caused by excessive evidence concentration in traditional EDL. Simultaneously, the HCF module designed in this invention achieves cross-layer fusion of deep semantic features and shallow spatial detail features, significantly enhancing the model's ability to perceive minute lesion boundaries. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the method steps in an embodiment of the present invention; Figure 2 This is a structural diagram of a SAM3 encoder with an adapter provided in an example of the present invention; Figure 3 This is a structural diagram of the HCF module provided in the example of the present invention. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0034] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention, but should not be used to limit the scope of the present invention.
[0035] This invention provides a reliable medical image segmentation method based on evidence learning and SAM3, such as... Figure 1 As shown, perform the following steps using a computer device: S1. Acquire medical image data and preprocess the medical image data; S2. Input the preprocessed medical image into the SAM3 encoder containing the Adapter module, fine-tune the parameters under the condition of freezing the SAM3 encoder backbone parameters, and extract multi-level features. S3. Using the HCF multi-scale feature fusion module, the deep semantic features and shallow spatial detail features in the multi-level features are fused across layers to obtain fused features. S4. Input the fused features into the decoder for step-by-step reconstruction, and obtain the mean and variance of the pre-evidence through a dual-head output structure; S5. Based on the pre-evidence mean and variance, construct the evidence distribution and Dirichlet parameters through the uncertainty estimation module, and perform joint loss optimization to finally obtain the medical image segmentation result and uncertainty estimation result; S6. Based on the segmentation results and uncertainty estimation results obtained in S5, the segmentation accuracy and confidence level of the model are measured by the distribution of mDice, mIoU, ECE, and NLL indices.
[0036] Example: In this embodiment, experiments were conducted on a publicly available colorectal polyp segmentation dataset. 1450 images from the CVC-ClinicDB and Kvasir datasets were used as training materials. The CVC-ClinicDB dataset (also known as CVC-612) contains 612 images from 25 colonoscopy videos, with an image size of 384×288. 550 of these images were used for training, and 112 were used for testing. The Kvasir dataset contains 1000 polyp images and corresponding annotations; 900 were used for training, and 100 were used for testing. To test the model's generalization ability, three new datasets were used as test sets: CVC-300, which comes from the EndoScene test dataset. The EndoScene dataset includes 912 images from 44 colonoscopy sequences of 36 patients. We used CVC-300, containing 609 images, as a test set. CVC-ColonDB, also from EndoScene, contains 380 images extracted from 15 colonoscopy sequences. The ETIS dataset contains 196 images from 34 colonoscopy videos, with an image size of 1225×966, making it the largest dataset in terms of image size. Furthermore, the polyps in these images are extremely small and difficult to distinguish, making it the most challenging test set.
[0037] Through the above-mentioned "two training sessions + three tests" multi-dataset experimental setup, the comprehensive performance of the model can be fully verified in various clinical scenarios such as routine, cross-device, small target, and high resolution, ensuring that the model has good generalization ability and clinical application potential.
[0038] Implementation begins: S11. First, obtain a two-dimensional medical image segmentation dataset, which includes a training set and a test set. The images contain pixel-level annotations of organs, tissues, or lesion regions. The input image tensor size is H×W, and the number of channels is 3.
[0039] S12, standardize and preprocess the medical image data, including but not limited to: normalizing the image so that the pixel values fall within a suitable range; and performing augmentation operations on the data according to the specific characteristics of the medical image modality, including rotation, scaling, flipping, etc., to increase the diversity of training data and improve the generalization ability of the model.
[0040] S21, Encoder architecture design, such as Figure 2 As shown, the SAM3 encoder with frozen parameters is used as the feature extraction backbone. This encoder is based on the ViT architecture. To achieve effective adaptation to medical imaging tasks, it innovatively incorporates features into each Transformer module of ViT. Figure 2 Insert the Adapter module into the converter module, i.e. Figure 2 The adapter in the process performs efficient parameter fine-tuning; at the same time, during training, the encoder backbone parameters are completely frozen and only the adapter performs efficient parameter fine-tuning to extract four-level multi-level features from local texture to global context, which are output from layers 8, 16, 24 and 32 respectively.
[0041] S22. Input the input features into the Transformer module of the SAM3 encoder for feature encoding; wherein, the Transformer module includes an Adapter module, a layer normalization module, a multi-head self-attention module, and a multilayer perceptron module; The processing order is as follows: the input features are sequentially processed by the Adapter module for feature transformation and residual connection, layer normalization, multi-head self-attention and residual connection, layer normalization again, and multilayer perceptron processing, to output the transformed feature representation; Wherein, the input features of the current layer are denoted as The output features are denoted as ; The Adapter module adopts a bottleneck structure, including a downsampling linear layer, a GELU activation function, an upsampling linear layer, and residual connections; The feature dimension is Where B represents the batch size. and These represent the height and width of the feature map, respectively. The SAM3 encoder includes 32 Transformer modules. The specific processing order of each Transformer module is as follows:
[0042] in: Input features for the current layer, with dimension . ; The adapter for the bottleneck structure consists of a downsampled linear layer, a GELU activation function, an upsampled linear layer, and residual connections. For layer normalization; This is a multi-head self-attention mechanism; This is a multilayer perceptron. The adapter module adopts a bottleneck structure, consisting of a downsampling linear layer, a GELU activation function, an upsampling linear layer, and a final GELU activation function in sequence. Let the input feature dimension be... The intermediate bottleneck dimension is The forward computation process of the Adapter is shown in the following equation:
[0043] in, This is the downsampling projection matrix. The model uses an upsampled projection matrix, and residual connections ensure that the model's behavior remains consistent with the original SAM3 encoder during the initial training phase. During training, the original parameters of the SAM3 encoder are frozen, with only the parameters of the Adapter module participating in gradient updates, thus achieving efficient parameter fine-tuning.
[0044] S23. During the encoding process of the 32-layer Transformer module, four levels of multi-level features are extracted from layers 8, 16, 24, and 32, respectively. These features include shallow detail features for representing local textures and deep semantic features for representing global context. The features are denoted from shallow to deep as follows: .
[0045] Shallow features contain more spatial detail and low-level texture features, which help to accurately depict boundaries and microstructures in medical images; deep features contain richer semantic information and contextual understanding, which help to improve the understanding of the overall lesion morphology.
[0046] The specific steps and procedures for S3 include: S31, the shallow detail features extracted in S23 Respectively with deep semantic features Input is fed into the HCF module for cross-level feature fusion; for any shallow detail feature ,in The shallow detail features With the deep semantic features By splicing along the channel dimension, the splicing feature is obtained; like Figure 3 As shown, the HCF module adopts a hierarchical 1-to-3 design, comparing features at different depths and achieving cross-level feature fusion by sampling to an appropriate feature resolution. First, a 1×1 convolution (Conv1×1) is used to perform channel dimensionality reduction and recalibration on the concatenated deep semantic features and shallow spatial details to reduce computation and enhance cross-channel information interaction. Next, a regular convolution (Conv, default 3×3) is used to extract local spatial details, preserving edge and texture information. Finally, a 3×3 convolution (Conv3×3) is used to capture medium-scale contextual semantics. These three convolutions are processed collaboratively in parallel or sequentially, outputting fused features rich in multi-granularity information, providing a feature representation that balances spatial resolution and semantic expressiveness for subsequent decoder-level reconstruction.
[0047] S32. The stitched features are sequentially input into a 1×1 convolutional layer, an interpolation upsampling layer, and a 3×3 convolutional layer to achieve channel adjustment, spatial scale alignment, and local context fusion, generating corresponding multi-scale context enhancement features. The processing procedure is as follows:
[0048] in, This indicates a splicing operation along the channel dimension. This represents a 1×1 convolution operation. This indicates a 3×3 convolution operation, and the 3×3 convolution operation does not change the feature map size; S33. For deep semantic features Separate interpolation upsampling is performed to obtain spatial scale features that match the multi-scale context enhancement features. The multi-scale context enhancement features and the upsampled deep semantic features are then sent to the decoder for subsequent segmentation.
[0049] The specific steps and procedures for S4 include: S41. The decoder consists of the following structure: The input features are first transformed by a 1×1 convolutional layer, and then split into several groups along the channel dimension by a split operation. Each group of features is extracted by a depthwise separable convolution, and then all grouped features are concatenated and fused. Finally, a 1×1 convolutional layer is used to restore the number of channels, and the decoded feature map is output. S42, Decoder Receive Spatial resolution is gradually restored through upsampling and convolution operations, outputting pixel-level feature maps. Where C is the number of categories, , and Pixel-level feature mapping The height, width, and number of categories of the feature map; S43. The decoder output is a dual-head output structure, receiving the pixel-level feature mapping. The pre-evidence mean was obtained. Pre-evidence variance ,use Activation function to ensure ,in, It is a mapping function for outputting the mean. , It is a mapping function that outputs the unactivated variance.
[0050] The specific steps and procedures of S5 include: S51, Based on the pre-evidence mean variance of pre-evidence Pre-evidence Establish a Gaussian random distribution, expressed as:
[0051] in, For the input medical images, Pre-evidence The probability distribution, Indicated by Each component is a diagonal covariance matrix composed of diagonal elements. The modeling method transforms the model output from a deterministic logits distribution into a probability distribution, enabling the model to explicitly express the uncertainty of pre-evidence. S52. To ensure that the gradient can be backpropagated, a reparameterization technique is used to sample the pre-evidence, as shown below:
[0052] in, For sampling index, For the number of samples, This represents the pre-evidence obtained from the k-th sampling. Standard Gaussian noise, For the first The standard Gaussian noise obtained from the second sampling Represents element-wise product. It is the identity matrix; during the training phase, for The loss obtained from the second sampling is averaged and optimized; during the inference phase, the loss is... The pre-evidence obtained from the second sampling is taken as the mean, and is expressed as:
[0053] S53, the pre-evidence pass Function mapping as nonnegative evidence , is represented as:
[0054] in, This represents the amount of evidence for each category of each pixel. It always holds true; 'c' represents the index of the category, and the amount of evidence. The current pixel is the first pixel. The support strength of a category; a higher value indicates that the model is more certain in its judgment of that category. S54, Based on the aforementioned nonnegative evidence Constructing Dirichlet distribution parameters , is represented as:
[0055] in, For the first The Dirichlet parameter corresponding to the category is incremented by 1 to ensure... The parameter constraints satisfy the Dirichlet distribution; the corresponding Dirichlet distribution is expressed as:
[0056] in, Let T be the class probability vector, and let T denote the transpose of the matrix. The sum of Dirichlet parameters. It is the Gamma function; S55, Based on the Dirichlet distribution parameters Calculate the predicted probabilities for each category, taking the expected value of the Dirichlet distribution, expressed as:
[0057] in, The current pixel belongs to the first... Predicted probability of the category The sum of Dirichlet parameters for all categories; Final segmentation result for: ; Uncertainty estimation results for: ; S56. Introduce information bottleneck regularization constraints into the pre-evidence space to affect the pre-evidence probability distribution. With standard Gaussian prior To minimize the KL divergence between them, the information bottleneck constraint term is expressed as:
[0058] in, This is an information bottleneck constraint. This represents the expectation operation. Let KL divergence be the denoted KL divergence. Analytical expansion of the above KL divergence yields:
[0059] in, These are the trainable parameters of the model; S57, to The pre-evidence obtained from the sub-sampling is used to calculate the MSE Bayesian risk loss based on the Dirichlet distribution, and the mean is taken as the segmentation supervision loss, expressed as:
[0060] in, Let b be the segmentation supervision loss, i be the spatial index in the height direction (i = 1, 2, ..., H), and j be the spatial index in the width direction (j = 1, 2, ..., W). Let be the label for the b-th batch, the c-th category, and the position (i,j) pixel. For the b-th batch, c-th category, and pixel (i,j); This represents the sum of all category Dirichlet parameters at pixel (i,j) in the b-th batch. S58. Jointly optimize the segmentation supervision loss and the information bottleneck constraint term to construct the overall loss function of the uncertainty estimation module, expressed as:
[0061] Where L is the overall loss function of the uncertainty estimation module. The weight hyperparameter is used to constrain information bottlenecks and balance segmentation accuracy with uncertainty and calibration performance.
[0062] In S5, the model is trained and optimized by minimizing the overall loss function; the overall loss function It is obtained by weighted summation of segmentation loss, Dirichlet distribution fitting loss, and information bottleneck constraint term, and is expressed as:
[0063] in, This indicates Dice's loss. The weights represent the Dice loss. This represents the binary cross-entropy loss. The weights represent the binary cross-entropy loss. This represents the fitting loss for the Dirichlet distribution. The weights represent the loss for fitting the Dirichlet distribution; The Dice loss and binary cross-entropy loss together constitute the segmentation loss. , is represented as:
[0064] Wherein, the Dice loss Represented as:
[0065] The binary cross-entropy loss Represented as:
[0066] in, Let be the label of the b-th batch, at position (i,j) pixel. Let f(x) represent the probability distribution of the b-th batch at position (i,j) pixel, which follows a Gaussian distribution.
[0067] By analyzing the mean Apply regularization constraints to constrain the Dirichlet parameters. This prevents the model from generating excessive evidence values, fundamentally alleviating the overconfidence problem of traditional EDL and improving model calibration performance and prediction reliability.
[0068] in, and All values were set to 1. The AdamW optimizer was used during training, and training was completed on a single NVIDIA RTX 4060 GPU. All experiments were uniformly set to 20 training epochs. To stabilize the training process and gradually introduce uncertainty modeling, this invention divides the training of the IE-USAM3 model into three stages: First freeze Head, only training The head is prioritized to learn stable segmentation representations. This stage is trained for 5 epochs by default, with a learning rate of [missing information]. , The initial weight is set to 0.05, and its weight is gradually increased during subsequent training until the preset full weight is reached in the 20th round.
[0069] After completing the initial training of the mean head, thaw. This phase initiates the model's joint learning of the prediction mean and uncertainty information. This phase defaults to 5 training epochs, with a learning rate set to [missing information]. .
[0070] In the final stage, the overall model is further jointly optimized to improve the coordination between segmentation performance and uncertainty estimation. This stage involves 10 training epochs by default, with a learning rate set to [value missing]. .
[0071] S6 includes the following steps: The Intersection over Union (IoU) ratio measures the quality of overlap between predicted segmentation regions of a single class and the ground truth labeled regions. The formula is as follows:
[0072] in, This represents the lesion area marked by the doctor. The lesion region represented by the model segmentation output is used to calculate the average Intersection over Union (mIoU) for all classes in multi-class segmentation tasks.
[0073] in, For the number of categories, For the first Intersection over Union (mIoU) of classes. mIoU takes into account the segmentation quality of each class equally and can comprehensively reflect the overall segmentation performance of the model in multi-class scenarios.
[0074] The Dice coefficient measures the similarity between the predicted and ground regions from the perspective of ensemble overlap, and is more sensitive to the segmentation quality of the foreground region. The calculation formula is as follows: The average Dice coefficient (mDice) is obtained by averaging the Dice coefficients of all categories.
[0075] in, For the first The Dice coefficient is used for segmentation assessment of small target lesion regions. Compared to IoU, the Dice coefficient is more sensitive to segmentation assessment of small target lesion regions and is widely used as a primary evaluation indicator in medical image segmentation tasks.
[0076] Expected calibration error (ECE) measures the deviation between the model's predicted confidence level and its actual accuracy, reflecting the model's calibration performance. The predicted confidence interval is divided into several intervals. The difference between the statistical prediction accuracy and the average confidence level within each interval is calculated using the following formula:
[0077] in, The total number of samples, For the first A sample set within a confidence interval For the first The predicted category for each sample, For the corresponding real category, For the model to the first The maximum class probability predicted for each sample. The lower the ECE value, the better the model's prediction confidence matches the actual accuracy, and the better the model's calibration performance; the higher the ECE value, the more overconfident or overly conservative the model may be.
[0078] Negative log-likelihood (NLL) assesses the model's confidence in the true label from a probability distribution perspective. The calculation formula is as follows:
[0079] in, The total number of samples, The total number of categories, For the first The first sample The true one-hot label of the class, For the model to the first The predicted value for the nth sample is... The NLL value represents the probability of a class. A lower NLL value indicates that the model assigns a higher predicted probability to the true class, and the higher the quality of the model's probability output. NLL penalizes both prediction accuracy and probability calibration, and is an important indicator for evaluating the overall performance of uncertainty-aware models.
[0080] In summary, this invention applies regularization constraints to evidence-based deep learning based on the information bottleneck theory, introduces the IB-EDL mechanism into medical image segmentation tasks, and simultaneously completes segmentation prediction and uncertainty quantification in a single forward propagation through a dual-head output structure, effectively overcoming the overconfidence problem caused by the excessive concentration of evidence in traditional EDL.
[0081] Meanwhile, the HCF module designed in this invention achieves cross-layer fusion of deep semantic features and shallow spatial detail features, significantly enhancing the model's ability to perceive minute lesion boundaries. Compared to Monte Carlo Dropout methods and ensemble learning methods that rely on multiple forward propagations, this invention provides more accurate and robust uncertainty estimation while significantly reducing computational overhead. Compared to traditional evidence-based deep learning methods, this invention achieves significant improvements in both calibration performance and segmentation accuracy, demonstrating strong clinical application value.
[0082] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.
[0083] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
[0084] For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media.
[0085] The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).
[0086] It should be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0087] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0088] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0089] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A reliable medical image segmentation method based on evidence learning and SAM3, characterized in that, Perform the following steps using a computer device: S1. Acquire medical image data and preprocess the medical image data; S2. Input the preprocessed medical image into the SAM3 encoder containing the Adapter module, fine-tune the parameters under the condition of freezing the SAM3 encoder backbone parameters, and extract multi-level features. S3. Using the HCF multi-scale feature fusion module, the deep semantic features and shallow spatial detail features in the multi-level features are fused across layers to obtain fused features. S4. Input the fused features into the decoder for step-by-step reconstruction, and obtain the mean and variance of the pre-evidence through a dual-head output structure; S5. Based on the pre-evidence mean and variance, construct the evidence distribution and Dirichlet parameters through the uncertainty estimation module, and perform joint loss optimization to finally obtain the medical image segmentation result and uncertainty estimation result; S6. Based on the segmentation results and uncertainty estimation results obtained in S5, the segmentation accuracy and confidence of the model are measured by mDice, mIoU, and ECE and NLL index distributions. The specific steps of S2 include: S21. Obtain the training data of the two-dimensional medical images to be processed from the preprocessed medical images, and construct the corresponding input features; S22. Input the input features into the Transformer module of the SAM3 encoder for feature encoding; wherein, the Transformer module includes an Adapter module, a layer normalization module, a multi-head self-attention module, and a multilayer perceptron module; The input features are sequentially processed by the Adapter module, including feature transformation and residual connection, layer normalization, multi-head self-attention processing and residual connection, further layer normalization, and multilayer perceptron processing, to output the transformed feature representation; wherein, the input features of the current layer are denoted as... The output features are denoted as ; The Adapter module adopts a bottleneck structure, including a downsampling linear layer, a GELU activation function, an upsampling linear layer, and residual connections; The feature dimension is Where B represents the batch size. and These represent the height and width of the feature map, respectively. The SAM3 encoder includes 32 Transformer modules. S23. During the encoding process of the 32-layer Transformer module, four levels of multi-level features are extracted from layers 8, 16, 24, and 32, respectively. These features include shallow detail features for representing local textures and deep semantic features for representing global context. The features are denoted from shallow to deep as follows: ; The specific steps of S3 include: S31, the shallow detail features extracted in S23 Respectively with deep semantic features Input is fed into the HCF module for cross-level feature fusion; for any shallow detail feature ,in The shallow detail features With the deep semantic features By splicing along the channel dimension, the splicing feature is obtained; S32. The stitched features are sequentially input into a 1×1 convolutional layer, an interpolation upsampling layer, and a 3×3 convolutional layer to achieve channel adjustment, spatial scale alignment, and local context fusion, generating corresponding multi-scale context enhancement features. The processing procedure is as follows: in, This indicates a splicing operation along the channel dimension. This represents a 1×1 convolution operation. This indicates a 3×3 convolution operation, and the 3×3 convolution operation does not change the feature map size; S33. For deep semantic features Separate interpolation upsampling is performed to obtain spatial scale features that match the multi-scale context enhancement features. The multi-scale context enhancement features and the upsampled deep semantic features are then sent to the decoder for subsequent segmentation.
2. The reliable medical image segmentation method based on information bottleneck-driven evidence learning and SAM3 as described in claim 1, characterized in that: The specific steps of S4 include: S41. The decoder is composed of the following structure: The input features are first transformed by a 1×1 convolutional layer, and then split into several groups along the channel dimension by a split operation. Each group of features is extracted by a depthwise separable convolution, and then all grouped features are concatenated and fused. Finally, a 1×1 convolutional layer is used to restore the number of channels, and the decoded feature map is output. S42, Decoder Receive Spatial resolution is gradually restored through upsampling and convolution operations, outputting pixel-level feature maps. Where C is the number of categories, , and Pixel-level feature mapping The height, width, and number of categories of the feature map; S43. The decoder output is a dual-head output structure, receiving the pixel-level feature mapping. The pre-evidence mean was obtained. Pre-evidence variance ,use Activation function to ensure ,in, It is a mapping function for outputting the mean. , It is a mapping function that outputs the unactivated variance.
3. The reliable medical image segmentation method based on information bottleneck-driven evidence learning and SAM3 as described in claim 2, characterized in that: The specific steps of S5 include: S51, Based on the pre-evidence mean variance of pre-evidence Pre-evidence Establish a Gaussian random distribution, expressed as: in, For the input medical images, Pre-evidence The probability distribution, Indicated by Each component is a diagonal covariance matrix composed of diagonal elements. Indicates a Gaussian distribution; The pre-evidence A Gaussian random distribution is established to transform the output from deterministic logits into a probability distribution, enabling the model to explicitly express the uncertainty of pre-evidence. S52. To ensure that the gradient can be backpropagated, a reparameterization technique is used to sample the pre-evidence, as shown below: in, For sampling index, For the number of samples, This represents the pre-evidence obtained from the k-th sampling. Standard Gaussian noise, For the first The standard Gaussian noise obtained from the second sampling Represents element-wise product. It is the identity matrix; during the training phase, for The loss obtained from the second sampling is averaged and optimized; during the inference phase, the loss is... The pre-evidence obtained from the second sampling is taken as the mean, and is expressed as: S53, the pre-evidence pass Function mapping as nonnegative evidence , is represented as: in, This represents the amount of evidence for each category of each pixel. It always holds true; 'c' represents the index of the category, and the amount of evidence. The current pixel is the first pixel. The support strength of a category; a higher value indicates that the model is more certain in its judgment of that category. S54, Based on the aforementioned nonnegative evidence Constructing Dirichlet distribution parameters , is represented as: in, For the first The Dirichlet parameter corresponding to the category is incremented by 1 to ensure... The parameter constraints satisfy the Dirichlet distribution; the corresponding Dirichlet distribution is expressed as: in, Let T be the class probability vector, and let T denote the transpose of the matrix. The sum of Dirichlet parameters. It is the Gamma function; S55, Based on the Dirichlet distribution parameters Calculate the predicted probabilities for each category, taking the expected value of the Dirichlet distribution, expressed as: in, The current pixel belongs to the first... Predicted probability of the category The sum of Dirichlet parameters for all categories; Final segmentation result for: ; Uncertainty estimation results for: ; S56. Introduce information bottleneck regularization constraints into the pre-evidence space to affect the pre-evidence probability distribution. With standard Gaussian prior To minimize the KL divergence between them, the information bottleneck constraint term is expressed as: in, This is an information bottleneck constraint. This represents the expectation operation. Let KL divergence be the denoted KL divergence. Analytical expansion of the above KL divergence yields: in, These are the trainable parameters of the model; S57, to The pre-evidence obtained from the sub-sampling is used to calculate the MSE Bayesian risk loss based on the Dirichlet distribution, and the mean is taken as the segmentation supervision loss, expressed as: in, Let b be the batch index, i be the spatial index in the height direction (i = 1, 2, ..., H), and j be the spatial index in the width direction (j = 1, 2, ..., W). Let be the label for the b-th batch, the c-th category, and the position (i,j) pixel. For the b-th batch, c-th category, and pixel (i,j); This represents the sum of all category Dirichlet parameters at pixel (i,j) in the b-th batch. S58. Jointly optimize the segmentation supervision loss and the information bottleneck constraint term to construct the overall loss function of the uncertainty estimation module, expressed as: Where L is the overall loss function of the uncertainty estimation module. The weight hyperparameter is used to constrain information bottlenecks and balance segmentation accuracy with uncertainty and calibration performance.
4. The reliable medical image segmentation method based on information bottleneck-driven evidence learning and SAM3 as described in claim 3, characterized in that: In step S5, the model is trained and optimized by minimizing the overall loss function; the overall loss function It is obtained by weighted summation of segmentation loss, Dirichlet distribution fitting loss, and information bottleneck constraint term, and is expressed as: in, This indicates Dice's loss. The weights represent the Dice loss. This represents the binary cross-entropy loss. The weights represent the binary cross-entropy loss. This represents the fitting loss for the Dirichlet distribution. The weights represent the loss for fitting the Dirichlet distribution; The Dice loss and binary cross-entropy loss together constitute the segmentation loss. , is represented as: Wherein, the Dice loss Represented as: The binary cross-entropy loss Represented as: in, Let be the label of the b-th batch, at position (i,j) pixel. Let f(x) represent the probability distribution of the b-th batch at position (i,j) pixel, which follows a Gaussian distribution.