Self-adaptive three-dimensional medical image segmentation method and system fusing state space model

Abstract

The invention provides a state space model-fused adaptive three-dimensional medical image segmentation method and system, and belongs to the technical field of medical image processing. According to the technical scheme, the method comprises the steps of preprocessing input three-dimensional medical image data; inputting the preprocessed image into an image encoder for feature extraction to obtain an image embedding vector; receiving externally input semantic prompt information, and generating a prompt embedding vector through a prompt encoder; fusing the image embedding vector and the prompt embedding vector to form a fusion feature; and inputting the fusion feature into a three-dimensional mask decoder for decoding, and generating a segmented mask graph. The three-dimensional medical image segmentation method has the beneficial effects that a three-dimensional medical image segmentation solution which is excellent in precision, efficiency, self-adaptability and practicability is successfully constructed through organic fusion of a state space model (Mama), a parameter efficient fine tuning technology (LoRA, Adapter), deformable convolution and an interactive prompt mechanism, and the three-dimensional medical image segmentation method has excellent generalization and robustness.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a medical image segmentation method and system based on deep learning, which is particularly suitable for the accurate segmentation of three-dimensional medical images (such as MRI and CT), providing technical support for clinical diagnosis, surgical planning, etc. Background Technology

[0002] Medical image segmentation is an important technique in computer vision and medical image processing, widely used in image-guided surgery, preoperative planning, lesion monitoring, and assisted diagnosis. Image data from medical imaging technologies such as CT and MRI typically exhibit significant noise interference, blurred boundaries, and large morphological differences in organs and lesions, posing considerable challenges to image segmentation.

[0003] Traditional medical image segmentation methods, such as region growing, thresholding, level set methods, and model-based methods, have certain applicability in different scenarios, but they generally suffer from limited segmentation accuracy, poor adaptability to complex anatomical structures, and low computational efficiency. In recent years, with the development of deep learning, especially the widespread application of convolutional neural networks (CNNs) in image segmentation, deep learning-based medical image segmentation methods have become a research and application hotspot. Among them, network structures represented by the U-Net series have achieved good segmentation results on multiple datasets.

[0004] SAM (Segment Anything Model) is a large-scale pre-trained model designed for natural images, and it performs excellently in natural image segmentation tasks. However, due to the significant differences between natural images and medical images in terms of texture patterns, imaging structures, and annotation strategies, directly transferring SAM to medical image scenarios results in insufficient acquisition of 3D spatial information and weak detail modeling capabilities, thus affecting its segmentation performance.

[0005] To improve the performance of models in medical image segmentation, some studies have introduced efficient parameter fine-tuning strategies (such as LoRA) and adaptation modules (such as Adapters) to enhance the domain adaptability of the models. However, existing methods still have limitations in modeling complex anatomical structures and fusing multi-scale contextual information, making it difficult to meet the practical needs of high-precision medical image segmentation tasks. Summary of the Invention

[0006] To address the technical problems existing in the prior art, this invention provides a medical image segmentation method based on multi-scale feature fusion, which improves the segmentation accuracy of anatomical structures and lesion regions in medical images and is applicable to segmentation scenarios of various medical images such as CT, MRI, and ultrasound images.

[0007] To achieve the above objectives, the present invention provides the following technical solution: An adaptive 3D medical image segmentation method incorporating a state-space model, comprising the following steps: S1. Preprocess the input 3D medical image data; S2. Input the preprocessed image into the image encoder for feature extraction to obtain the image embedding vector; S3. Receive semantic prompt information from external input and generate a prompt embedding vector through a prompt encoder; S4. The image embedding vector and the prompt embedding vector are fused to form a fused feature; S5. Input the fused features into the 3D mask decoder for decoding to generate a segmentation mask image; wherein, the image encoder includes several Transformer blocks, each Transformer block integrates an Adapter module, the Adapter module inserts a Mamba module, and the Transformer blocks are connected to each other through a LoRA parameter adjustment layer.

[0008] Furthermore, the Adapter module in the image encoder is a Mamba-enhanced Adapter module, whose structure includes: a 1×1×1 channel compressed convolution, a 3×3×3 deformable convolution, and a 1×1×1 channel restoring convolution connected in sequence, and includes a residual connection path integrating a Mamba module.

[0009] Furthermore, the Mamba module uses learnable weight coefficients. The global attention weight g and the local attention weight l are weighted and fused, i.e. , where g is obtained by performing global average pooling and a fully connected layer with the Sigmoid activation function on the input feature map, and l is obtained by performing a 1×1×1 convolution and the Sigmoid activation function on the input feature map.

[0010] Furthermore, the cue encoder is used to process point cue and / or box cue; for point cue, the 3D coordinates are mapped into a high-dimensional vector through position encoding; for box cue, the 3D bounding box is rendered into a binary mask and its features are extracted by a lightweight 3D convolutional encoder and then mapped into a cue embedding vector.

[0011] Furthermore, the image embedding vector and the cue embedding vector are fused through a cross-attention mechanism, where the cue embedding serves as the query Q and the image embedding serves as the key K and value V.

[0012] Furthermore, the 3D mask decoder includes alternating stacked Transformer and Mamba modules, as well as a 3D transposed convolutional layer; the decoder is connected to a 1×1 convolutional layer, a Sigmoid activation function, and a multilayer perceptron module at its end, for outputting the final segmentation mask image.

[0013] This invention also includes an adaptive 3D medical image segmentation system based on a fused state-space model, used to implement the aforementioned adaptive 3D medical image segmentation method based on a fused state-space model, the system comprising: The image encoder is used to extract features from the input 3D medical image and output an image embedding vector. It includes multiple Transformer blocks, LoRA parameter adjustment layers, and a Mamba enhancement adapter module integrated within the Transformer block. A cue encoder is used to receive external semantic cue information and generate cue embedding vectors; The 3D mask decoder is used to fuse image embedding vectors and cue embedding vectors and decode to generate a segmentation mask map. It includes a multimodal fusion unit, multiple Mamba modules, a Transformer module, and a 3D transposed convolutional layer.

[0014] Furthermore, the Transformer block in the image encoder includes a multi-head self-attention layer and integrates a LoRA structure in the projection path of query Q and value V, achieving efficient parameter fine-tuning through a low-rank decomposition matrix.

[0015] Furthermore, the multimodal fusion unit in the 3D mask decoder employs a cross-attention mechanism, using the cue embedding vector as the query and the image embedding vector as the key and value.

[0016] The beneficial effects of this invention are: Compared with existing technologies, the adaptive 3D medical image segmentation method and system based on the fusion of state-space models described in this invention has the following technical features and beneficial effects: (1) Significantly improve segmentation accuracy: By introducing the Mamba module and deformable convolution, the model can efficiently model long-range dependencies in 3D medical images and adaptively capture complex and irregular organ and lesion boundaries. Experimental data show that on multiple authoritative abdominal multi-organ segmentation datasets such as AMOS2022 and BTCV, the average Dice coefficient of this invention reaches 0.904 and 0.901, respectively, which surpasses all mainstream methods including nnUNet, TransUNet, Swin-UNetr and various diffusion models. The segmentation accuracy is particularly significantly improved for organs with complex shapes or small volumes such as pancreas, gallbladder, esophagus and adrenal gland.

[0017] (2) Enhanced ability to model three-dimensional spatial context information: Traditional two-dimensional segmentation or partial three-dimensional models are difficult to effectively utilize the spatial information across slices; This invention achieves unified modeling of the global and local context of three-dimensional volume data through three-dimensional Patch Embedding, three-dimensional Transformer encoder and three-dimensional mask decoder Mamba module and three-dimensional transposed convolution; This not only improves the spatial consistency of segmentation, but also significantly improves the segmentation effect for regions with blurred boundaries, which has been verified in the optimization of HD95 index.

[0018] (3) Achieving efficient parameter fine-tuning and powerful domain adaptation capabilities: By integrating the LoRA structure and Adapter enhancement module into the Transformer block, this invention can efficiently adapt to different medical image segmentation tasks by training only a small number of new parameters while freezing the weights of the main intervention training model. This design greatly reduces computational costs and the risk of overfitting, while maintaining the powerful feature extraction capabilities obtained by the model from large-scale natural image pre-training, enabling it to quickly and efficiently transfer to various medical imaging modalities such as CT, MRI, and ultrasound.

[0019] (4) Supports flexible human-computer interaction and improves clinical applicability: By designing an independent prompt encoder, this invention supports interactive prompts in the form of points and boxes; this allows doctors to intervene in the segmentation process and guide the model to perform fine segmentation of specific regions of interest; combined with the cross-attention multimodal fusion mechanism, the model can accurately respond to these prompts and generate segmentation results that are highly consistent with the user's intent, which greatly enhances the applicability and reliability of this method in clinical diagnosis and surgical planning.

[0020] (4) Optimize boundary segmentation quality and improve morphological fidelity: By introducing boundary-aware loss into the triple loss function and combining it with the three-dimensional transposed convolutional layer in the decoder for progressive upsampling and boundary restoration, this invention pays special attention to optimizing the boundary of the segmentation target; this makes the generated segmentation mask boundary clearer and smoother, with a higher degree of consistency with the real anatomical structure, which is crucial for downstream tasks such as surgical navigation and quantitative analysis of lesions.

[0021] (5) It has excellent generalization and robustness. Extensive experiments conducted on publicly available datasets covering various modalities and organs, including fundus, brain, thyroid, and breast ultrasound, demonstrate that the proposed method maintains leading segmentation performance under different imaging devices, anatomical structures, and lesion characteristics. This proves its strong generalization ability and robustness to common problems in medical images, such as noise and inhomogeneity. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is the overall flowchart of the present invention; Figure 2 This is a general framework diagram of the present invention; Figure 3 This is a structural diagram of the Mamba-enhanced Adapter module invented in this paper; Figure 4 This is a structural diagram of the Mamba module invented in this paper; Figure 5 This is a segmentation result of a medical image. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The following description, in conjunction with the accompanying drawings... Figures 1-5 The adaptive 3D medical image segmentation method and system based on the fusion state-space model are further explained.

[0024] Example 1 This embodiment provides an adaptive 3D medical image segmentation method based on a fused state-space model. Specifically, it is an improved segmentation method for 3D medical images (such as MRI and CT scans), aiming to address the problems of insufficient 3D spatial modeling and low computational efficiency in existing methods. The method includes constructing a network model for segmentation, training the network model, and using the network model to segment the input medical image to obtain the segmentation result.

[0025] In one approach, Figure 1 The training method shown includes the following steps: S1: Medical image input and processing S2: The image encoder takes multi-scale features. S3: Prompt Message Input and Encoding S4: Image and cues embedding and fusion S5: Mask encoder performs feature decoding S6: The output module generates and outputs the segmentation mask. In one approach, step S1, medical image processing, includes input image processing that sequentially performs format conversion, window width and level truncation, grayscale normalization, size resampling and unification, and data augmentation through random flipping / rotation / scaling / deformation / brightness enhancement. The final output is fixed-size, numerically normalized 3D volume data for subsequent Patch Embedding. Patch Embedding converts high-resolution 3D images into fixed-length token sequences that can be processed by the Transformer, while preserving spatial information. Medical image segmentation datasets include the REFUGE dataset, BraTs dataset, TNSCUI dataset, and DDTI dataset. The medical image data used in this study are all taken from publicly available benchmark subsets: REFUGE2018 provides 1200 fundus images for optic disc and optic cup segmentation; BraTs 2020 contributes 369 multi-sequence brain MRIs for brain tumor segmentation; TNSCUI 2020 contains 3644 thyroid ultrasound images for nodule detection; and the BUSI dataset contains 780 breast ultrasound images for breast tumor analysis. After standardization, deduplication, and quality control, approximately 6000 samples were obtained, which were divided into a training set of 4200 samples, a validation set of 900 samples, and a test set of 900 samples, in a ratio of 7:1.5:1.5. During the training phase, the 3D volume was subjected to random rotation ±30°, flipping along any axis (probability 50%), isotropic scaling of 0.8–1.2, voxel intensity perturbation ±20%, Gaussian noise σ=0.01, and 3D elastic deformation (α=15, σ=3), while ensuring consistency across the slice space.

[0026] In one approach, step S2 involves an image encoder that performs multi-scale feature extraction on the input 3D medical image. This encoder primarily comprises 12 identical Transformer modules. Within each Transformer module, the data first enters a LayerNorm to flatten the distribution of the token sequence. It is then decomposed into Query, Key, and Value by three linear projections. The projections of Query and Value are additionally connected in parallel with a LoRA low-rank bypass, with only these two small matrices participating in gradient updates while the backbone weights remain frozen. Subsequently, multi-head self-attention is calculated: Q is multiplied by the transpose of K, scaled, softmaxed, and then multiplied back into V to obtain the attention-weighted features. These features are then enhanced by an Adapter module and added to the residual. Next, LayerNorm processing is performed again, and the data is fed into an MLP for GELU activation. The MLP output is then added to the skip connection result that has already undergone Adapter enhancement in the residual path; this completes one Transformer module.

[0027] The LoRA low-rank bypass introduces two trainable low-rank matrices. The query weight matrix in the original multi-head self-attention module (Equation 1) of the Transformer. and key-value weight matrix Performing linear incremental modeling involves adding two bypasses to Q and V. In Q (as shown in Equations 2 and 4). The main road is frozen. To enable the addition of trainable bypasses, V is also trained by updating only the two small matrices A and B during training.

[0028] (1) (2) (3) (4) in, The attention module is the core of the Transformer module. Q is the query vector, K is the key vector, V is the value vector, d is the feature dimension of each input token, and softmax is the normalization function.

[0029] The Adapter enhancement module (Formula 5) enhances the perception of lesion contours. As a bottleneck structure, the Adapter enhancement module consists of a 1×1×1 channel compressive convolution, a 3×3×3 deformable convolution, a 1×1×1 channel restoring convolution, and a residual connection path including the Mamba module (Formula 6). The deformable convolution captures shape-sensitive structural information by learning sampling offsets, while the Mamba module simultaneously utilizes "global-local hybrid attention" to fuse global context and local geometric details in the spatial dimension, significantly enhancing the fine-grained perception capability of 3D medical images.

[0030] (5) (6) (7) (8) DCN stands for Deformable Convolutional Network. Here, g represents the learnable mixture coefficients, l represents the global weight, FC represents the fully connected layer, and GlobalAvgPool represents the global average pooling.

[0031] Ultimately, 12 identical but parameter-independent Transformer modules (Equation 9) are stacked sequentially to form the backbone of the image encoder. After 12 such global-local co-modeling iterations, the token representation output by the encoder retains fine-grained anatomical details and has long-range dependencies across slices, thus providing highly discriminative image embedding for the downstream mask decoder.

[0032] (9) (10) in, The Adapter module is an enhancement of Mamba. MHSA stands for Multi-Head Self-Attention, LN stands for Layer Normalization, and MLP stands for Multi-Layer Perceptron.

[0033] In one approach, step S3 prompts the encoder module to process interactive prompts from external input. We convert these prompts into a high-dimensional vector using sine / cosine position encoding (PE) (Equation 11) for addition or concatenation with image features. The 3D version of the PE formula is: (11) PE stands for Positional Encoding. x, y, z represent the spatial location information of the current click. First, the sine and cosine functions can map integer coordinates to a continuous and smooth high-dimensional space, solving the problem that networks have difficulty learning discrete coordinates. Second, the three scalars (x, y, z) are expanded into a d-dimensional vector, aligning its dimensions with the image token. Then, different frequencies are used... Different periods are generated to ensure that each coordinate combination has a unique vector code; finally, the sine and cosine functions enable the difference in relative positions to be represented by the dot product, which facilitates the attention mechanism to capture positional relationships.

[0034] For the position encoding of box prompts, a box consists of two points, the top left corner and the bottom right corner, and shares the PositionEmbedding module with point prompts.

[0035] In one approach, in step S4, the image and cue embeddings are fused by fusing the multi-scale embedding features from the image encoder with the semantic guidance embedding generated by the cue encoder through cross-attention to form a unified fused context representation (Formula 12).

[0036] (12) in To indicate that the encoder outputs PE, The output of the image encoder is d, where d is the feature dimension and Z is the fused feature that serves as the output of the mask decoder.

[0037] In one approach, the mask decoder in step S5 is used to upsample the fused semantic embedding layer by layer and restore it to the original image size, completing structural boundary restoration and spatial reconstruction. First, the original Transformer and Mamba modules are each repeated twice, then... The (3D transposed convolutional layer) is repeated twice, just like the Mamba unit, with four groups of components cascaded sequentially to form the main body. This is followed by a 1×1 convolution and a Sigmoid activation. Finally, to further optimize boundary sharpness and region consistency, a lightweight MLP module is added to output a segmentation mask image, as shown below: (13) Where M is the Mask, which is then used for loss calculation in the Ground Truth graph. For Sigmoid activation function, Mamba is the Mamba module, and TC3d is... (3D transposed convolutional layer), Attention is the original Transformer block.

[0038] In one approach, during model parameter initialization, an image encoder is constructed based on a pre-trained ViT-Base model. The Mamba module is initialized using a Xavier normal distribution (Glorot initialization), and the output layer MLP weights are initialized using a normal distribution with a mean of 0 and a standard deviation of 0.02. The bias term is initialized to 0. The training configuration uses the AdamW optimizer (β1=0.9, β2=0.999, weight decay=0.05), with a gradient clipping threshold of 1.0. The learning rate strategy is set to an initial value of 2e-4, which remains constant after 500 iterations, then linearly decays to 2e-5 between 500 and 1500 iterations, and further decreases to 2e-6 between 1500 and 2000 iterations. For hardware configuration, mixed-precision training is implemented using a single NVIDIA RTX 3090 GPU (24GB VRAM), with 4GB of VRAM allocated to the 3D data due to batch size constraints. Data augmentation includes a composite strategy of random rotation (±30°), scaling (0.8-1.2x), grayscale shift (20%), elastic deformation (2.0, 20.0), and a 50% probability mirror flip. The loss function uses a triple-weighted combination: 0.6Dice loss + 0.4Focal loss (γ=2.0, α=0.8) + 0.2 boundary-aware loss. The training cycle consists of 2000 iterations, with early stopping (patience=2) implemented every 100 iterations for validation. In multiple medical image segmentation tasks (covering MRI / CT / fundus / ultrasound), multi-dimensional evaluation is performed using Dice score (main metric), HD95, and IoU. The final model is saved based on the optimal Dice value from the validation set. Experiments show that this method works on the AMOS2022 abdominal CT dataset (…). Figure 5 It significantly improves the segmentation accuracy of small organs such as the pancreas.

[0039] Example 2 like Figure 1 The image shown illustrates a training method for the medical image segmentation model provided in this embodiment, comprising: Step 1: Acquire medical image datasets and perform sample augmentation. The datasets used include the REFUGE dataset, BraTS brain tumor dataset, TNSCUI thyroid ultrasound image dataset, and DDTI breast image dataset, covering multimodal medical images such as fundus, MRI, and ultrasound, totaling approximately 6000 images. The training, test, and validation sets are divided in a 7:1.5:1.5 ratio. To improve model robustness, various augmentation strategies are employed, including random rotation (range ±30°), random rotation (-30° to +30°), horizontal / vertical flipping (50% probability), random scaling (0.8-1.2x), color perturbation, and elastic deformation. Cross-slice consistency enhancement is also implemented for the 3D image sequences.

[0040] Step 2: Build and initialize the segmentation model ( Figure 2 The model structure is as described in Example 1. The image encoder consists of multiple layers of Transformer blocks, with an embedded Adapter enhancement module and LoRA tuning structure. Figure 3 The algorithm extracts cross-scale semantic information through a self-attention mechanism and enhances boundary representation capabilities by combining locally deformable convolution. The prompt encoder module supports multiple interactive input methods depending on the specific task, including points and bounding boxes, and generates guiding vectors aligned with image semantics through position embedding and multimodal fusion mechanisms. In the decoding stage, the Mamba module is introduced to construct a state-space modeling path, enabling the modeling of long-range dependencies in the image with linear complexity. Figure 4 It is particularly suitable for the restoration of complex anatomical structures and cross-slice consistency modeling in three-dimensional medical images.

[0041] Step 3: Training Phase. The 3D medical images and their segmentation labels (Ground Truth) are fed into the network in batches. The model executes the following steps sequentially: The image encoder divides the data and outputs 64×64×64×256 embedding features; the cue encoder converts user point / box cues into 256-dimensional vectors and then performs cross-attention fusion with the image tokens; the fused features are progressively upsampled by the mask decoder; at the decoding end, a prediction mask is generated using a 1×1×1 convolution + Sigmoid; finally, the error is calculated with Ground Truth using a triple loss of Dice + Focal + Edge, and the parameters are updated in reverse.

[0042] Step 4: Define and optimize the loss function. In this embodiment, to simultaneously optimize overall region accuracy, address the foreground-background imbalance problem, and improve the model's ability to recognize structural boundaries, a triple-weighted combined loss function is adopted, comprehensively incorporating Dice loss, Focal loss, and Edge loss (boundary-aware loss). The overall loss function is designed as follows: (14) The Dice loss, used to measure the overlap between the predicted mask and the ground truth label in the overall region, is defined as: (15) in ∈ [0,1] indicates the first... The predicted probability of individual elements, ∈ {0,1} represents its corresponding real label. =1e-5 is a very small positive number to prevent division by zero.

[0043] Focal loss is used to alleviate the imbalance between foreground and background samples, focusing on hard-to-classify samples. In this embodiment, γ=2.0 and α=0.8 are used, and their definitions are as follows: (16) This loss function assigns higher weights to misclassified samples, which helps the model learn boundary regions and small-volume lesions.

[0044] Edge loss is used to improve the model's ability to identify structural edges. Specifically, it applies the Sobel edge detection operator to the label mask, aligning the boundary regions of the predicted mask with the true boundary regions, and then calculates the Dice error. Its expression is as follows: (17) in and These are the edge images of the predicted mask and the label mask after being processed by the edge extraction operator, respectively.

[0045] Joint loss function During training, the network is provided with multiple constraints from the overall region, difficult samples, and boundary regions, significantly improving segmentation accuracy and boundary clarity. By adjusting various weight coefficients, it can be adapted to segmentation tasks of different organs or lesion structures.

[0046] The medical image segmentation model proposed in this invention is primarily designed for medical image analysis tasks involving complex 3D structures, blurred tissue boundaries, and varied lesion morphologies. It is particularly suitable for the automatic segmentation of multimodal and multi-organ images. To verify the universality and segmentation performance of the proposed model, experiments were conducted on several representative and challenging publicly available medical image datasets, and a systematic comparison was performed with current mainstream medical image segmentation networks.

[0047] Specifically, the selected datasets include: AMOS2022 dataset (abdominal multi-organ MRI / CT data, covering 15 organ classes), REFUGE dataset (segmentation of optic disc and optic cup structures in the fundus), BraTs dataset (segmentation of brain tumors from multiple sequences of brain MRI), TNSCUI dataset (annotation of nodule regions in thyroid ultrasound images), DDTI dataset (tumor segmentation in breast ultrasound images), and BTCV dataset (CT images of organs such as the liver, spleen, and kidneys). These datasets cover a variety of medical imaging modalities (such as MRI, CT, ultrasound, and fundus images), and have characteristics such as high resolution, class imbalance, and blurred boundaries, enabling a comprehensive evaluation of the model's performance in complex medical scenarios.

[0048] The comparison models include seven current mainstream medical image segmentation models: nnUNet, TransUNet, SwinUNetr, UNetr, MedSegDiff, SegDiff, and EnsDiff. Among them, nnUNet is widely regarded as the benchmark for 3D segmentation tasks; TransUNet and SwinUNetr represent advanced models that incorporate Transformer structures; while MedSegDiff, SegDiff, and EnsDiff represent the latest advancements in diffusion-based segmentation methods.

[0049] To objectively quantify model performance, this invention employs the following three standard evaluation metrics: Dice coefficient (Dice Similarity Coefficient, DSC): Used to measure the degree of spatial overlap between the predicted result and the real mask. The value range is [0,1], and the closer it is to 1, the higher the segmentation accuracy. Intersection over Union (IoU): This evaluates the ratio of the intersection to the union of the predicted region and the actual region, reflecting the overall segmentation accuracy. A higher IoU value indicates better performance. 95% Hausdorff Distance (HD95): The 95th percentile value of the farthest distance between the predicted boundary and the true boundary. It represents the model's error sensitivity in boundary identification. The lower the value, the more accurate the boundary identification.

[0050] Performance comparison of this invention on the AMOS dataset, evaluated by Dice scores: In comparative experiments on the AMOS abdominal multi-organ segmentation task, the 3D-MSA algorithm proposed in this invention significantly outperforms the existing best model MedSegDiff (0.889) with an average Dice score of 0.904, achieving accuracy improvements in 12 out of 15 organs: key organs such as gallbladder (+1.8% to 0.935), esophagus (+3.8% to 0.884), pancreas (+2.5% to 0.879), and right adrenal gland (+3.3% to 0.836) show outstanding improvements; in terms of small organ optimization, the left adrenal gland (+2.0% to 0.771) and inferior vena cava (+1.6% to 0.934) perform excellently, with only the duodenum showing a slight decrease of 0.4% due to its clinically high variability. Horizontal comparison shows that it leads all organs compared to Swin-UNetr (average +2.4%), outperforms nnUNet in 14 organs (pancreas +5.8%), and improves all organs by more than 5% compared to TransUNet.

[0051] Table 1 In a comparative experiment on the BTCV abdominal multi-organ segmentation task, the segmentation algorithm proposed in this invention set a new performance record with an average Dice of 0.901, which is 0.003 higher than the previous best Med-SA model (0.898). Key organs such as the esophagus achieved the largest improvement (0.808→0.826, +1.8%), the liver segmentation accuracy reached 0.987 (+0.4%), and the spleen and left kidney improved to 0.987 / 0.977 (+0.2%), respectively. Although the segmentation of veins decreased slightly by 1.2% (0.852→0.840), the segmentation accuracy of the other 11 organs was the same as or better than Med-SA. Horizontal comparison shows that it improves by 2.2% compared to the medical diffusion benchmark MedSegDiff (0.879), achieves full organ suppression for Swin-UNetr (0.869) (e.g., gallbladder +4.9% to 0.843), and improves all organs for TransUNet (0.838) by more than 5% (adrenal gland +18.8% to 0.825).

[0052] Table 2 In a comprehensive evaluation of three multimodal medical image tasks—optic cup segmentation, brain tumor segmentation, and thyroid nodule segmentation—the segmentation algorithm proposed in this invention achieved a comprehensive performance breakthrough: In optic cup segmentation, its Dice coefficient reached 88.5% (an improvement of 0.8% and 1.4% compared to 3D-MSA and Med-SA, respectively), and its IoU surpassed 80% for the first time, reaching 80.9% (outperforming TransBTS by 5.2%); in brain tumor segmentation, the boundary accuracy HD95 was significantly reduced to 8.75mm (an improvement of 2.3mm compared to 3D-MSA and 16.0% compared to MedSegDiff), and the IoU reached 84.1% (surpassing MedSegDiff). Although the Dice coefficient decreased slightly by 0.2% to 90.5%, the 1.1% improvement in IoU and the optimization of HD95 demonstrate its substantial progress in boundary accuracy; in thyroid nodule segmentation, the Dice coefficient reached 89.5% (an improvement of 0.9% and 5.0% compared to 3D-MSA and UltraUNet, respectively), and the IoU increased from 81% to 81.2% (an improvement of 10.5% compared to ResUNet).

[0053] Table 3 The implementation method of the present invention has been described in detail above, but the present invention is not limited to the above-described implementation method. The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

[0054] Example 3 The input 3D medical image data undergoes preprocessing, including window width and level adjustment and grayscale normalization, to improve image quality. The preprocessed image is then input into a segmentation model, where PatchEmbedding (downsampling) is applied to the input medical image. This allows the Transformer to perform self-attention per token (single vector), improving spatial resolution and enhancing the detail representation of the input image. The image is then input into an image encoder for feature extraction. Within the image encoder, each Transformer block contains an Adapter module, and each Adapter module contains a Mamba module. Transformer blocks are connected via LoRA (Low-Rank Adaptation) as a bypass. The image embedding vector output from the image encoder is then upscaled and input into a 3D mask decoder. The system also includes a cue encoder module to receive semantic cue information from external input. This cue information, after upscaling, is input along with the image embedding into the 3D mask decoder, achieving fine segmentation based on interactive cueing. Finally, the multi-channel feature map output by the mask decoder is input into a 1×1 convolutional layer and a sigmoid function for activation processing, and a segmentation mask map is generated by the multilayer perceptron (MLP) module.

[0055] This invention also provides a medical image segmentation network system, the system comprising: Image encoder: includes multiple Transformer blocks and LoRA parameter adjustment layers; Prompt encoder: Used to generate an embedded representation of prompt information; The 3D mask decoder includes a multimodal fusion unit (cross-attention mechanism), multiple Mamba modules, a Transformer module and a 3D transposed convolutional layer (Transposed Conv 3D), as well as Sigmoid and MLP structures to generate the final segmentation mask map.

[0056] The image encoder comprises multiple Transformer blocks, each containing multiple Mamba-enhanced Adapter modules (hereinafter referred to as Adapter enhancement modules) for extracting multi-level, multi-scale image feature information. The image encoder internally uses LoRA modules as bypasses of the Transformer blocks for efficient parameter fine-tuning, thereby improving the model's transferability and adaptability in the medical field.

[0057] As a supplement to the technical solution, the image encoder module is used to perform layer-by-layer semantic encoding on the input 3D medical image to extract multi-scale spatial features and generate semantic embedding representations for subsequent mask decoder use. The image encoder includes an input processing unit, multiple Transformer encoding blocks, a LoRA parameter adaptation structure, an Adapter enhancement module, and an optional positional encoding mechanism. The input processing unit performs patch embedding on the original 3D medical image, dividing the image into fixed-size 3D patches using 3×3×3 convolutional kernels (stride=16). The Transformer blocks are used to model long-range dependencies in the 3D medical image. Each block contains a multi-head self-attention layer, residual connections, layer normalization, a feedforward neural network, and an Adapter enhancement module. The LoRA structure is integrated into the query (Q) and value (V) projection paths of the self-attention layer, and a parameter update bypass is constructed through a low-rank decomposition matrix to achieve efficient parameter fine-tuning while freezing the original weights. Each Transformer block deploys two Adapter enhancement modules: the first Adapter is inserted after the multi-head attention layer, and the second Adapter is embedded in the MLP residual path. Both adopt a bottleneck structure of 1×1×1 convolution dimensionality reduction → 3×3×3 deformable convolution → 1×1×1 convolution dimensionality increase. Figure 3The adapter module, consisting of a 1×1×1 channel compression convolution, a 3×3×3 deformable convolution connected to the residual connection path of the Mamba module, and a 1×1×1 channel recovery convolution, adaptively adjusts the shape of the convolution kernel through dynamically predicted offsets to accurately capture the three-dimensional morphological features of organs such as the liver and pancreas. The deformable convolution learns sampling offsets to capture shape-sensitive structural information, adapting to changes in organ boundaries and lesion morphology. The 1×1×1 channel recovery convolution restores the channels to their original dimensions. Furthermore, the MLP module integrated in the adapter consists of two fully connected layers (i.e., 1×1×1 convolutional layers). The first layer reduces the input features from dimension C to C / r (r is the compression ratio, default r=4), and the second layer restores them to the original dimension C. A GELU activation function is inserted between the bottleneck structures to enhance the model's ability to transform local semantic features of medical images through nonlinear mapping. The image encoder as a whole adopts a stacked structure, preferably with 12 groups of Transformer combination units, each enhanced by the aforementioned adapter and LoRA. The final encoder outputs the embedded features, which are then input into the embedding up-dimensionality module for subsequent mask decoding.

[0058] The Transformer module is used to enhance the image encoder's ability to express the fine-grained structure of medical images. Internally, two Adapter enhancement modules are deployed in the original Transformer: the first Adapter is inserted after the multi-head attention layer, and the second Adapter is embedded in the MLP residual path. A multi-head attention mechanism is also set after the Adapter block to capture long-range dependency features in the image.

[0059] The cue encoder module receives semantic cue information from external input and encodes it into an embedding vector. After dimensionality upscaling, this cue information, along with the image embedding, is input into the mask decoder to achieve fine segmentation based on interactive cueing.

[0060] As a supplement to the technical solution, the prompt encoder module receives prompt information provided by the user and encodes it into a prompt embedding representation aligned with image features, thereby guiding the segmentation model to accurately segment specific regions. This study employs point and bounding box prompts. For point prompts, the 3D coordinates (x, y, z) of the user's click are first mapped to a 64×64×64 grid index aligned with the image embedding space. Then, a 3D position vector is obtained through sine and cosine functions or a learnable position embedding matrix, which is then projected into a 256-dimensional prompt vector through two layers of MLP. For bounding box prompts, the 3D bounding box is first rendered into a binary mask of the same size as the original volume, then downsampled to 64×64×64, and spatial features are extracted using a lightweight 3D convolutional encoder (two layers of 2×2×2 stride convolutions), ultimately also mapped to 256 dimensions. All prompt embeddings are of the same dimension as the output of the image encoder, ensuring spatial alignment and semantic consistency.

[0061] The multimodal fusion unit is used to fuse image information and cue information into a unified feature representation. A cross-attention mechanism is employed, using the cue embedding as the query vector Q and the image encoder output imageembedding as the key-value pair KV. A weighted fusion method is used to extract the image feature subspace strongly correlated with the cue location.

[0062] The mask decoder includes multiple alternating stacked Transformer modules and Mamba modules. The Mamba module is an attention unit based on state space modeling, used to enhance the decoder's ability to model three-dimensional structural information and effectively fuse global contextual information and local detail information of the image. It also sets up multiple three-dimensional transposed convolutional layers (Transposed Conv 3D) to progressively upsample and restore spatial dimensions while preserving boundary and shape information in the encoded features.

[0063] As a supplement to the technical solution, the 3D mask decoder module receives the 3D embedding features output by the image encoder and the guiding features generated by the cue encoder, and fuses them to gradually decode a spatially aligned segmentation mask. This module is designed to balance global contextual understanding with local boundary restoration capabilities, making it particularly suitable for 3D medical image segmentation tasks with complex structures and blurred boundaries. The fused embedding features output by the image encoder and the cue embeddings output by the cue encoder are input into the main body of the 3D mask decoder. The main body of the 3D mask decoder consists of multiple original Transformer modules and Mamba modules. The Transformer modules are used to model global semantic dependencies, and their structure includes multi-head attention layers, residual connections, feedforward networks, and layer normalization units, maintaining the channel dimension unchanged. The Mamba modules are built based on a state-space modeling mechanism to model long-distance dependencies with linear complexity. The Transformer and Mamba modules are combined twice, and the 3D transposed convolutional layer (Transposed Conv 3D) and the Mamba module are combined twice. The four are then connected in series as the main part of the 3D decoder, followed by a 1×1 convolutional layer and a sigmoid function for activation processing. The segmentation mask map is generated by the MLP (Multilayer Perceptron) module.

[0064] The Mamba module features parallel layer and local attention layer, fused via learnable weights. The global attention layer first performs global average pooling along the spatial dimension on the input feature map to obtain a single global feature vector, reducing the number of parameters and suppressing overfitting. This global feature vector is then input into a fully connected layer and processed by a sigmoid activation function to generate a global channel weight vector corresponding to the number of channels in the input feature map, representing the global importance of each channel. The local attention submodule performs local convolution operations on the input feature map using 1×1×1 convolutions to obtain a local feature map with the same number of channels as the input feature map. This local feature map is then processed by a sigmoid activation function to generate a local attention weight map, where the value of each element represents the local importance of the corresponding spatial location. Finally, the global channel weight vector and the local attention weight map are fused element-wise with weights, achieving adaptive integration of global and local information from the input feature map, thereby enhancing the model's ability to represent and segment complex 3D medical images. The mask image size is consistent with the input image, enabling accurate representation of the voxel distribution of the target organ or lesion region in 3D space, achieving high-precision volumetric segmentation. By introducing the Mamba module and a multi-level upsampling structure, the decoder not only has the ability to model the global context, but also has the ability to refine and restore layer by layer, which can significantly improve the segmentation accuracy in complex structural regions (such as liver boundaries and tumor edges), and is suitable for 3D medical image segmentation tasks such as MRI and CT.

[0065] The mask image is used to indicate the specific location and boundaries of the target area (such as a tumor, organ, etc.).

[0066] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An adaptive 3D medical image segmentation method incorporating a state-space model, characterized in that, The steps are as follows: S1. Preprocess the input 3D medical image data; S2. Input the preprocessed image into the image encoder for feature extraction to obtain the image embedding vector; S3. Receive semantic prompt information from external input and generate a prompt embedding vector through a prompt encoder; S4. The image embedding vector and the prompt embedding vector are fused to form a fused feature; S5. Input the fused features into the 3D mask decoder for decoding to generate a segmentation mask image; wherein, the image encoder includes several Transformer blocks, each Transformer block integrates an Adapter module, the Adapter module inserts a Mamba module, and the Transformer blocks are connected to each other through a LoRA parameter adjustment layer.

2. The adaptive 3D medical image segmentation method based on a fused state-space model according to claim 1, characterized in that, The Adapter module in the image encoder is a Mamba-enhanced Adapter module, whose structure includes: a 1×1×1 channel compressed convolution, a 3×3×3 deformable convolution, and a 1×1×1 channel restoring convolution connected in sequence, and includes a residual connection path integrating the Mamba module.

3. The adaptive 3D medical image segmentation method based on a fused state-space model according to claim 2, characterized in that, The Mamba module uses learnable weight coefficients. The global attention weight g and the local attention weight l are weighted and fused, i.e. , where g is obtained by performing global average pooling and a fully connected layer with the Sigmoid activation function on the input feature map, and l is obtained by performing a 1×1×1 convolution and the Sigmoid activation function on the input feature map.

4. The adaptive 3D medical image segmentation method based on the fusion state-space model according to claim 1, characterized in that, The cue encoder is used to process point cues and / or bounding box cues; for point cues, 3D coordinates are mapped into high-dimensional vectors through position encoding; for bounding box cues, 3D bounding boxes are rendered into binary masks and their features are extracted by a lightweight 3D convolutional encoder and then mapped into cue embedding vectors.

5. The adaptive 3D medical image segmentation method based on a fused state-space model according to claim 1, characterized in that, The image embedding vector and the cue embedding vector are fused through a cross-attention mechanism, where the cue embedding serves as the query Q and the image embedding serves as the key K and value V.

6. The adaptive 3D medical image segmentation method based on a fused state-space model according to claim 1, characterized in that, The 3D mask decoder includes alternating stacked Transformer and Mamba modules, as well as a 3D transposed convolutional layer; the decoder is connected to a 1×1 convolutional layer, a Sigmoid activation function, and a multilayer perceptron module at the end, for outputting the final segmentation mask image.

7. An adaptive 3D medical image segmentation system based on a fused state-space model, used to implement the adaptive 3D medical image segmentation method based on a fused state-space model as described in any one of claims 1-6, characterized in that, The system includes: The image encoder is used to extract features from the input 3D medical image and output an image embedding vector. It includes multiple Transformer blocks, LoRA parameter adjustment layers, and a Mamba enhancement adapter module integrated within the Transformer block. A cue encoder is used to receive external semantic cue information and generate cue embedding vectors; The 3D mask decoder is used to fuse image embedding vectors and cue embedding vectors and decode to generate a segmentation mask map. It includes a multimodal fusion unit, multiple Mamba modules, a Transformer module, and a 3D transposed convolutional layer.

8. The adaptive 3D medical image segmentation system based on the fusion of state-space models according to claim 7, characterized in that, The Transformer block in the image encoder contains a multi-head self-attention layer and integrates a LoRA structure in the projection path of query Q and value V, achieving efficient parameter fine-tuning through a low-rank decomposition matrix.

9. The adaptive three-dimensional medical image segmentation system based on the fused state-space model according to claim 7, characterized in that, The multimodal fusion unit in the 3D mask decoder employs a cross-attention mechanism, using the cue embedding vector as the query and the image embedding vector as the key and value.

Images