A medical image segmentation method and segmentation system

Abstract

This invention relates to the field of medical image processing, specifically disclosing a medical image segmentation method and system. The method includes: firstly, preprocessing 3D CT data to unify data format and spatial attributes; then, extracting multi-view 2D slices along three orthogonal directions of the RAS coordinate system; using U-Net for preliminary segmentation prediction of the slices; and inputting the predicted foreground region as a bounding box cue into the SAM2 model; calculating the comprehensive similarity and model confidence based on the outputs of U-Net and SAM2; and dynamically selecting an iterative optimization or direct fusion strategy based on preset conditions. Finally, integrating the 2D masks from different views / models using a voxel-level weighted method to generate a high-quality 3D mask, while using different colors to visualize the confidence level. This method achieves a highly automated and high-precision lung cancer screening technology through multi-model complementarity and intelligent decision-making mechanisms.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and specifically to a medical image segmentation method and segmentation system. Background Technology

[0002] Lung cancer is one of the leading causes of cancer-related morbidity and mortality worldwide. Early-stage lung cancer typically presents as nodules in the lungs, and 3D CT imaging is the primary method for diagnosis. However, manual image interpretation is subjective and labor-intensive. Therefore, the use of computer-aided diagnostics (CAD) for the automated segmentation and screening of lung nodules has become an important research direction.

[0003] Existing technologies mainly include the following types of methods: 1. Traditional U-Net-based segmentation methods.

[0004] While end-to-end lung nodule segmentation using an encoder-decoder structure (such as U-Net) improves efficiency to some extent, it suffers from insufficient ability to identify low-contrast lesion boundaries, leading to missed and false positives, a high false positive rate, and strong dependence on a large amount of labeled data, thus limiting its generalization ability.

[0005] 2. Zero-shot-based segmentation methods (such as SAM).

[0006] While SAM (Segment Anything Model) has general segmentation capabilities, it suffers from poor adaptability in medical scenarios, lacks 3D spatial modeling capabilities, and its prompt generation heavily relies on manual annotation, making it difficult to automate fully.

[0007] The closest technology to this invention is the Lung-CADex system, which uses a variant of MedSAM to detect lung nodules. However, it suffers from insufficient fusion of multi-view features and an overly simple and fixed fusion method, resulting in limited accuracy and a high false positive rate.

[0008] In summary, existing technologies still suffer from problems such as low precision due to blurred lesion boundaries, insufficient fusion of multi-view segmentation results, high false positive rate, and lack of three-dimensional spatial continuity. There is an urgent need to propose more efficient, accurate, and automated methods. Summary of the Invention

[0009] The purpose of this invention is to overcome the above-mentioned defects in the existing technology and propose a medical image segmentation method and system. Through multi-view collaborative segmentation, large model prompting for refined optimization and three-dimensional weighted fusion, it can achieve accurate and automatic segmentation and screening of lung nodules.

[0010] In a first aspect, embodiments of this application provide a medical image segmentation method, comprising the following steps: A medical image segmentation method, characterized in that it comprises the following steps: Step S1: Preprocessing three-dimensional CT data images.

[0011] Step S2: Extract two-dimensional slices from the preprocessed data along the three orthogonal directions of the RAS coordinate system from multiple perspectives.

[0012] Step S3: Use the U-Net model to perform independent segmentation prediction on the two-dimensional slice, and extract the foreground region predicted by the U-Net model as bounding box cue information and input it into the SAM2 model.

[0013] Step S4: The SAM2 model receives the foreground region bounding box cue information predicted by the U-Net model, generates a fine segmentation mask on the corresponding slice, refines the target region, and obtains the details and contour information of the lung nodules.

[0014] Step S4.1: Determine the difference between the SAM2 output and the U-Net output, and calculate the overall similarity C1.

[0015] The calculation process of the comprehensive similarity C1 is as follows: the overlap, intersection-overall ratio and spatial distance of the mask results of the two models are evaluated by three indicators: Dice coefficient, IoU and Euclidean distance. Each indicator is standardized and mapped to a unified interval. The three standardized indicators are weighted and summed according to the preset weights to obtain the comprehensive similarity C1, which measures the overall difference between the segmentation results of the two models.

[0016] Step S4.2: Set a comprehensive similarity threshold T. If the comprehensive similarity C1 is less than this threshold, proceed to step S4.3; otherwise, proceed to step S5.

[0017] Step S4.3: Input the segmentation mask output by the SAM2 model back into the U-Net model. The U-Net model receives the two-dimensional mask prompt information output by the SAM2 model and generates the iterative two-dimensional segmentation mask.

[0018] In this step, the 2D mask and the image are stitched together by channel stitching. In addition, when using U-Net in step S3, a full white mask is used to ensure that the model can be reused.

[0019] Step S5: Determine the overall similarity C1 between the outputs of the U-Net model and the SAM2 model, as well as the confidence level of the model outputs, and perform 3D reconstruction using different 2D masks.

[0020] In step S5, if the first condition is met, the two-dimensional mask output in step S4 is used for three-dimensional reconstruction; if the second condition is met, the bounding box cue information output by the U-Net network model in step S3 and the two-dimensional mask output by the SAM2 model in step S4 are used for three-dimensional reconstruction respectively, and the prediction results of different view spaces are fused at the voxel level, and multi-view three-dimensional mask fusion is completed based on uncertainty weighting; if the third condition is met, the iterated two-dimensional segmentation mask output by the U-Net network model in step S4.3 and the two-dimensional mask output by the SAM2 model in step S4 are used for three-dimensional reconstruction respectively, and the prediction results of different view spaces are fused at the voxel level, and multi-view three-dimensional mask fusion is completed based on uncertainty weighting.

[0021] In step S5, the first condition is that the comprehensive similarity C1 is greater than or equal to the threshold T, and the foreground region confidence index Z of the SAM2 model in step S4 is greater than or equal to the threshold T1.

[0022] The second condition is that the overall similarity C1 is greater than or equal to the threshold T, and the foreground region confidence index Z of the SAM2 model in step S4 is less than the threshold T1. The foreground region confidence index Z is the mean of the foreground region average confidence Z1 and the foreground region median confidence Z2. At this time, step S4.3 is not triggered, and the two-dimensional mask output by the U-Net network model in step S3 and the SAM2 model in step S4 is used for three-dimensional reconstruction.

[0023] The third condition is that the overall similarity C1 is less than the threshold T. At this point, step S4.3 is executed, the U-Net model generates an iterated two-dimensional segmentation mask, and three-dimensional reconstruction is performed using the iterated two-dimensional segmentation mask and the two-dimensional mask output by the SAM2 model in step S4.

[0024] The specific process of the multi-view 3D mask fusion based on uncertainty weighting includes: Step S5.1: Perform 3D reconstruction on the 2D slice masks of U-Net and SAM2 models in different spatial directions.

[0025] Step S5.2: Perform independent uncertainty estimation on the prediction results of U-Net and SAM2 models.

[0026] Step S5.3: Perform voxel-level fusion of the prediction results of U-Net and SAM2 in different view spaces, and complete multi-view 3D mask fusion based on uncertainty weighting. The weight allocation of each voxel position is determined by the uncertainty of the two models during fusion.

[0027] The uncertainty estimation is achieved using the Monte Carlo Dropout method, which includes: for each three-dimensional spatial location, i.e., voxel, analyzing the stability of the prediction result at that location in multiple predictions. Voxels with small changes in prediction results indicate high reliability of the model prediction, i.e., low uncertainty, while voxels with large changes in prediction results indicate high uncertainty of the model prediction; the above uncertainty assessment is performed on the prediction results of U-Net and SAM2 respectively to obtain independent voxel-level uncertainty information for the two models.

[0028] Secondly, this application provides a medical image segmentation system, comprising: a three-dimensional data preprocessing module for converting three-dimensional CT image data into NIfTI format and performing grayscale normalization and orientation correction; a slice generation module for extracting two-dimensional slices along three orthogonal directions of the RAS coordinate system; a U-Net segmentation module for performing segmentation prediction on the two-dimensional slice data and generating bounding box prompts for input to SAM2; a SAM2 fine segmentation module for receiving bounding box prompts and generating fine segmentation masks; and a three-dimensional reconstruction and fusion module for performing three-dimensional reconstruction and fusion on the multi-view two-dimensional segmentation masks.

[0029] Thirdly, this application also provides an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the steps of the method described in the first aspect.

[0030] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0031] Beneficial effects: One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. Significantly improve the accuracy of lung nodule segmentation: By using multi-view slicing technology and cascade segmentation with U-Net and SAM2 models, the problem of traditional single-view methods being unable to accurately identify the boundaries of lung cancer lesions is effectively solved, achieving more accurate and complete identification of lesion areas and significantly improving the overall accuracy of segmentation.

[0032] 2. Effectively reduce false positive detection rate and enhance segmentation robustness: Utilize comprehensive similarity (including IoU, Dice, and Euclidean difference) and confidence to dynamically trigger iterative optimization or fusion processes, achieving adaptive correction and balancing of model results, and improving the system's ability to handle complex cases. Employing a voxel-level multi-view fusion strategy, spatial consistency constraints reduce misjudgments caused by background noise and interference, significantly reducing the probability of false detection in non-lesion areas and improving the reliability and stability of small-sized lung nodule identification.

[0033] 3. Significantly improves the ability to finely reconstruct lung nodule boundary details: By utilizing the zero-sample high-precision segmentation characteristics of the SAM2 model and combining it with the initial prompts provided by the U-Net model, it can more effectively achieve fine recognition of complex boundaries such as spiculated and lobulated lung nodules, and significantly improve the segmentation quality of lung cancer lesion contour details.

[0034] 4. Provides high-precision three-dimensional quantitative analysis capabilities: Through a three-dimensional reconstruction strategy based on probabilistic fusion, it can accurately calculate the volume and shape characteristic parameters of lung nodules, such as sphericity and surface area to volume ratio, providing efficient, accurate and objective quantitative indicators for the staging and efficacy evaluation of lung cancer lesions in clinical practice, significantly improving the efficiency and accuracy of clinical analysis.

[0035] 5. Significantly reduces the need for labeled data and improves data efficiency: By combining the prompting learning capability of the SAM2 model, only a small amount of labeled data is needed to effectively train the initial prompting model, which greatly reduces the workload and cost of data labeling, and enables stable and excellent segmentation performance even with limited data.

[0036] 6. Achieve end-to-end automation and improve clinical application efficiency: The entire process, from 3D CT data preprocessing and multi-view slice generation to final 3D fusion reconstruction, is automated, significantly reducing manual intervention, greatly improving the efficiency of lung cancer screening and diagnosis, meeting the actual needs of large-scale clinical screening, and significantly improving the efficiency of medical resource utilization. Attached Figure Description

[0037] Figure 1 A flowchart of a medical image segmentation method provided in this application embodiment.

[0038] Figure 2 This is a schematic diagram illustrating the orientation of NIfTI format data provided in an embodiment of this application.

[0039] Figure 3 A flowchart illustrating the logic of selecting a two-dimensional mask for three-dimensional reconstruction, provided in an embodiment of this application.

[0040] Figure 4 A flowchart of the multi-view 3D mask fusion method provided in the embodiments of this application.

[0041] Figure 5 A schematic diagram of a medical image segmentation system provided in an embodiment of this application.

[0042] Figure 6 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that these descriptions are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] Example 1: As Figure 1 As shown, in one embodiment, taking lung cancer screening based on three-dimensional CT data images as an example, the steps of the technical solution of the present invention are as follows: Step S1: Preprocess the three-dimensional CT data images.

[0045] This step aims to perform standardized preprocessing of the 3D CT image data used for lung cancer screening to ensure the accuracy and stability of subsequent analysis.

[0046] Data Format Conversion: Raw lung CT data is typically stored in the medical imaging standard DICOM format. To facilitate batch processing and standardized analysis workflows by computer programs, the DICOM data first needs to be converted to the NIfTI format, which is more suitable for deep learning models. Unlike DICOM, the NIfTI format can self-contain medical image data and its detailed spatial attribute information, such as image dimensions, pixel spacing, coordinate system, and other metadata. A specific orientation diagram is shown below. Figure 2 As shown, this spatial metadata is crucial for accurate image segmentation and subsequent 3D reconstruction.

[0047] In the NIfTI data format, the RAS (Right-Anterior-Superior) anatomical coordinate system is used by default as the positive direction of space. The coordinate axes are defined as follows: X-axis: the positive direction points to the right side of the patient (Right).

[0048] Y-axis: The positive direction points in front of the patient (Anterior).

[0049] Z-axis: The positive direction points towards the top of the patient's head (Superior).

[0050] The anatomical planes and section orientations are defined as follows: The Axial Plane is the XY plane. Slices for this view are stacked along the Z-axis (up / down direction). The Coronal Plane is the XZ plane. Slices for this view are stacked along the Y-axis (anterior / posterior direction). The Sagittal Plane is the YZ plane. Slices for this view are stacked along the X-axis (left / right direction).

[0051] This unified spatial coordinate system definition ensures a high degree of consistency in spatial location for subsequent slicing, analysis, and 3D reconstruction operations, avoiding spatial position deviations caused by inconsistencies in the coordinate system.

[0052] The raw pixel values ​​(CT values, in HU) in CT images typically have a wide range. To reduce the impact of data noise and standardize the input range of the model, this invention uses a linear mapping method to standardize the CT values, linearly mapping the CT value range [-1000, +1000] to the interval [0, 1]. Voxels outside this range (i.e., CT values ​​<-1000 or >+1000) are considered noise and uniformly cropped to boundary values ​​0 or 1.

[0053] This standardized data processing ensures the consistency and stability of data from different patients and different scanning devices when inputting into the model, significantly improving the generalization performance of the deep learning model.

[0054] Step S2: Extract multi-view two-dimensional slices from the preprocessed data along the three orthogonal directions of the RAS coordinate system. To maximize the preservation of the structural features of lung nodules in space, this step also includes performing two-dimensional slicing of the preprocessed three-dimensional data along the axial, coronal, and sagittal planes, respectively, along the three orthogonal directions of the RAS coordinate system. Preferably, the size of all two-dimensional slices is uniformly adjusted to 512×512 pixels to meet the input requirements of subsequent deep learning models. It is understood that the specific size can be set according to actual needs, and this invention does not impose any limitations.

[0055] During the slicing process, the original pixel spacing and layer thickness must be kept constant to ensure accurate reconstruction of the subsequent two-dimensional segmentation mask in three-dimensional space, thereby accurately reproducing the true three-dimensional structural information of the lung nodules.

[0056] The final three-dimensional lung nodule segmentation results were obtained by stacking two-dimensional segmentation masks in reverse order along the original slice direction. In the subsequent three-dimensional reconstruction process, each two-dimensional segmentation mask was placed back into its corresponding position in three-dimensional space according to its original slice position and direction. The reverse stacking ensured that the coordinate system and direction of the reconstruction results were consistent with those of the original three-dimensional data, ensuring the accuracy and comparability of the three-dimensional segmentation results, and providing accurate and reliable basic data support for further quantitative analysis and clinical diagnosis.

[0057] Step S3: Use the U-Net network model to perform independent segmentation prediction on the 2D slices, and extract the foreground region predicted by U-Net as bounding box cue information and input it into the SAM2 model; In medical image segmentation tasks, U-Net is a classic and efficient deep convolutional network structure. U-Net is responsible for performing preliminary segmentation prediction on the slice data. Its feature is that it captures the global contour and structural information of the image and has good preliminary localization ability.

[0058] During training, the aforementioned slice data containing tumors (only a small portion) is used as input, paired with corresponding 2D labeled masks as supervision signals. Common training details include: data augmentation (e.g., random rotation, translation, scaling) to increase sample diversity; parameter updates using the Adam optimizer; and setting appropriate learning rates (1e-4) and training epochs (100 EPOCHS). Regarding the loss function selection, medical image segmentation often suffers from class imbalance (foreground pixels are far fewer than background pixels). Therefore, the Dice loss function is used to measure the similarity between the predicted results and the ground truth annotations. Based on the Dice coefficient (DSC), the Dice loss effectively ignores most background pixels by training only on slice data containing tumors, thus alleviating the foreground-background imbalance problem and accelerating convergence.

[0059] The segmentation results of the U-Net network are used to automatically generate prompts for the SAM2 model, achieving zero-shot segmentation of CT images. The output of the U-Net network is a two-dimensional segmentation mask in which each pixel is labeled with a category label, typically a binary matrix (0 and 1), where 1 indicates that the pixel belongs to the foreground (i.e., the lung nodule region), and 0 indicates the background (non-lung nodule region). This segmentation mask is obtained based on the analysis and learning of the input image by U-Net, and it initially locates the lung nodules in the image, but there may be some blurring or incompleteness in the boundary details.

[0060] Specifically, the foreground region predicted by U-Net is extracted as a bounding box cue and fed into the SAM2 model. The bounding box plays the following role in this process: Locating the target region: The bounding box provides the approximate location and extent of the target object or region in the image, which helps the SAM2 model quickly locate the region to be segmented, improving segmentation efficiency.

[0061] Reduced search area: By providing bounding box cues, the SAM2 model can perform segmentation operations within a defined region, reducing the computational cost of searching for targets across the entire image and thus improving processing speed.

[0062] Providing preliminary shape information: The shape and size of the bounding box can provide the SAM2 model with some preliminary information about the shape and size of the target object, which helps the model generate more accurate segmentation results.

[0063] Step S4: The SAM2 model receives foreground region bounding box cues predicted by the U-Net model, generates a fine-grained segmentation mask on the corresponding slice, refines the target region, and obtains details and contour information of the lung nodules. Segment Anything Model 2 (SAM2) is an upgraded version of the original SAM model. SAM2 has significant zero-shot generalization ability and can work on unseen image data without additional training. It uses a promptable architecture: by inputting different types of cues (such as dot cues, box cues, or even text descriptions), corresponding object masks can be generated. Compared with traditional dedicated segmentation networks, the advantage of SAM2 is that it does not require fine-tuning for specific tasks; segmentation can be performed simply by designing appropriate cues.

[0064] SAM2, based on the initial segmentation of U-Net, performs more refined segmentation using cue information as input.

[0065] After receiving this prompt, SAM2 can generate a more refined segmentation mask on the corresponding slice, thereby refining the target region and accurately capturing the details and contours of the lung nodules. The segmentation mask output by SAM2 is filtered by confidence, with a confidence threshold of 0.5, retaining only high-confidence regions to ensure the accuracy and reliability of the output mask.

[0066] Step S4.1: Determine the difference between the SAM2 output and the U-Net output, and calculate the comprehensive similarity C1. Calculate the difference: After the SAM2 model generates the fine-grained segmentation mask, compare the SAM2 output with the U-Net output and calculate their difference. The difference can be calculated in various ways, such as the Dice coefficient, IoU (Intersection over Union), and Euclidean distance. The Dice coefficient measures the degree of overlap between the two segmentation results, and IoU calculates the ratio of the intersection to the union of the segmentation results.

[0067] The Dice coefficient and IoU are normalized to the interval [0,1]. The normalization formula is: normalized value = (current value - minimum value) / (maximum value - minimum value).

[0068] We use a weighted summation method to merge the normalized difference indexes into a comprehensive similarity coefficient.

[0069] The fusion formula is: comprehensive similarity. .

[0070] in , These are the weighting coefficients, and The weighting coefficients can be adjusted according to the actual application scenario and the degree of importance attached to different difference indicators. The Dice coefficient and IoU are similarity indicators. The larger the value, the more overlap there is between the two sets (such as segmentation results), the closer the model output is, and the larger the corresponding C1 value.

[0071] Step S4.2: Set a comprehensive similarity threshold T. If the comprehensive similarity C1 is less than this threshold, proceed to step S4.3; otherwise, proceed to step S5.

[0072] Based on the actual application scenario and the characteristics of the dataset, a reasonable comprehensive similarity threshold is set. Preferably, the comprehensive similarity threshold T can be set to a range of [0.8-0.95]. If the comprehensive similarity is less than this threshold, it is considered that the output results of SAM2 and U-Net have a large difference in this region, indicating that the two models have significant inconsistencies in the segmentation results, and further optimization is required in step S4.2; otherwise, the subsequent fusion process continues.

[0073] Step S4.3: Input the segmentation mask output by the SAM2 model back into the U-Net model. The U-Net model receives the two-dimensional mask prompt information output by the SAM2 model and generates the iterative two-dimensional segmentation mask. In this step, the two-dimensional mask and the image are stitched together by channel stitching. In addition, when using U-Net in step S3, a full white mask is used to ensure that the model can be reused.

[0074] When the outputs of the SAM2 model and the U-Net model differ significantly, it's necessary to feed back into U-Net for optimization. The reasons are as follows: A large difference indicates that at least one model has significant segmentation errors. Feeding back allows U-Net to relearn using the feedback from SAM2, correcting segmentation results, reducing errors, and thus improving segmentation accuracy. When dealing with complex structures (such as lung nodules with blurred boundaries or irregular shapes), a single model may struggle to segment accurately; a large difference may indicate missed or false positives. Feeding back into U-Net allows it to re-examine these regions, reducing false positive and false negative rates and improving detection accuracy.

[0075] The feedback process allows U-Net to reference the results of SAM2, integrate more feature information, enhance its understanding and segmentation capabilities of complex structures, reduce segmentation failures caused by image noise and other factors, and improve the stability and reliability of the entire system. Furthermore, the feedback information from SAM2 can provide U-Net with additional feature guidance, helping it to better extract and utilize key features in images, further enhancing its feature representation capabilities.

[0076] Through this reverse process, a closed-loop optimization system is formed between the SAM2 and U-Net models, enabling them to collaborate and complement each other, jointly improving the accuracy and reliability of the segmentation results. This bidirectional optimization mechanism effectively addresses the error accumulation problem that may occur in cascaded models, and can significantly improve segmentation performance, especially when dealing with complex lesions and low-contrast regions.

[0077] Step S5: Determine the overall similarity C1 between the outputs of the U-Net model and the SAM2 model, as well as the confidence level of the model outputs, and perform 3D reconstruction using different 2D masks.

[0078] For details, please see the appendix. Figure 3 The detailed discrimination logic is described. In step S5, when the first condition is met, the two-dimensional mask output in step S4 is used for three-dimensional reconstruction; when the second condition is met, the bounding box cue information output by the U-Net network model in step S3 and the two-dimensional mask output by the SAM2 model in step S4 are used for three-dimensional reconstruction respectively, and the prediction results of different view spaces are fused at the voxel level, and multi-view three-dimensional mask fusion is completed based on uncertainty weighting; if the third condition is met, the iterated two-dimensional segmentation mask output by the U-Net network model in step S4.3 and the two-dimensional mask output by the SAM2 model in step S4 are used for three-dimensional reconstruction respectively, and the prediction results of different view spaces are fused at the voxel level, and multi-view three-dimensional mask fusion is completed based on uncertainty weighting.

[0079] Specifically, the first condition is: the comprehensive similarity C1 is greater than or equal to the threshold T, and the foreground region confidence index Z of the SAM2 model in step S4 is greater than or equal to the threshold T1. Preferably, the value range of the threshold T is [0.8-0.95], and the value range of the threshold T1 is [0.85-0.95]. Then, the output result of SAM2 in step S4 is used to directly perform three-dimensional reconstruction, thereby improving processing efficiency.

[0080] Specifically, the confidence index Z is the mean of the average confidence level Z1 of the foreground region and the median confidence level Z2 of the foreground region.

[0081] Foreground region average confidence Z1: The average confidence is calculated only for foreground pixels, and the formula is: ; where N is the total number of foreground pixels, and Confidence(i) is the confidence level of the i-th pixel.

[0082] Foreground region confidence median Z2: This is the median of the confidence scores of all foreground region pixels. The median is not sensitive to outliers and can reflect the middle level of the overall confidence score.

[0083] The confidence index Z combines the advantages of average confidence and median confidence. It considers both the overall average confidence level and the distribution of confidence levels. The median is robust to outliers and is not easily affected by extremely high or low confidence values. By incorporating the median, the confidence index Z is more stable in evaluation and can better represent the confidence level of most pixels.

[0084] When the first condition is met, the output of the SAM2 model is relatively close to the output of the U-Net model. Furthermore, considering the confidence index of the SAM2 model, the two-dimensional mask generated by the SAM2 model in step S4 is directly used for three-dimensional reconstruction. This avoids indiscriminate fusion processing of all cases, reduces unnecessary computational resource consumption, and improves the operating efficiency of the system.

[0085] The second condition is that the comprehensive similarity C1 is greater than or equal to the threshold T, and the foreground region confidence index Z of the SAM2 model in step S4 is less than the threshold T1. When the second condition is met, step S4.3 is not triggered. Three-dimensional reconstruction is performed on the two-dimensional masks output by the U-Net network model in step S3 and the SAM2 model in step S4, respectively. The prediction results from different viewpoints are fused at the voxel level, and multi-view three-dimensional mask fusion is completed based on uncertainty weighting.

[0086] The specific fusion process is as follows: After obtaining the segmentation results of each slice, the two-dimensional segmentation mask needs to be integrated back into the three-dimensional volume, and the fine two-dimensional mask results from three perspectives (axial plane, coronal plane and sagittal plane direction) are back-projected into the three-dimensional space.

[0087] This step further improves the accuracy and robustness of lung cancer segmentation by fully combining the advantages of both U-Net and SAM2 models. Specifically, it fuses results based on the uncertainty of model predictions to achieve more accurate identification and localization of lung cancer lesions. Figure 4 As shown, the detailed implementation logic and specific steps are as follows: Step S5.1 Perform three-dimensional reconstruction of the two-dimensional slice mask of each model in different spatial directions.

[0088] After completing independent predictions for the two models, 3D reconstruction was performed on the 2D slice masks of each model in different spatial directions (axial, coronal, and sagittal). The slice masks in each direction were then stacked back into 3D space. This step resulted in a total of six sets of 3D segmentation masks: three sets of 3D masks for the U-Net model (corresponding to the axial, coronal, and sagittal planes); and three sets of 3D masks for the SAM2 model (corresponding to the axial, coronal, and sagittal planes).

[0089] This process yielded spatial prediction results from two models at different perspectives, providing rich information on the three-dimensional structure of the lesions.

[0090] Step S5.2 Perform independent uncertainty estimation on the prediction results of U-Net and SAM2 models.

[0091] To achieve accurate fusion of model prediction results, the uncertainty of the prediction results of U-Net and SAM2 models is estimated independently. The specific logic is as follows: During the model inference stage, Monte Carlo Dropout or similar methods are used to obtain the probability distribution of each model's prediction by making multiple predictions for each model.

[0092] For each 3D spatial location (i.e., voxel), the stability of the prediction result at that location across multiple predictions is analyzed. Voxels with small changes in prediction results indicate high reliability of the model prediction, i.e., low uncertainty; conversely, voxels with large changes in prediction results represent higher uncertainty of the model prediction. The above uncertainty assessment is performed on the prediction results of U-Net and SAM2 respectively, obtaining independent voxel-level uncertainty information for the two models. This step allows us to understand the reliability of each model's predictions in different spatial regions, facilitating further refined fusion.

[0093] Step S5.3 Perform voxel-level fusion of the prediction results of U-Net and SAM2 in different view spaces, and complete multi-view 3D mask fusion based on uncertainty weighting.

[0094] After mastering the prediction masks and corresponding uncertainty information of the two models, we further carry out uncertainty-based weighted fusion. The logical steps are as follows: voxel-level fusion of the prediction results of U-Net and SAM2 in different view spaces, and the weight allocation of each voxel position is determined according to the uncertainty of the two models during fusion.

[0095] Specifically, for any voxel location, the model prediction result with lower uncertainty has a higher weight, while the model prediction result with higher uncertainty has a correspondingly lower weight, and the total weight is 1. The above uncertainty-weighted fusion is performed in multiple view directions (axial plane, coronal plane, sagittal plane) to obtain a more refined and stable spatial fusion result.

[0096] This step effectively integrates the advantages of both models: U-Net's global structure prediction advantage (coarsely accurate localization of lesion areas); and SAM2's fine boundary capture advantage (precise delineation of lesion edges), generating a final, highly accurate 3D mask of the lung nodule. Further, the volume and morphological parameters of the lung nodule (such as sphericity and surface area-to-volume ratio) are calculated to assist in the clinical diagnosis and staging assessment of lung cancer; this will not be elaborated further in this embodiment.

[0097] When the second condition is met, the advantages of both U-Net and SAM2 models can be fully utilized. Through uncertainty-weighted fusion, the accuracy of the segmentation results is further improved. It can adapt to cases of different complexities. For complex or difficult-to-segment lesion areas, more accurate segmentation results can be obtained through the fusion process. It can effectively reduce the risk of misjudgment caused by single model judgment and improve the overall performance of the system.

[0098] The third condition is that the comprehensive similarity C1 is less than the threshold T. After step S4.3, the U-Net model outputs an iterated two-dimensional segmentation mask. Three-dimensional reconstruction is then performed on both the iterated two-dimensional segmentation mask output by the U-Net network model in step S4.3 and the two-dimensional mask output by the SAM2 model in step S4. The prediction results from different viewpoints are fused at the voxel level, and multi-view three-dimensional mask fusion is completed based on uncertainty weighting. The specific fusion process is the same as described in steps S5.1-S5.3 above, and will not be repeated here.

[0099] To enhance users' intuitive understanding of the segmentation results, this embodiment also incorporates a confidence visualization mechanism. This mechanism provides real-time, visual feedback to clinical users on the model's predictive reliability for each region. The specific implementation is as follows: Confidence levels can be standardized and mapped to grayscale or percentage levels to form a confidence map. This confidence map is then overlaid with the original CT image or the final segmentation mask, providing users with a clear and intuitive decision-making reference. The overlay method is as follows: high-confidence regions are highlighted in green, and low-confidence regions are marked in red. Users can dynamically adjust the confidence threshold on the visualization interface, allowing for free selection and real-time updates to the display. Preferably, a confidence level greater than or equal to 80 can be defined as a high-confidence region. Users can also click or zoom in on regions of interest to precisely view the model's distribution.

[0100] The high / low confidence levels are visually represented by color-coded distinctions, clearly indicating the model's confidence in the segmentation results for each region. Dynamic threshold adjustment and localized viewing are supported, facilitating personalized use and precise analysis by physicians. Low confidence markers help physicians identify the locations where segmentation errors are most likely to occur, prompting manual review or secondary interpretation. High confidence coverage areas enhance confidence in the segmentation results and help physicians quickly focus on key lesions.

[0101] Example 2: Figure 5 This is a schematic diagram of the medical image segmentation system provided in Embodiment 2 of this application, specifically including: a 3D data preprocessing module: used to convert 3D CT image data into NIfTI format and perform grayscale normalization and orientation correction; a slice generation module: used to extract 2D slices along three orthogonal directions of the RAS coordinate system; a U-Net segmentation module: used to perform segmentation prediction on the 2D slice data and generate bounding box prompts for input to SAM2; a SAM2 fine segmentation module: used to receive bounding box prompts and generate fine segmentation masks; and a 3D reconstruction and fusion module: used to perform 3D reconstruction and fusion on the multi-view 2D segmentation masks.

[0102] Example 3: Figure 6 This is a schematic diagram of the electronic device provided in Embodiment 3 of this application. Figure 6 As shown, the electronic device 4 in this embodiment includes a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, it implements the steps described in the various method embodiments above. The electronic device 4 can be a desktop computer, tablet computer, mobile terminal, cloud server, or other computing device. The electronic device may include, but is not limited to, the processor 40 and the memory 41. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0103] The processor 40 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0104] The memory 41 can be an internal storage unit of the electronic device 4, such as a hard disk or memory. The memory 41 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 41 can include both internal and external storage units of the electronic device 4. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0107] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A medical image segmentation method, characterized in that, Includes the following steps: Step S1: Preprocess the 3D CT data images; Step S2: Extract two-dimensional slices from the preprocessed data along three orthogonal directions of the RAS coordinate system from multiple perspectives; Step S3: Use the U-Net model to perform independent segmentation prediction on the two-dimensional slice, and extract the foreground region predicted by the U-Net model as bounding box cue information and input it into the SAM2 model; Step S4: The SAM2 model receives the foreground region bounding box cue information predicted by the U-Net model, generates a fine two-dimensional mask on the corresponding slice, refines the target region, and obtains the details and outline of the lung nodules. Step S4.1: Determine the difference between the SAM2 output and the U-Net output, and calculate the overall similarity C1; Step S4.2: Set a comprehensive similarity threshold T. If the comprehensive similarity C1 is less than this threshold, proceed to step S4.3; otherwise, proceed to step S5. Step S4.3: Input the two-dimensional mask output by the SAM2 model in step S4 back into the U-Net model. The U-Net model receives the two-dimensional mask prompt information output by the SAM2 model, generates the iterative two-dimensional segmentation mask, and executes step S5. Step S5: Determine the overall similarity C1 between the outputs of the U-Net model and the SAM2 model, as well as the confidence level of the model outputs, and perform 3D reconstruction using different 2D masks.

2. The method according to claim 1, characterized in that, The calculation process for the comprehensive similarity C1 is as follows: The overlap, intersection-overlap ratio, and spatial distance of the mask results of the two models are evaluated using three metrics: Dice coefficient, IoU, and Euclidean distance. Each metric is standardized and mapped to a unified interval. The standardized three metrics are weighted and summed according to preset weights to obtain the comprehensive similarity C1, which measures the overall difference between the segmentation results of the two models.

3. The method according to claim 1, characterized in that, In step S5, if the first condition is met, the two-dimensional mask output in step S4 is used for three-dimensional reconstruction; if the second condition is met, the bounding box cue information output by the U-Net network model in step S3 and the two-dimensional mask output by the SAM2 model in step S4 are used for three-dimensional reconstruction respectively, and the prediction results of different view spaces are fused at the voxel level, and multi-view three-dimensional mask fusion is completed based on uncertainty weighting; if the third condition is met, the iterated two-dimensional segmentation mask output by the U-Net network model in step S4.3 and the two-dimensional mask output by the SAM2 model in step S4 are used for three-dimensional reconstruction respectively, and the prediction results of different view spaces are fused at the voxel level, and multi-view three-dimensional mask fusion is completed based on uncertainty weighting.

4. The method according to claim 3, characterized in that, In step S5, the first condition is that the comprehensive similarity C1 is greater than or equal to the threshold T, and the foreground region confidence index Z of the SAM2 model in step S4 is greater than or equal to the threshold T1.

5. The method according to claim 4, characterized in that, In step S5, the second condition is: The overall similarity C1 is greater than or equal to the threshold T, and the foreground region confidence index Z of the SAM2 model in step S4 is less than the threshold T1.

6. The method according to claim 4, characterized in that, The confidence index Z for the foreground region is the mean of the average confidence index Z1 and the median confidence index Z2 for the foreground region.

7. The method according to claim 3, characterized in that, The specific process of completing the multi-view 3D mask fusion step based on uncertainty weighting includes: Step S5.1: Perform 3D reconstruction of the 2D masks of U-Net and SAM2 models in different spatial directions; Step S5.2: Perform independent uncertainty estimation on the prediction results of the U-Net and SAM2 models; Step S5.3: Perform voxel-level fusion of the prediction results of U-Net and SAM2 in different view spaces, and complete multi-view 3D mask fusion based on uncertainty weighting.

8. The method according to claim 6, characterized in that, The uncertainty estimation is achieved through the Monte Carlo Dropout method, which includes: for each three-dimensional spatial location, i.e., voxel, analyzing the stability of the prediction result at that location in multiple predictions. Voxels with small changes in prediction results indicate high reliability of the model prediction, i.e., low uncertainty, while voxels with large changes in prediction results indicate high uncertainty of the model prediction; the above uncertainty estimation is performed on the prediction results of U-Net and SAM2 respectively to obtain independent voxel-level uncertainty information for the two models.

9. A medical image segmentation system for implementing the medical image segmentation method as described in any one of claims 1-8, characterized in that, include: 3D data preprocessing module: used to convert 3D CT image data into NIfTI format and perform grayscale normalization and orientation correction; The slice generation module is used to extract two-dimensional slices along the three orthogonal directions of the RAS coordinate system; The U-Net segmentation module is used for segmentation prediction of two-dimensional slice data; The SAM2 fine segmentation module is used to receive bounding box prompts and generate a fine two-dimensional mask; The 3D reconstruction and fusion module is used to perform 3D reconstruction and fusion of 2D segmentation masks from multiple perspectives.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 8.

Images