Human tissue CT image three-dimensional reconstruction method based on SAM series model

Abstract

The invention discloses a human tissue CT image three-dimensional reconstruction method based on an SAM series model, and the method comprises the steps: collecting a CTSpine1K data set and a query set image provided by a cooperation hospital, and carrying out the screening and cutting of a support set and a query set, so as to meet the input requirements of a model; extracting features of the support set and the query set, and performing feature matching; segmenting the image according to a point, a frame or a mask prompt provided by a user to generate a segmentation mask; a segmentation result is evaluated, and images with excellent performance are added into a query set; the method comprises the following steps of: storing and utilizing embedding information of a past frame through a memory encoder to generate a segmentation mask; performing three-dimensional reconstruction based on the continuous image sequence with the segmentation mask; and collecting the feedback of the doctor user to the three-dimensional reconstruction result. According to the invention, through a small sample learning technology, the model can be trained only through a small amount of annotation data, and the cost and time for obtaining and annotating a large number of medical images are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and in particular to a three-dimensional reconstruction method of human tissue CT images based on SAM series models. Background Art

[0002] In the field of medical image analysis, deep learning technology has become an important tool for image segmentation and 3D reconstruction, and is widely used in disease diagnosis, treatment planning, and surgical navigation. Traditional deep learning models, such as convolutional neural networks (CNNs), rely on large-scale annotated datasets to optimize model parameters by training on large amounts of data, thereby achieving high-precision image segmentation and 3D reconstruction. However, obtaining large-scale, high-quality annotated medical image datasets is significantly challenging.

[0003] First, the acquisition and annotation process of medical images usually requires professional medical knowledge and rich clinical experience, which is not only time-consuming and labor-intensive, but also costly. Especially in the imaging data of some rare diseases or specific parts, labeled samples are even more scarce, making it difficult for existing deep learning models to achieve ideal performance in these areas. Secondly, medical imaging data is highly professional and complex. There may be significant differences between images from different devices, different scanning parameters, and even different batches of the same device, which increases the difficulty of model generalization.

[0004] In addition, although some advanced image segmentation models have emerged in recent years, such as the SegmentAnything Model (SAM) launched by Meta AI, which have performed well in general image segmentation tasks, they still face challenges in medical image segmentation. When applied to medical images, models such as SAM often require a lot of manual prompts and annotations to optimize the segmentation results, and their performance is highly dependent on the quality of prompts provided by users. For specific medical application scenarios, the SAM model may need to be adjusted and optimized in a targeted manner, further increasing its complexity and practical difficulty in clinical applications.

[0005] Existing technologies are not good at dealing with the scarcity of medical imaging data and the high cost of annotation, which limits the widespread application of deep learning models in medical image segmentation and 3D reconstruction. Traditional methods are difficult to train robust and generalizable models on limited annotated data, and the high cost and low efficiency of manual annotation further hinder the promotion of deep learning technology in clinical practice. Therefore, it is urgent to develop learning methods that can be effectively trained with a small amount of annotated data to reduce data dependence and improve the efficiency and accuracy of model application in medical image analysis. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention proposes a 3D reconstruction method for human tissue CT images based on the SAM series model. Users can adjust the prompt information in real time according to the segmentation results, and continuously improve the segmentation quality through iterative optimization to ensure that the final result meets clinical needs. The lightweight mask decoder and efficient feature extraction method are used to reduce the consumption of computing resources and processing time while ensuring segmentation accuracy, which is suitable for rapid deployment and use in clinical environments.

[0007] To achieve the above object, the present invention proposes a method for three-dimensional reconstruction of human tissue CT images based on the SAM series model, comprising:

[0008] Step 1: Collect the CTSpine1K dataset and query set images provided by the partner hospital, and filter and trim the support set and query set to match the model input requirements;

[0009] Step 2: Use the pre-trained ResNet-18 to extract features of the support set and query set, and perform feature matching through cosine similarity calculation;

[0010] Step 3: Load the pre-trained SAM v2 model, segment the image according to the point, box or mask prompts provided by the user, and generate a segmentation mask;

[0011] Step 4: Evaluate the segmentation results, add the images with good performance to the query set, and randomly update the support set to enhance the generalization ability of the model;

[0012] Step 5: Storing and utilizing the embedding information of past frames through the memory encoder, combined with the current hint to generate a more accurate segmentation mask;

[0013] Step 6: Based on the continuous image sequence with segmentation mask, 3D reconstruction is performed using techniques such as maximum density projection to generate an intuitive 3D model.

[0014] Furthermore, step 1 is as follows:

[0015] Step 1.1: Use the CTSpine1K dataset, which collects 1,005 spinal CT images from four public datasets and annotates 25 types of vertebrae, including C1-C7, T1-T12, and L1-L6;

[0016] Step 1.2: Randomly select 8 samples from the CTSpine1K dataset as the initial support set to ensure the diversity and representativeness of the samples;

[0017] Step 1.3: Select samples based on the similarity of spine parts to ensure that the images in the support set are highly relevant to the parts in the query set, thereby improving the performance of the model on specific tasks;

[0018] Step 1.4: Adjust the image resolution to match the model’s input requirements, mainly by cropping out the black background area that does not contain information to ensure that the image content is complete and does not lose key information;

[0019] Step 1.5: A total of 494 CT images of 5 patients and their corresponding labels were provided by the cooperating hospital;

[0020] Step 1.6: Perform the same screening and cropping operations on the query set images as the support set to ensure the consistency of the query set and the support set in image resolution and content for subsequent feature extraction and matching.

[0021] Furthermore, step 2 is as follows:

[0022] Step 2.1: Organize the support set images into a tensor of shape [Ns,3,H,W], where Ns represents the number of samples in the support set, 3 represents the RGB channels, and H and W are the height and width of the image respectively;

[0023] Step 2.2: Organize the query set images into a tensor of shape [Nq, 3, H, W], where Nq represents the number of samples in the query set;

[0024] Step 2.3: Use the pre-trained ResNet-18 model provided by the torchvision library in Python as the feature extractor;

[0025] Step 2.4: Forward propagate the support set image through ResNet-18, and after global pooling, obtain a feature vector of shape [Ns, C], where C is the dimension of the feature vector;

[0026] Step 2.5: Also use ResNet-18 to extract features of the query set image and obtain a feature vector of shape [Nq, C]. The reason for choosing ResNet-18 is that it can extract robust geometric features and is suitable for measuring the geometric similarity between images without relying on high-level semantic information.

[0027] Step 2.6: Use cosine similarity as an indicator to measure the similarity between the query set and the support set features;

[0028] Step 2.7: For each query set feature vector, calculate its cosine similarity with all support set feature vectors, and select the support set feature with the highest similarity as the matching result;

[0029] Step 2.8: Concatenate the matched support set image with the corresponding query set image to form a new image sequence of shape [Nq+1,3,H,W], and use the label corresponding to the support set as the mask required for segmentation.

[0030] Furthermore, step 3 is as follows:

[0031] Step 3.1: Select Segment Anything Model (SAM) v2 because it shows higher accuracy and processing speed in image segmentation tasks;

[0032] Step 3.2: Download and load the pre-trained SAM v2 model weights from the specified link;

[0033] Step 3.3: Use the default parameter settings of the model to ensure the stability and consistency of the model in the initial stage;

[0034] Step 3.4: Depending on the specific needs, the user can provide hints in the form of points, boxes, or masks to guide the SAM model to focus on the area of ​​interest for segmentation;

[0035] Step 3.5: Based on the initial prompt, the SAM model generates a preliminary segmentation mask;

[0036] Step 3.6: The user observes the segmentation results. If the results are not satisfactory, the user can adjust the position or form of the prompt, re-enter the prompt information, and perform multiple iterations to optimize the segmentation effect;

[0037] Step 3.7: Finally, an image with a segmentation mask is generated as the basic data for subsequent 3D reconstruction.

[0038] Furthermore, step 4 is as follows:

[0039] Step 4.1: Using the initially selected 8 support set images as the benchmark, perform a preliminary segmentation operation through the SAM model to generate an initial segmentation mask;

[0040] Step 4.2: Evaluate the segmentation results of each image in the query set and select the images with excellent segmentation results as the basis for further optimization;

[0041] Step 4.3: Add the segmentation result images with excellent performance after evaluation to the query set to form a more representative query set and improve the performance of the model in subsequent segmentation tasks;

[0042] Step 4.4: Randomly extract new samples from the support set and add them to the support set to enhance the generalization ability of the model and prevent overfitting to the initial samples;

[0043] Step 4.5: Call the SAM model multiple times to perform segmentation, evaluate the segmentation results, and update the query set and support set to gradually improve the segmentation accuracy and stability of the model;

[0044] Step 4.6: After each iteration, the model can use the latest query set and support set information to further optimize the segmentation effect, ensuring that the final generated segmentation mask has high accuracy and robustness.

[0045] Furthermore, step 5 is as follows:

[0046] Step 5.1: Combine the image embedding of the current frame and the prompt information provided by the user;

[0047] Step 5.2: The lightweight mask decoder generates the segmentation mask of the current frame based on the image embedding and hint information;

[0048] Step 5.3: The memory encoder creates a frame embedding based on the prediction result of the current frame and stores it in the memory bank;

[0049] Step 5.4: Get the embedding of each frame from the image encoder and combine it with the stored memory information to generate the segmentation mask of the current frame.

[0050] Furthermore, step 6 is as follows:

[0051] Step 6.1: Continuous image sequence with segmentation mask;

[0052] Step 6.2: Extract tissue or structure regions of interest using segmentation masks;

[0053] Step 6.3: Convert the two-dimensional slices into three-dimensional images using the maximum intensity projection (MIP) method;

[0054] Step 6.4: Apply the segmented 2D mask to the MIP algorithm to generate a 3D model;

[0055] Step 6.5: The generated 3D reconstruction model is used by doctors for clinical diagnosis and treatment planning.

[0056] Furthermore, the formula of step 2.7 is as follows:

[0057]

[0058] In which, given two vectors A and B, Ai and Bi represent the components of vectors A and B respectively.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention provides a three-dimensional reconstruction method for human tissue CT images based on the SAM series model. Through small sample learning technology, only a small amount of labeled data is needed to train the model, reducing the cost and time of acquiring and annotating a large number of medical images. Combining the pre-trained ResNet-18 and SAM v2 models, the cosine similarity is used for feature matching, so that high-precision segmentation results can be obtained even when data is scarce.

[0061] 2. The present invention provides a three-dimensional reconstruction method for human tissue CT images based on the SAM series models. It adopts an interactive segmentation method. The user only needs to provide simple prompts (such as points, boxes or masks) to guide the model to perform precise segmentation, which significantly reduces the tedious pixel-level labeling tasks; by dynamically updating the query set and support set, the model can continuously adapt to new data distributions and different medical images, thereby improving its applicability and stability in diversified clinical applications.

[0062] 3. The present invention provides a three-dimensional reconstruction method for human tissue CT images based on the SAM series model, which uses technologies such as maximum intensity projection (MIP) to quickly convert the segmented two-dimensional image sequence into an intuitive three-dimensional model to assist doctors in making more accurate clinical diagnoses and treatment plans.

[0063] 4. The present invention provides a three-dimensional reconstruction method for human tissue CT images based on the SAM series model. Under the background of data scarcity and high annotation costs, this method can still provide efficient and reliable image segmentation and three-dimensional reconstruction solutions, and promote the practical application of deep learning technology in medical image analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 It is a schematic diagram of the steps of the present invention

[0066] Figure 2 This is an example of a 4-way-1-shot classification task.

[0067] Figure 3 MIP schematic diagram

[0068] Figure 4 To organize the flow chart

[0069] Figure 5 This is a schematic diagram of the ResNet-18 structure

[0070] Figure 6 (a) and (b) are the comparison results before and after the support set size

[0071] Figure 7 It is a user experience process diagram DETAILED DESCRIPTION

[0072] The technical solution of the present invention will be more clearly and completely explained below through description of preferred embodiments of the present invention in combination with the accompanying drawings.

[0073] Technical terms:

[0074] 1. CTSpine1K dataset

[0075] CTSpine1K Dataset: This is a large-scale computed tomography (CT) dataset specifically for spine segmentation. The dataset collects a total of 1,005 spine CT images from four public datasets and annotates them in detail, including 25 types of spines, including C1-C7, T1-T12, and L1-L6.

[0076] 2. Support Set and Query Set

[0077] Support Set: In few-shot learning, the support set is a small number of labeled examples used for learning and building models.

[0078] Query Set: The query set is the sample that needs to be predicted or classified, usually unlabeled, and is used to evaluate model performance.

[0079] 3. ResNet-18

[0080] ResNet-18: A deep convolutional neural network (CNN) architecture that belongs to the Residual Network family. ResNet-18 contains 18 layers and is commonly used for image classification and feature extraction tasks due to its effective feature extraction capabilities and small number of parameters.

[0081] 4. Feature Vector

[0082] Feature Vector: A numerical representation extracted from input data by a neural network or other feature extraction method, used to describe the key features of the input data. In this method, the feature vector extracted by ResNet-18 is used to measure the similarity between images.

[0083] 5. Cosine Similarity

[0084] Cosine Similarity: A measure of the directional similarity between two vectors. The calculation formula is the dot product of the two vectors divided by the product of their moduli. The value range is between -1 and 1, where 1 means they are exactly the same, 0 means no correlation, and -1 means they are completely opposite. In this method, it is used to measure the similarity between the query set and the support set features.

[0085] 6.Segment Anything Model(SAM)v2

[0086] Segment Anything Model (SAM) v2: An image segmentation model launched by Meta AI, designed to handle a variety of segmentation tasks. SAM v2 is its second version, with enhanced support for videos and memory mechanisms, enabling more accurate and faster image and video segmentation.

[0087] 7. Model Weights

[0088] Model Weights: The parameters learned in a neural network after training, used to make predictions on new data. In this method, the pre-trained SAM v2 model weights are downloaded and loaded from the specified link for the image segmentation task.

[0089] 8. Maximum Intensity Projection (MIP)

[0090] Maximum Intensity Projection (MIP): A 3D image reconstruction technique commonly used in medical imaging. MIP traverses the volume data along a specific direction and selects the voxel with the maximum intensity value on the path as the final display value for each pixel point, highlighting high-contrast structures such as blood vessels and bones.

[0091] like Figure 1 As shown, the present invention is specifically:

[0092] Step 1: Collect the CTSpine1K dataset and query set images provided by the partner hospital, and filter and trim the support set and query set to match the model input requirements;

[0093] Step 2: Use the pre-trained ResNet-18 to extract features of the support set and query set, and perform feature matching through cosine similarity calculation;

[0094] Step 3: Load the pre-trained SAM v2 model, segment the image according to the point, box or mask prompts provided by the user, and generate a segmentation mask;

[0095] Step 4: Evaluate the segmentation results, add the images with good performance to the query set, and randomly update the support set to enhance the generalization ability of the model;

[0096] Step 5: Store and utilize the embedding information of past frames through the memory encoder and combine it with the current prompt to generate a segmentation mask;

[0097] Step 6: Perform 3D reconstruction based on the continuous image sequence with segmentation mask to generate an intuitive 3D model;

[0098] As a specific implementation method:

[0099] Excellent deep learning models usually rely on large-scale labeled data sets. Models are repeatedly trained and optimized on large amounts of data and have strong analytical capabilities to give robust results on new problems. In the medical field, due to the lack of annotated medical images, the performance of deep learning models is highly limited and cannot provide the main force for solving key and difficult problems in the medical field. In this case, Few Shot Learning (FSL) can reduce the problem of data scarcity and improve the speed and robustness of medical image analysis.

[0100] Definition: Let x represent input data, y represent supervision information, X and Y represent the space of input data and supervision information respectively. Given a dataset D specific to task T containing a small amount of available supervised information T and an auxiliary dataset D that is not related to T A The goal of small sample learning is to construct a function f for task T, which is accomplished by using D T There is little supervision information and D A The knowledge in the above definition completes the task of mapping the input to the target. The terms not related to T in the above definition represent D T and D A The categories in are orthogonal, that is If DA covers the tasks in T, that is, Y T ∩Y A =Y T , then the FSL problem will collapse into the traditional large-sample learning problem.

[0101] According to the work, FSL methods are divided into FSL based on meta-learning and FSL based on non-meta-learning. The three main meta-learning methods are: metric-based, optimization-based, and model-based meta-learning. In addition, variants of the FSL problem using meta-learning are classified as hybrid methods. In this study, we used the basic paradigm of metric learning, and we also focused on the small sample learning method based on metric learning to complete the segmentation task.

[0102] Metric Learning: is the task of learning distance functions on data samples. Consider two image label pairs (x1, y1) and (x2, y2) and a distance function d to measure the distance between the two images. If we want to assign a label to a query image x3, we can calculate two distances d(x1, x3) and d(x2, x3) and assign a shorter label to the image corresponding to the distance, which is also the key idea of ​​the K-Nearest Neighbor (k-NN) algorithm. However, for high-dimensional inputs such as images, an embedding function g is usually used to convert the input to a lower dimension before calculating the distance.

[0103] The training process is performed by randomly sampling M-way-K-shot episodes from the training set, each of which has a support set and a query set. The average error calculated on the query set of multiple training few-shot events is used to update the parameters of the embedding function and the distance function (if any). Finally, new M-way-K-shot episodes are sampled from the test set to evaluate the performance of the network.

[0104] like Figure 2 As shown, the embedding function g outputs the embedding vectors of the support image (labeled) and the query image (unlabeled, represented by "?"). The distance function d measures the distance between the support and query vectors to output a similarity score. The category of the query image is close to the category with the highest toe similarity.

[0105] This paper mainly uses SAM v2. Before introducing SAM v2, we need to briefly mention SAM v1, which is a basic model launched by Meta AI and designed for image segmentation tasks. It aims to handle a variety of different segmentation tasks through a general model without the need for special training for each specific application scenario. The architecture of SAM mainly includes the following key modules:

[0106] 1. Image Encoder: This module is responsible for converting the input image into feature representation. It uses a pre-trained Convolutional Neural Network (CNN) to extract high-level features from the image.

[0107] 2. Hint encoder: used to encode hint information provided by the user, such as points, boxes, or masks. These hints can be interactive, and users can provide additional information to guide the model to generate more accurate segmentation results.

[0108] 3. Lightweight decoder: Combines the outputs from the image encoder and the hint encoder to produce the final segmentation mask. The decoder is designed with efficiency in mind, allowing the model to run quickly on a variety of devices.

[0109] Although SAM v1 solves the problem that traditional image segmentation methods require a large amount of labeled data, high-quality segmentation results can be obtained through a small amount of interaction, and the flexibility of image segmentation is improved, allowing it to be applied to new fields or objects without retraining. However, in some specific applications, it still requires a certain number of manually labeled samples to optimize model performance. In fact, the performance of the model is highly dependent on the quality of the prompts provided; if the prompts are inaccurate or unclear, it may affect the final results.

[0110] v2 can be seen as a generalization of v1 to the video (and image) domain. SAM2 supports point, box, and mask hints on single pixel frames to define the spatial extent of objects to be segmented in videos. For image inputs, the model behaves similarly to SAM. A hintable lightweight mask decoder takes in a frame embedding and hints (if any) on the current frame and outputs a segmentation mask for that frame. Hints can be iteratively added on frames to refine the mask. Unlike SAM, the frame embeddings used by the SAM 2 decoder do not come directly from the image encoder, but are based on a memory of past predictions and hint frames. It is also possible that the hint frame comes from "from the future" relative to the current frame. The memory of a frame is created by the memory encoder based on the current prediction and placed in the memory bank for use in subsequent frames. The memory attention operation takes the embedding of each frame from the image encoder and sets it in the memory bank to produce an embedding that is then passed to the mask decoder.

[0111] In such an overall framework, coupled with a real-time video processing pipeline with streaming memory, SAM 2, trained on the above data, provides strong performance in a wide range of tasks. In video segmentation, the model directly presents better accuracy, using 3 times fewer interactions than previous methods. In image segmentation, v2 is also more accurate and faster than v1.

[0112] Maximum Intensity Projection (MIP) is a 3D image reconstruction technique commonly used in medical imaging, such as Figure 3 This technology can transform a series of two-dimensional slice data into an intuitive three-dimensional image.

[0113] In the MIP process, for each pixel, the algorithm traverses the entire volume data set along a specific direction (usually the line of sight) and selects the voxel with the maximum intensity value on this path as the final display value of the pixel. This method is particularly suitable for highlighting high-contrast structures, such as contrast agents in blood vessels, bones, etc., because it can emphasize the brightest parts and ignore background information.

[0114] The advantages of MIP include:

[0115] 1. Improve contrast: By selecting the brightest or strongest signal point at each viewing angle as the display object, MIP can significantly enhance the contrast between high-density structures such as blood vessels and bones and surrounding tissues, making these structures more clearly visible.

[0116] 2. Easy to identify abnormalities: For detecting tumors, bleeding spots or other pathological changes, MIP can quickly highlight target areas with higher density characteristics, helping doctors to detect potential problems earlier.

[0117] 3. Intuitive display of anatomical structure: Compared with the traditional slice browsing method, the images generated by MIP provide a more three-dimensional and intuitive visual effect, which makes it easier for observers to understand the complex internal structure and its interrelationships.

[0118] Experimental data:

[0119] Supported source data: CTSpine1K dataset. CTSpine1K is a large-scale CT dataset dedicated to spine segmentation. This dataset collects a total of 1005 CT data from four public datasets, and annotates 25 types of vertebrae including C1-C7, T1-T12, and L1-L6. It should be noted that most patients do not have L6 spines, so this type of data is relatively rare and rich in content.

[0120] Query source data: provided by the cooperating hospital, a total of 5 people, 494 images and their corresponding labels

[0121] The experimental process is as follows Figure 4 As shown,

[0122] Data preprocessing

[0123] Support set preparation: 8 samples are randomly selected from the CTSpine1K dataset as the initial sample library, and then appropriately screened and cropped. Screening is mainly based on the similarity of the parts. The closer the query set is to the initial sample library, the better the effect. Cropping is to change the image resolution to match the model input requirements. There will be no information loss in this part because the main processing and deletion is the black background without information. The query set is processed in the same way.

[0124] Few-shot learning framework

[0125] The composite features obtained from preparing the query set and test set to similarity calculation are the small sample learning framework mentioned in our previous method.

[0126] 2.1 First prepare the support set and query set. In a batch, they are [Ns, 3, H, W] and [Nq, 3, H, W] respectively.

[0127] 2.2 Embedding function: Here we use the pre-trained model ResNet-18 provided by torchvision in Python, such as Figure 5 As shown in the figure, feature extraction is completed. Since similarity calculation requires robust geometric similarity (robust to geometric transformation and noise) rather than high-level semantic similarity, a small-scale, non-medical image pre-trained ResNet after global pooling is sufficient. Here, the features are converted from [Ns, 3, H, W] and [Nq, 3, H, W] to [Ns, C] and [Nq, C], respectively, where C is the length of the feature vector corresponding to a single image. This form is also used for similarity calculation.

[0128] 2.3 Distance function: The widely used cosine similarity is used here. Cosine similarity is a commonly used method to measure the similarity between two vectors. It determines whether their directions are similar by calculating the cosine value of the angle between the two vectors, regardless of the length (size) of the vectors. The value range of cosine similarity is between -1 and 1, where 1 means exactly the same, 0 means no correlation, and -1 means completely different.

[0129] Given two vectors A and B, the formula for cosine similarity is as follows:

[0130]

[0131] Among them, Ai and Bi represent the components of vectors A and B respectively. Therefore, for a batch of query set features, the support set feature with the highest similarity can always be found. Then the corresponding source images are spliced ​​together to form a new image sequence [Nq+1,3,H,W]. The label corresponding to the support set individual is the mask, MS required by the experiment. Since the image in CT itself is a continuous sequence, it can be considered as a special "video", and the SAM series model has added the processing capability of the video, so naturally this patent uses the SAM model to process the obtained robust image sequence and mask to obtain better segmentation results. The support set scale comparison effect is shown in Figure 6 (a) (b).

[0132] In the specific operation, the query set is based on the first 8 selected pictures, and is gradually adjusted as the user calls the model multiple times. The adjustment method is as follows: after the segmentation results of the query set pictures are obtained, a batch of pictures with better segmentation results are selected and added to the query set, and the subsequent support sets are randomly selected to enhance generalization. Compared with the use of larger models and more data in traditional model tuning, the small sample learning framework is undoubtedly less stringent in terms of the amount of data after stabilizing sufficient performance, which reduces the difficulty of AI implementation.

[0133] Image segmentation using the SAM series model

[0134] In this step, the SAM model can identify and segment the region of interest based on the user's prompts. These prompts can be in the form of points, boxes, or masks. The user needs to select the appropriate prompt type according to specific needs and mark it on the image. Furthermore, the user observes the output effect, adjusts the position or form of the prompt, and tries different settings multiple times to obtain a better segmentation effect.

[0135] 3D reconstruction

[0136] After obtaining the image with the segmentation mask processed by the SAM model, the process will automatically perform 3D reconstruction, obtain relevant results, and return them to the doctor user, providing a good clinical basis for the next treatment of the patient. If the user is not satisfied with the final result, the user can re-label the query set. More detailed labeling will make it easier to get better results.

[0137] Experience process like Figure 7 As shown;

[0138] In the present invention, labeling does not mean directly classifying each pixel to obtain a perfect label image. Simply selecting a rough area is also effective, which greatly reduces the energy consumption and professional requirements of users. When users are not satisfied, they can calibrate the previous label more carefully, and this round of effective interaction can get better results.

[0139] As a specific implementation, in this implementation, 8 spinal CT images are randomly selected from the CTSpine1K dataset as the initial support set. These images are screened to ensure similarity with the query set parts, and the image resolution is adjusted by cropping to match the model input requirements, and the black background without information is removed. Subsequently, 494 CT images provided by the cooperative hospital are used as the query set, and the same screening and cropping process is performed to ensure the consistency of the support set and the query set in image resolution and content.

[0140] Next, we build a small sample learning framework and use the pre-trained ResNet-18 model to extract features of the support set and query set images, and obtain feature vectors of shape [Ns, C] and [Nq, C], respectively. By calculating the cosine similarity between the query set features and the support set features, we find the most similar support image for each query image, and concatenate the matching support image with the query image to form a new image sequence, while retaining the label corresponding to the support set as the mask required for segmentation.

[0141] Then, the pre-trained Segment Anything Model (SAM) v2 model is loaded, and the spliced ​​image sequence is segmented according to the point, box or mask prompts provided by the user to generate an image with a segmentation mask. The user can adjust the position or form of the prompt information according to the segmentation result and repeatedly optimize the segmentation effect. The image embedding of the current frame is stored through the memory encoder, and the stored memory information is combined in the segmentation process of subsequent frames to further improve the accuracy of segmentation.

[0142] After image segmentation, continuous image sequences with segmentation masks are used for 3D reconstruction. The maximum intensity projection method (MIP) is used to convert 2D slices into 3D models, generating intuitive 3D reconstruction results for doctors to use to assist clinical diagnosis and treatment planning. If the doctor is not satisfied with the 3D reconstruction results, he can re-annotate the query set images and provide more detailed prompt information. The system will repeat the process of feature extraction, similarity calculation, image segmentation and 3D reconstruction based on the new query set and support set. Through multiple rounds of iterative optimization, it will eventually generate a high-precision 3D reconstruction model to meet clinical needs.

[0143] During the entire implementation process, the small sample learning framework was used to reduce dependence on large-scale labeled data. Combined with the powerful segmentation capabilities and memory mechanism of the SAM series models, efficient and robust medical image segmentation and three-dimensional reconstruction were achieved under data scarce conditions, significantly improving the feasibility and effectiveness of clinical applications.

[0144] The above specific implementations are only descriptions of the preferred implementations of the present invention, and do not limit the protection scope of the present invention. Without departing from the design concept and spirit of the present invention, various modifications, substitutions and improvements made by ordinary technicians in this field to the technical solution of the present invention based on the text description and drawings provided by the present invention should all fall within the protection scope of the present invention. The protection scope of the present invention is determined by the claims.

Claims

1. A three-dimensional reconstruction method of human tissue CT images based on the SAM series model, characterized in that: include Step 1: Collect the CTSpine1K dataset and query set images provided by the partner hospital, and filter and trim the support set and query set to match the model input requirements; Step 2: Use the pre-trained ResNet-18 to extract features of the support set and query set, and perform feature matching through cosine similarity calculation; Step 3: Load the pre-trained SAM v2 model, segment the image according to the point, box or mask prompts provided by the user, and generate a segmentation mask; Step 4: Evaluate the segmentation results, add the images with good performance to the query set, and randomly update the support set to enhance the generalization ability of the model; Step 5: Store and utilize the embedding information of past frames through the memory encoder and combine it with the current prompt to generate a segmentation mask; Step 6: Based on the continuous image sequence with segmentation mask, perform 3D reconstruction to generate an intuitive 3D model.

2. The method for three-dimensional reconstruction of human tissue CT images based on the SAM series model according to claim 1, characterized in that: Step 1 is as follows: Step 1.1: Use the CTSpine1K dataset, which collects 1,005 spinal CT images from four public datasets and annotates 25 types of vertebrae, including C1-C7, T1-T12, and L1-L6; Step 1.2: Randomly select 8 samples from the CTSpine1K dataset as the initial support set to ensure the diversity and representativeness of the samples; Step 1.3: Select samples based on the similarity of spine parts to ensure that the images in the support set are highly correlated with the parts of the query set; Step 1.4: Adjust the image resolution to match the model’s input requirements, primarily by cropping out the uninformative black background areas. Step 1.5: A total of 494 CT images of 5 patients and their corresponding labels were provided by the cooperating hospital; Step 1.6: Perform the same screening and cropping operations on the query set images as the support set to ensure the consistency of the query set and the support set in image resolution and content for subsequent feature extraction and matching.

3. The method for three-dimensional reconstruction of human tissue CT images based on the SAM series model according to claim 1, characterized in that: Step 2 is as follows: Step 2.1: Organize the support set images into a tensor of shape [Ns,3,H,W], where Ns represents the number of samples in the support set, 3 represents the RGB channels, and H and W are the height and width of the image respectively; Step 2.2: Organize the query set images into a tensor of shape [Nq,3,H,W], where Nq represents the number of samples in the query set; Step 2.3: Use the pre-trained ResNet-18 model provided by the torchvision library in Python as the feature extractor; Step 2.4: Forward propagate the support set image through ResNet-18, and after global pooling, obtain a feature vector of shape [Ns, C], where C is the dimension of the feature vector; Step 2.5: Also use ResNet-18 to extract the features of the query set image and obtain a feature vector with a shape of [Nq, C]; Step 2.6: Use cosine similarity as an indicator to measure the similarity between the query set and the support set features; Step 2.7: For each query set feature vector, calculate the cosine similarity between the query set feature vector and all support set feature vectors, and select the support set feature with the highest similarity as the matching result; Step 2.8: Concatenate the matched support set image with the corresponding query set image to form a new image sequence of shape [Nq+1,3,H,W], and use the label corresponding to the support set as the mask required for segmentation.

4. The method for three-dimensional reconstruction of human tissue CT images based on the SAM series model according to claim 1, characterized in that: Step 3 is as follows: Step 3.1: Select Segment Anything Model v2, also known as SAM v2; Step 3.2: Download and load the pre-trained SAM v2 model weights from the specified link; Step 3.3: Use the default parameter settings of the model to ensure the stability and consistency of the model in the initial stage; Step 3.4: Based on specific needs, the user provides prompt information to guide the SAM model to focus on the area of ​​interest for segmentation; Step 3.5: Based on the initial prompt, the SAM model generates a preliminary segmentation mask; Step 3.6: The user observes the segmentation results. If the results are not satisfactory, the user adjusts the position or form of the prompt, re-enters the prompt information, and performs multiple iterations to optimize the segmentation effect; Step 3.7: Finally, an image with a segmentation mask is generated as the basic data for subsequent 3D reconstruction.

5. The method for three-dimensional reconstruction of human tissue CT images based on the SAM series model according to claim 1, characterized in that: Step 4 is as follows: Step 4.1: Using the initially selected 8 support set images as the benchmark, perform a preliminary segmentation operation through the SAM model to generate an initial segmentation mask; Step 4.2: Evaluate the segmentation results of each image in the query set and select the images with excellent segmentation results as the basis for optimization; Step 4.3: Add the segmentation result images with excellent performance after evaluation to the query set to form a more representative query set and improve the performance of the model in subsequent segmentation tasks; Step 4.4: Randomly extract new samples from the support set and add them to the support set to enhance the generalization ability of the model and prevent overfitting to the initial samples; Step 4.5: Call the SAM model multiple times to perform segmentation, evaluate the segmentation results, and update the query set and support set to gradually improve the segmentation accuracy and stability of the model; Step 4.6: After each iteration, the model can use the latest query set and support set information to further optimize the segmentation effect, ensuring that the final generated segmentation mask has high accuracy and robustness.

6. The method for three-dimensional reconstruction of human tissue CT images based on the SAM series model according to claim 1, characterized in that: Step 5 is as follows: Step 5.1: Combine the image embedding of the current frame and the prompt information provided by the user; Step 5.2: The lightweight mask decoder generates the segmentation mask of the current frame based on the image embedding and hint information; Step 5.3: The memory encoder creates a frame embedding based on the prediction result of the current frame and stores it in the memory bank; Step 5.4: Get the embedding of each frame from the image encoder and combine it with the stored memory information to generate the segmentation mask of the current frame.

7. The method for three-dimensional reconstruction of human tissue CT images based on the SAM series model according to claim 1, characterized in that: Step 6 is as follows: Step 6.1: Continuous image sequence with segmentation mask; Step 6.2: Extract tissue or structure regions of interest using segmentation masks; Step 6.3: Use the maximum intensity projection method, or MIP, to convert the two-dimensional slices into three-dimensional images; Step 6.4: Apply the segmented 2D mask to the MIP algorithm to generate a 3D model; Step 6.5: The generated 3D reconstruction model is used by doctors for clinical diagnosis and treatment planning.

8. The method for three-dimensional reconstruction of human tissue CT images based on the SAM series model according to claim 1, characterized in that: The formula for step 2.7 is as follows: In which, given two vectors A and B, Ai and Bi represent the components of vectors A and B respectively.

Images