Skeletal tissue subdivision and reconstruction method based on SAM series model

Abstract

The invention discloses a skeleton tissue subdivision and reconstruction method based on an SAM series model. The method comprises the following steps: converting image data into a format supported by the model; generating an optimized feature map in the convolutional network; performing feature slicing operation on the feature map, calibrating slices and reconstructing the slices into a complete feature map; inputting the feature map into a detection head to generate a candidate box and a confidence score, embedding prompt information into a segmentation decoder, and fusing the prompt information with the feature map to generate a segmentation mask; storing the feature result of the current frame in a memory bank; the prompt information is coded into an embedded mode, and after the prompt information is fused with the feature map, segmentation masks related to the prompt are generated; and formatting the target detection result, adjusting the size of the segmentation mask to be aligned with the input image, and storing the detection and segmentation result as a specified format file. According to the method, the bone tissue can be efficiently segmented, and boundaries can be accurately distinguished under the condition that the upper and lower bones are similar in shape and close in position.

Description

Technical Field

[0001] The present invention relates to the field of bioinformatics, and in particular to a bone tissue segmentation and reconstruction method based on a SAM series model. Background Art

[0002] In the medical field, image segmentation technology plays an important role in disease diagnosis, treatment planning and efficacy evaluation. The precise segmentation of medical images aims to identify and outline specific anatomical structures or lesion boundaries from complex medical images. This technology is of great significance for the accurate diagnosis of diseases and the formulation of treatment plans. However, the segmentation task of medical images faces many challenges, such as overlapping shadows between different structures, ambiguity of target boundaries, complex anatomical structures and individual differences among patients. These problems are particularly prominent in the CT vertebral segmentation task. Due to the complexity of the spinal structure, the similarity between adjacent structures and the volatility of contrast, the difficulty of automated segmentation tasks is further increased.

[0003] Existing segmentation methods rely on deep learning technology, and analyze medical images through algorithms such as convolutional neural networks to achieve fully automatic spinal segmentation. These methods have made some progress. For example, Suzani et al. developed a deep neural network model to achieve automatic detection and positioning of vertebrae in CT images and significantly shortened the calculation time; Wang et al. proposed a solution with smaller errors for the cervical and lumbar regions, but the segmentation accuracy of the thoracic region still needs to be improved. In addition, traditional methods such as random forest algorithms also show high accuracy in some scenarios. However, these methods generally rely on a large number of high-quality pixel-level annotated data sets, and the time-consuming and labor-intensive annotation work limits the promotion and application of the methods.

[0004] Clinical data in the medical field are often first-hand information with unique and important value. However, the amount of data is huge and the annotation is difficult. The reliance of traditional models on data annotation in practical applications has become a prominent problem. At the same time, existing methods lack the generalization ability for small or even zero-sample data and cannot fully cope with the situation of insufficient data annotation in clinical environments. Especially in the task of spinal segmentation, due to the anatomical complexity of the spine and the similarity of the shapes of bones in different segments, it is difficult for existing algorithms to accurately distinguish the boundaries of adjacent segments. In addition, the inter-observer and intra-observer variability caused by annotation errors, as well as the overlap of anatomical structures caused by spinal curvature, make it difficult for existing models to meet the needs of clinical high-precision segmentation.

[0005] Although deep learning segmentation models are emerging in an endless stream, there are still application gaps in bone tissue segmentation. The fine segmentation of bone tissue, especially when the upper and lower bones are similar in shape and close in position, is still a technical difficulty. This not only requires the model to accurately capture the detailed features of the bones, but also requires accurate distinction of boundary information in complex anatomical structures. In addition, whether the model can quickly adapt and generalize to the latest patient data is also an important link. Due to the dynamic nature of medical data and the anatomical differences between patients, the model may not be able to cope with new data when its generalization ability is insufficient, resulting in a decrease in segmentation accuracy, thus affecting the effect of clinical application.

[0006] In this context, how to achieve rapid segmentation of bone tissue, especially when bones in different segments are highly similar, while improving the model's ability to generalize to new patient data, has become a technical problem that needs to be solved in the field of medical image segmentation. This will not only help reduce dependence on large-scale annotated datasets, but also provide more efficient and reliable segmentation and analysis tools for clinical work. Summary of the invention

[0007] In order to overcome the shortcomings of the prior art, the present invention proposes a bone tissue segmentation and reconstruction method based on the SAM series model. By enhancing the few-sample and zero-sample generalization capabilities of the model, the method can still run stably in the case of insufficient data annotation, reducing the dependence on large-scale pixel-level annotation datasets and alleviating the pressure of clinical annotation work.

[0008] To achieve the above object, the present invention provides a bone tissue segmentation and reconstruction method based on the SAM series model, comprising the following steps:

[0009] Step 1: Read high-resolution images or video frames, resize them to fit the model input requirements, perform pixel normalization, and convert the data into a format supported by the model.

[0010] Step 2: Extract features layer by layer in the convolutional network, compress and expand the channels through the C3K module, and use the C3K2 module to fuse the multi-layer features to generate an optimized feature map;

[0011] Step 3: Pass the feature map into the C2PSA module, perform feature slicing, calculate the attention weight of the slice, calibrate the slice and reconstruct it into a complete feature map;

[0012] Step 4: Input the feature map into the detection head to generate candidate boxes and confidence scores, and perform non-maximum suppression on the candidate boxes; at the same time, in the segmentation task, embed the prompt information into the segmentation decoder and generate the segmentation mask after fusion with the feature map;

[0013] Step 5: Store the feature results of the current frame into the memory bank, extract the features and prediction information of the previous frames from the memory bank, optimize the features of the current frame through the memory attention module, and update the results into the memory bank;

[0014] Step 6: Receive the prompt information (such as point, box or mask) input by the user, encode the prompt information into an embedding, and fuse it with the feature map to generate a segmentation mask related to the prompt;

[0015] Step 7: Format the target detection results, adjust the segmentation mask size to align with the input image, and save the detection and segmentation results to the specified format file.

[0016] Furthermore, step 1 is as follows:

[0017] Step 1.1: Read high-resolution images or video frames and load the data into memory;

[0018] Step 1.2: Check whether the image resolution meets the model requirements. If not, adjust it to the standard size supported by the model.

[0019] Step 1.3: Normalize the pixel values ​​and convert them to the numerical range supported by the model (such as 0 to 1);

[0020] Step 1.4: Convert the preprocessed data into a tensor in the model input format.

[0021] Furthermore, step 2 is as follows:

[0022] Step 2.1: Input the preprocessed image or frame into the convolutional network of the model to extract preliminary features;

[0023] Step 2.2: Use the C3K module to optimize the features. The C3K module is a bottleneck structure consisting of two convolutional units. It reduces the computational complexity and enhances the feature expression capability by compressing the number of channels first and then expanding the number of channels.

[0024] Step 2.3: Perform deeper feature fusion in the C3K2 module. The C3K2 module is an extension of C3K, which increases the cross-layer feature reuse capability and can better integrate multi-layer feature information.

[0025] Step 2.4: Pass the optimized features to subsequent modules for further processing.

[0026] Furthermore, step 3 is as follows:

[0027] Step 3.1: Input the fused features into the C2PSA module;

[0028] Step 3.2: Perform feature slicing in the C2PSA module to divide the input feature map into multiple segments;

[0029] Step 3.3: Calculate the attention weight for each feature segment, and adaptively adjust the weight distribution of different segments based on the pyramid slicing mechanism;

[0030] Step 3.4: Reassemble the calibrated feature fragments into a complete feature map to provide optimized input for object detection and segmentation.

[0031] Furthermore, step 4 is as follows:

[0032] Step 4.1: Input the calibrated feature map into the detection head module to generate candidate boxes and their corresponding confidence scores;

[0033] Step 4.2: Perform non-maximum suppression (NMS) operation on the candidate boxes, remove overlapping boxes, and generate the final target detection result;

[0034] Step 4.3: At the same time, the feature map is passed to the segmentation decoder, and the segmentation mask is gradually generated according to the features;

[0035] Step 4.4: Combine user hints (such as points, boxes, or masks) for position encoding and embed hint information into the segmentation decoding process;

[0036] Step 4.5: Output the segmentation mask aligned with the original input image size.

[0037] Furthermore, step 5 is as follows:

[0038] Step 5.1: Store the feature extraction result of the current frame into the memory bank;

[0039] Step 5.2: Extract the historical feature information of the previous frames in the memory bank to provide context support for the processing of the current frame;

[0040] Step 5.3: Use the memory attention mechanism to calculate the correlation between the current frame features and the historical frame features, and optimize the embedding of the current frame features;

[0041] Step 5.4: Generate detection and segmentation results of the current frame based on the optimized features;

[0042] Step 5.5: Update the segmentation results and prompt information of the current frame into the memory library to provide a reference for the processing of subsequent frames.

[0043] Furthermore, step 6 is as follows:

[0044] Step 6.1: receiving prompt information (such as a point, a box or a mask) input by the user;

[0045] Step 6.2: Positionally encode the prompt information and convert it into an embedded representation;

[0046] Step 6.3: Combine the hint embedding with the feature map embedding as the input to the decoder.

[0047] Step 6.4: Generate accurate object segmentation masks based on the cues in the segmentation decoder.

[0048] Furthermore, step 7 is as follows:

[0049] Step 7.1: Format the target detection results to generate an output file containing the target category and location;

[0050] Step 7.2: Resize the segmentation mask to match the input image resolution.

[0051] Step 7.3: Save the detection and segmentation results in a specified format (such as JSON or PNG);

[0052] Step 7.4: Output the processed detection and segmentation results for subsequent application or analysis.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention provides a bone tissue segmentation and reconstruction method based on the SAM series model, which has good generalization ability for the latest patient data, can quickly adapt to the dynamically changing sample data in the clinical environment, and ensure the reliability and consistency of the segmentation results.

[0055] 2. The present invention provides a bone tissue segmentation and reconstruction method based on the SAM series model. By refining the model's feature extraction and boundary recognition capabilities for complex anatomical structures, this method significantly improves the accuracy of segmentation, especially the processing of overlapping and fuzzy boundaries.

[0056] 3. The present invention provides a bone tissue segmentation and reconstruction method based on the SAM series model, which can further generate a three-dimensional bone model, provide support for the visualization of different bone segments and regions, and help clinicians calculate indicators and plan treatment.

[0057] 4. The present invention provides a bone tissue segmentation and reconstruction method based on the SAM series model. It takes the automated segmentation process as the core, reduces the dependence on professional knowledge and operational experience, enables non-professionals to use it efficiently, and broadens the practical application scope of the technology. Through efficient segmentation and modeling processes, the method can quickly generate the analysis results required for clinical practice, shorten the preparation time for diagnosis and treatment, and strive for more timely treatment opportunities for patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 It is a schematic diagram of the steps of the present invention

[0060] Figure 2 This is the structure diagram of YOLOv11

[0061] Figure 3 Introduction to the SAM v2 module

[0062] Figure 4 Schematic diagram of the overall functional flow

[0063] Figure 5 This is a schematic diagram of the model reconstruction process

[0064] Figure 6 is the accuracy curve of the training process

[0065] Figure 7 This is a diagram of the accuracy, loss, and learning rate curves.

[0066] Figure 8 It is a statistical graph of accuracy and loss for the test set

[0067] Fig. 9 It is a complete static CT image

[0068] Fig.10 It is to capture bone tissue images

[0069] Fig.11 is the bone tissue segmentation image

[0070] Fig.12 This is an uncolored image of bone tissue modeling. DETAILED DESCRIPTION

[0071] The technical solution of the present invention will be more clearly and completely explained below through description of preferred embodiments of the present invention in combination with the accompanying drawings.

[0072] like Figure 1 As shown, the present invention is specifically:

[0073] Step 1: Read the image or video frame, adjust the size to fit the model input requirements, perform pixel normalization, and convert the data into a format supported by the model;

[0074] Step 2: Extract features layer by layer in the convolutional network, compress and expand the channels through the C3K module, and use the C3K2 module to fuse the multi-layer features to generate an optimized feature map;

[0075] Step 3: Pass the feature map into the C2PSA module, perform feature slicing, calculate the attention weight of the slice, calibrate the slice and reconstruct it into a complete feature map;

[0076] Step 4: Input the feature map into the detection head to generate candidate boxes and confidence scores, and perform non-maximum suppression on the candidate boxes; at the same time, in the segmentation task, embed the prompt information into the segmentation decoder and generate the segmentation mask after fusion with the feature map;

[0077] Step 5: Store the feature results of the current frame into the memory bank, extract the features and prediction information of the previous frames from the memory bank, optimize the features of the current frame through the memory attention module, and update the results into the memory bank;

[0078] Step 6: Receive the prompt information input by the user, encode the prompt information into an embedding, and fuse it with the feature map to generate a segmentation mask related to the prompt;

[0079] Step 7: Format the target detection results, adjust the segmentation mask size to align with the input image, and save the detection and segmentation results to the specified format file.

[0080] As a specific implementation method, considering that the static CT resolution of the first-line medical data used is relatively large (1536×1536 or 3072×3072), and the proportion of the bone part is relatively small, the accuracy of directly segmenting the large image will be more or less lost. To avoid this situation, the detection model YOLOv11 is used here to first perform an image detection on the CT image to achieve the purpose of extracting only the part containing bone tissue.

[0081] The structure of YOLOv11 is shown in the figure. The overall structure still maintains the idea of ​​extracting features from a convolutional network plus multi-level feature fusion and detecting results with multiple detection heads. The details of each structure are in Figure 2 Displayed on the left and right sides.

[0082] The number of input channels of each module, where Conv is a standard conv+bn+silu structure. The C3K structure is a common botteleneck structure consisting of two Convs, that is, the channels are first reduced and then increased (such as Figure 2 upper right corner).

[0083] The C3K2 structure inherits C2f. The core idea is to integrate the features of this level with the features of multiple levels in the previous order to improve feature reuse. Correspondingly, according to the middle two-layer structure, the flag c3k is True or False and is divided into C3K or BottleNeck.

[0084] The C3K2 module is a more efficient CSP bottleneck variant. It uses two convolutions instead of one large convolution, which speeds up feature extraction. At the end of the model, it can be found that a layer of C2PSA similar to the attention mechanism is added after SPPF. SPPF (Spatial Pyramid Pooling-Fast) is an optimized version of Spatial Pyramid Pooling (SPP) (such as Figure 2 At the same time, pooled features at different levels are spliced ​​together to improve feature richness.

[0085] SAM v2 is an improved version of SAM. SAM is a general segmentation model developed by the Meta AI team that can generate high-quality segmentation masks in various tasks. SAMv2 has made many improvements on this basis, aiming to solve some specific problems and improve the overall performance of the model. Its architecture is as follows Figure 3 As shown; Figure 3 In the figure, the figure consists of two parts, namely (a) cued visual segmentation and (b) key components of the model; (a) users can mark different forms of cues in a single or multiple videos (or image sequences), such as anchor boxes, points, masks, and the cues are fed into the model to obtain individual segmentation results in the video. (b) The segmentation model has the following components: image encoder; cue word encoder; memory attention; memory group; mask decoder.

[0086] Video reasoning process: As shown in the upper left corner, some video frames can be manually annotated in different formats. The video is processed in a streaming manner. The image encoder consumes one frame at a time and cross-focuses on the memory of the target object of the previous frame (from the memory group). The mask decoder can selectively accept the input prompt of the prompt word encoder to predict the segmentation mask of the frame. Finally, a memory encoder converts the prediction and image encoder embedding (not shown in the figure) for use in future frames (this part is stored in the memory group).

[0087] Image Encoder: This module is responsible for converting the input image into a feature representation. It uses a pre-trained Convolutional Neural Network (CNN) that can extract high-level features from the image. Here, the Hiera (Ryali et al., 2023; Bolya et al., 2023) image encoder pre-trained with MAE (He et al., 2022) is used, which is hierarchical and allows the use of multi-scale features in the decoding process.

[0088] Memory Attention Mechanism: Uses normal attention operations for self-attention and cross-attention, allowing the model to benefit from recent developments in effective attention kernels. Stack L Transformer blocks, with the first block taking the image encoding from the current frame as input.

[0089] Hint Encoder: The hint encoder is the same as that of SAM, and can be hinted by clicks (positive or negative), bounding boxes, or masks to define the extent of an object in a given frame. Sparse hints are represented by summing the positional encoding of each hint type with the learned embedding, while masks are embedded using convolutions and summed with the frame embedding.

[0090] Mask Decoder: The mask decoder design largely follows SAM, stacking “bidirectional” transformer blocks that update prompt and frame embeddings.

[0091] Memory Encoder: The memory encoder downsamples the output mask using convolutional modules and uses the unconditional frame embedding of the image encoder, followed by a lightweight convolutional layer to fuse the information.

[0092] Memory Group: Preserves past prediction information of target objects in the video by maintaining a FIFO memory queue of up to N most recent frames, and stores hint information in a FIFO queue of up to M hint frames.

[0093] Overall, v2 uses the base model, trained on the large-scale SA-V dataset collected. SAM 2 is able to solve fast visual segmentation tasks by interactively segmenting regions via cues (clicks, boxes, or masks) over one or more video frames, using a streaming memory that stores previous cues and predictions.

[0094] In such an overall framework, coupled with a real-time video processing pipeline with streaming memory, SAM 2, trained on the above data, provides strong performance in a wide range of tasks. In video segmentation, the model directly presents better accuracy, using 3 times fewer interactions than previous methods. In image segmentation, v2 is also more accurate and faster than v1.

[0095] As a specific implementation method, Figure 4 As shown in the figure, take a xxx.zip file containing many dicom files as an example. The user doctor uploads it to the reconstruction platform through the file upload function zip_opload(), clicks the reconstruction function model_show(), and then calls the xxx.zip file to obtain the model through the processing flow build_pipeline(), and the result obtained by the calculation analysis cal_analyse() is returned to the reconstruction platform, and then the model is displayed. The analysis results can also generate a report report and return it to the doctor. The doctor uploads a zip file containing a considerable number of dicom files. The reconstruction platform will temporarily save the file. If the file needs to be modeled, the background server will get the file for reconstruction processing, and the obtained file will be returned to the reconstruction platform for display.

[0096] The model reconstruction process is as follows Figure 5 As shown (xy plane), the xxx.zip file will be decompressed and read by the background and converted into the corresponding nii file, which is convenient for reading intermediate results and reading related 3D parameters, such as spacing: 0.308×0.308×1. Spacing represents the physical size of each voxel in the x, y and z directions, and the unit is usually millimeter (mm). When performing image registration, segmentation, reconstruction and other operations, spacing needs to be considered to ensure the accuracy of geometric relationships; a xxx.nii.gz file converted from an original dicom file format can be used to obtain the spine detection result xxx_spine.nii.gz through the image detection module detection. The detection result is passed through the image segmentation module segmentation to obtain the spine segmentation result xxx_spine_seg.nii.gz, and the segmentation result is passed through the modeling module build to obtain the corresponding model file xxx.obj. For this reason, the intermediate results that need to be read are saved as nii files. After obtaining the original image, it is sent to the detection process. Because the essence of this process is to extract the spine part, the original image is first compressed and then detected to obtain an image containing only the spine part. The corresponding segmented image is sent to the segmentation network SAM model. Different parts are colored differently according to the different labels of the segmented image and finally saved as an obj file for display on the reconstruction platform.

[0097] Experimental data:

[0098] Static CT: It was collected by special medical equipment, a total of 20 cases (more in the future). Each case was taken from the coccyx and above, and the research mainly included the lumbar spine from L1 to L4. Each case of 360+ slices can completely cover this part. All data have completed ethical approval and comply with data privacy and security regulations.

[0099] CTSpine1K dataset: CTSpine1K is a large-scale CT dataset dedicated to spine segmentation. This dataset collects a total of 1,005 CT data from four public datasets and annotates 25 types of vertebrae, including C1-C7, T1-T12, and L1-L6. The data does not need to be preprocessed before entering the model. There are many augmentation methods in the training detection network: random radiation transformation, random color transformation, random copy paste, and the model input resolution is 256×256.

[0100] Processing Flow

[0101] Training YOLO to detect bone tissue

[0102] The CTSpine1K dataset is divided into a training and test set with a ratio of 4:1 to fine-tune the pre-trained model of YOLOv11. After 5-fold cross-validation to obtain the weights of 5 models, the inference process uses the output mean of the 5 models. During the training process, the deep learning framework used is Pytorch, and GPU acceleration is used throughout the process without multi-threading. The accuracy curve is as follows Figure 6 As shown, Figure 6 Metrics: Metrics

[0103] mAP50: specifically refers to the mAP when the IoU (Intersection over Union) threshold is 0.5. This means that if the IoU between the predicted box and the true box is greater than or equal to 0.5, the prediction is considered correct.

[0104] mAP50-95: refers to the average value of mAP calculated over multiple IoU thresholds (from 0.5 to 0.95, with a step size of 0.05).

[0105] Precision: Precision is defined as the number of correctly predicted positive samples divided by the number of all predicted positive samples.

[0106] Recall: Recall rate. It is defined as the number of correctly predicted positive samples divided by the number of all actual positive samples;

[0107] Among them, multiple curves represent the training results of different scales and random seeds. mAP, precision and recall are common detection accuracy indicators. Some specific hyperparameters are: learning rate: 1e-2 cosine down to 0, batch size: 32.

[0108] Image segmentation using the SAM series model

[0109] At this step, the SAM model can identify and segment the area of ​​interest based on the user's prompts. These prompts can be in the form of points, boxes, or masks. The user needs to select the appropriate prompt type according to specific needs and mark it on the image. Furthermore, the user observes the output effect, adjusts the position or form of the prompt, and tries different settings multiple times to obtain a better segmentation effect. Specifically, we use the common segmentation model U-net to initially obtain a segmentation result as the prompt of the SAM model. Because an initial segmentation mask can guide SAM to segment specific areas more accurately. This is very useful for tasks that require high precision, especially the medical image segmentation tasks we are conducting. In addition, the initial mask can be used as prior knowledge to help the model converge to the correct segmentation result faster, thereby reducing computing time and resource consumption.

[0110] Therefore, a regular U-net is also trained here as a preliminary segmentation tool. The specific hyperparameters are: using the CTSpine1K dataset, learning rate 1e-3, weight decay 1e-4, Adam optimizer, learning for 100 epochs, and learning rate cosine decay to 1e-6. The deep learning framework used is Pytorch, GPU acceleration is used throughout the process, and multithreading is not used.

[0111] During the training process, the accuracy, loss and learning rate curves are as follows Figure 7 As shown, Figure 7 Train: training set

[0112] Dice: segmentation accuracy indicator

[0113] Lr: learning rate

[0114] Mt loss: mean teacher loss: semi-supervised teacher model loss

[0115] Seg loss: segmentation loss, dice loss+ce loss

[0116] Loss: loss, seg loss+mt loss;

[0117] At the same time, the accuracy and loss statistics of the test set are as follows: Figure 8 As shown, Figure 8 Val: test set;

[0118] Dice: Segmentation accuracy indicator, used to evaluate the overlap between the predicted segmentation results and the true labels.

[0119] Loss: loss function value.

[0120] 3D reconstruction

[0121] After obtaining the segmented image processed by the SAM model and saving the processing result in the nii file, we will continue to model it and convert it into an obj file.

[0122] 1.1 Reading NIfTI files

[0123] Use the Nibabel library to read NIfTI files. NIfTI files typically contain medical imaging data, such as MRI or CT scans.

[0124] 2.1 Normalized Data

[0125] The segmented image data is normalized to the range of [0,1] through minimum-maximum normalization for subsequent processing. Because it is a segmentation result, the segmented parts generally appear collectively with a certain number of voxel points, so there is almost no single outlier or noise. If there is a voxel label classification error, the segmentation model should be further tuned.

[0126] 2.2 Threshold segmentation

[0127] Choose a suitable threshold to segment the region of interest (ROI). The choice of threshold depends on the characteristics of the data and the characteristics of the target area. In the actual processing process, the non-zero part of the data is the region of interest that needs to be processed.

[0128] 3.1 Generating surface mesh using Marching Cubes algorithm

[0129] Marching Cubes algorithm is a commonly used 3D reconstruction method that can generate surface meshes from voxel data. Use the Marching Cubes algorithm in the VTK or PyVista library to generate surface meshes and obtain the corresponding point and surface parameters. The level takes the default value of 0.5. To maintain the size and shape of the model, the spacing obtained by adding the original data is input here.

[0130] verts,faces,_,_=marching_cubes(volume_data,spacing,level), we can get the required points and faces.

[0131] 3.2 Smooth Mesh

[0132] The generated mesh may contain steps or irregular surfaces. Smoothing filters can be used to optimize the mesh. Common smoothing methods include Laplacian smoothing and Taubin smoothing. Laplacian smoothing is used here for surface smoothing, and the relevant parameters are kept at the default values ​​of the Python library.

[0133] 4.1 Save as OBJ file

[0134] The generated surface mesh is saved as an OBJ file, which can be read and processed by many 3D modeling and rendering software, and can also be displayed online on the website. The converted obj file is statistically moderate in size and does not need to be optimized.

[0135] As a specific implementation, a complete static CT image is loaded from a storage device as input data. The image is formatted to ensure that its resolution meets the model requirements. If the image resolution is inconsistent with the model input requirements, the image size is adjusted to a standard size through interpolation. The image is normalized to convert the pixel values ​​to a range between 0 and 1. After preprocessing, the data is formatted into a tensor format supported by the model in preparation for subsequent processing.

[0136] The preprocessed complete static CT image is input into the convolutional network of the model, such as Fig. 9 As shown in the figure, features are extracted layer by layer. First, low-level features are extracted through the basic convolution layer and passed to the C3K module. The C3K module performs channel compression and expansion on the feature map, and then passes the extracted features to the C3K2 module. The C3K2 module fuses the current features with the multi-level features of the previous layers to generate an optimized feature map, laying the foundation for the subsequent operation of capturing bone tissue images.

[0137] The optimized feature map is input into the C2PSA module. In the C2PSA module, a slicing operation is performed on the feature map to divide the features into several segments. The attention weight of each segment is calculated, and the segments are calibrated. After the calibration is completed, all slices are reassembled into a complete feature map for detecting specific areas of bone tissue. The candidate region boxes of bone tissue are generated by the model detection head, and the redundant candidate boxes are removed by the non-maximum suppression algorithm to retain the optimal candidate regions. Based on these candidate regions, the bone tissue image is intercepted from the complete static CT image, such as Fig.10 shown.

[0138] The captured bone tissue image is input into the segmentation decoder for bone tissue segmentation processing. The segmentation decoder gradually restores the feature map to a segmentation mask, and combines the prompt information (such as points or boxes) to accurately segment the target area to generate a bone tissue segmentation image, such as Fig.11 After the segmentation is completed, the size of the segmentation mask is adjusted to be consistent with the original CT image.

[0139] In the video processing task, for dynamic data processed frame by frame, the bone tissue features of the current frame are stored in the memory bank, and the historical feature information of the previous frames is extracted from the memory bank. The memory attention module is combined to calculate the correlation between the current frame and the historical frames to optimize the segmentation result of the current frame. The optimized bone tissue segmentation image result is updated to the memory bank to provide a reference for subsequent frame processing.

[0140] In the bone tissue modeling phase, the generated bone tissue segmentation image is used to extract the three-dimensional feature points of the bone. Based on these feature points, the three-dimensional structure of the bone tissue is reconstructed through the three-dimensional modeling algorithm to complete the bone tissue modeling task, such as Fig.12 The modeling results can be used for medical analysis, surgical planning, or further research and development.

[0141] Finally, the detection results (including the bone tissue area) and the bone tissue segmentation image are saved as files in the specified format, and the three-dimensional model of the bone tissue is output as a visualization file or other standard model format for subsequent analysis or direct application in related scenarios.

[0142] The above specific implementations are only descriptions of the preferred implementations of the present invention, and do not limit the protection scope of the present invention. Without departing from the design concept and spirit of the present invention, various modifications, substitutions and improvements made by ordinary technicians in this field to the technical solution of the present invention based on the text description and drawings provided by the present invention should all fall within the protection scope of the present invention. The protection scope of the present invention is determined by the claims.

Claims

1. A method for bone tissue segmentation and reconstruction based on the SAM series model, characterized in that: The following steps are involved: Step 1: Read the image or video frame, adjust the size to fit the model input requirements, perform pixel normalization, and convert the data into a format supported by the model; Step 2: Extract features layer by layer in the convolutional network, compress and expand the channels through the C3K module, and use the C3K2 module to fuse the multi-layer features to generate an optimized feature map; Step 3: Pass the feature map into the C2PSA module, perform feature slicing, calculate the attention weight of the slice, calibrate the slice and reconstruct it into a complete feature map; Step 4: Input the feature map into the detection head to generate candidate boxes and confidence scores, and perform non-maximum suppression on the candidate boxes; at the same time, in the segmentation task, embed the prompt information into the segmentation decoder and generate the segmentation mask after fusion with the feature map; Step 5: Store the feature results of the current frame into the memory bank, extract the features and prediction information of the previous frames from the memory bank, optimize the features of the current frame through the memory attention module, and update the results into the memory bank; Step 6: Receive the prompt information input by the user, encode the prompt information into an embedding, and fuse it with the feature map to generate a segmentation mask related to the prompt; Step 7: Format the target detection results, adjust the segmentation mask size to align with the input image, and save the detection and segmentation results to the specified format file.

2. A method for bone tissue segmentation and reconstruction based on the SAM series model according to claim 1, characterized in that: Step 1 is as follows: Step 1.1: Read the image or video frame and load the data into memory; Step 1.2: Check whether the image resolution meets the model requirements. If not, adjust it to the standard size supported by the model. Step 1.3: Normalize the pixel values ​​and convert them to the numerical range supported by the model; Step 1.4: Convert the preprocessed data into a tensor in the model input format.

3. The method for bone tissue segmentation and reconstruction based on the SAM series model according to claim 1, characterized in that: Step 2 is as follows: Step 2.1: Input the preprocessed image or frame into the convolutional network of the model to extract preliminary features; Step 2.2: Use the C3K module to optimize the features. The C3K module is a bottleneck structure consisting of two convolutional units. Step 2.3: Perform deeper feature fusion in the C3K2 module, which is an extension of C3K. Step 2.4: Pass the optimized features to subsequent modules for processing.

4. The method for bone tissue segmentation and reconstruction based on the SAM series model according to claim 1, characterized in that: Step 3 is as follows: Step 3.1: Input the fused features into the C2PSA module; Step 3.2: Perform feature slicing in the C2PSA module to divide the input feature map into multiple segments; Step 3.3: Calculate the attention weight for each feature segment, and adaptively adjust the weight distribution of different segments based on the pyramid slicing mechanism; Step 3.4: Reassemble the calibrated feature fragments into a complete feature map to provide optimized input for object detection and segmentation.

5. The method for bone tissue segmentation and reconstruction based on the SAM series model according to claim 1, characterized in that: Step 4 is as follows: Step 4.1: Input the calibrated feature map into the detection head module to generate candidate boxes and their corresponding confidence scores; Step 4.2: Perform non-maximum suppression on the candidate boxes, remove overlapping boxes, and generate the final target detection result; Step 4.3: At the same time, the feature map is passed to the segmentation decoder, and the segmentation mask is gradually generated according to the features; Step 4.4: Perform position encoding based on user prompts and embed the prompt information into the segmentation and decoding process; Step 4.5: Output the segmentation mask aligned with the original input image size.

6. The method for bone tissue segmentation and reconstruction based on the SAM series model according to claim 1, characterized in that: Step 5 is as follows: Step 5.1: Store the feature extraction result of the current frame into the memory bank; Step 5.2: Extract historical feature information of previous frames in the memory bank to provide context support for the processing of the current frame; Step 5.3: Use the memory attention mechanism to calculate the correlation between the current frame features and the historical frame features, and optimize the embedding of the current frame features; Step 5.4: Generate detection and segmentation results of the current frame based on the optimized features; Step 5.5: Update the segmentation results and prompt information of the current frame into the memory library to provide a reference for the processing of subsequent frames.

7. The method for bone tissue segmentation and reconstruction based on the SAM series model according to claim 1, characterized in that: Step 6 is as follows: Step 6.1: Receive prompt information input by the user; Step 6.2: Positionally encode the prompt information and convert it into an embedded representation; Step 6.3: Combine the hint embedding with the feature map embedding as the input to the decoder. Step 6.4: Generate accurate object segmentation masks based on the cues in the segmentation decoder.

8. The method for bone tissue segmentation and reconstruction based on the SAM series model according to claim 1, characterized in that: Step 7 is as follows: Step 7.1: Format the target detection results to generate an output file containing the target category and location; Step 7.2: Resize the segmentation mask to match the input image resolution. Step 7.3: Save the detection and segmentation results in the specified format; Step 7.4: Output the processed detection and segmentation results for subsequent application or analysis.

Images