Craniopharyngeal tube tumor magnetic resonance image segmentation system and method based on basic large model

Abstract

The invention discloses a craniopharyngeal tube tumor magnetic resonance image segmentation system and method based on a basic large model. The basic large model comprises a three-dimensional image encoder, a three-dimensional prompt encoder and a three-dimensional mask decoder. An image segmentation reasoning module receives the pre-training weight of the basic large model and the three-dimensional craniopharyngeal tube tumor image data to carry out iterative segmentation reasoning to generate a craniopharyngeal tube tumor segmentation mask; the three-dimensional image encoder converts the three-dimensional image into image features; the interactive click generation module simulates a click operation of a user on the three-dimensional craniopharyngeal tubular tumor magnetic resonance imaging image to generate a click point prompt; the predictor module predicts, simulates and clicks to generate a corrected low-resolution segmentation mask in combination with the image features and click point prompts; the three-dimensional prompt encoder encodes the click point prompt and the low-resolution segmentation mask into prompt features; a three-dimensional mask decoder extracts a contextual relationship between the image feature and the cue feature to generate a segmentation mask. The segmentation quality is improved through an interactive function.

Description

Technical Field

[0001] The present invention relates to medical imaging analysis, and in particular to a system and method for segmenting a craniopharyngeal duct tumor magnetic resonance image based on a basic large model. Background Art

[0002] Craniopharynx duct tumors are intracranial sellar region tumors with deep lesions and complex adjacent structures. During surgical resection of a craniopharynx duct tumor, damage to adjacent sellar structures can lead to a poor prognosis, including postoperative complications such as cerebrospinal fluid leakage, intracranial infection, visual field loss, and diabetes insipidus. Accidental injury to the internal carotid artery can even result in death. Therefore, preoperative use of cranial magnetic resonance imaging (MRI) to distinguish tumors from normal tissue and achieve accurate segmentation of craniopharynx duct tumors is crucial for the diagnosis and treatment of craniopharynx duct tumor patients.

[0003] Deep learning continues to advance in the field of medical image processing, achieving remarkable results in segmentation tasks in particular. Existing deep learning-based segmentation algorithms primarily include U-net and Segment Anything Model (SAM) networks. In real-world clinical applications, batch processing of medical images is relatively rare, with interactive applications often focusing on single medical images. In such cases, interactive segmentation tasks using nnUnet are difficult to achieve. Furthermore, the overall generalization of U-net models is limited, requiring the training of multiple models to achieve segmentation of craniopharynx duct tumors and key structures in the sellar region, and personalized segmentation remains difficult. SAM performs well in 2D natural image segmentation, but due to a lack of general medical image knowledge, it is unable to process medical images and is not applicable to 3D images. MedSAM achieves its application in medical imaging by fine-tuning a decoder with 1.1 million masks. SAM-Med 2D, through comprehensive adaptation using adapters and approximately 20 million masks, demonstrates superior performance in general medical image segmentation. However, SAM-Med 2D processes volumetric images slice by slice: it decomposes the 3D data into 2D slices, processes each slice independently, and then aggregates the 2D results into 3D predictions. This ignores the 3D spatial information between slices and also performs poorly on 3D medical images of some diseases. As a result, SAM-Med 2D cannot effectively segment craniopharynx duct tumors using brain MRI.

[0004] Due to the unique bony structure of the sellar region, its limited space means that craniopharynx tumors often compress adjacent tissues such as the optic chiasm, pituitary gland, and cavernous sinus, making them difficult to distinguish. Furthermore, accidental damage to normal tissue during surgical resection can seriously affect the patient's prognosis and quality of life. Therefore, in actual clinical work, in order to provide more valuable segmentation results for surgical resection of craniopharynx tumor patients, a model with strong segmentation performance for craniopharynx tumors is required. The segmentation results of existing segmentation models cannot effectively enhance neurosurgeons' preoperative understanding of the structure and distribution of craniopharynx tumors.

[0005] On this basis, there is still a need for an intelligent segmentation method for craniopharynx duct tumors based on magnetic resonance imaging that is highly interactive and capable of recognizing three-dimensional medical images, so as to provide more accurate and complete craniopharynx duct tumor information to help neurosurgeons make clinical decisions. Summary of the Invention

[0006] The purpose of the present invention is to provide a craniopharyngeal duct tumor magnetic resonance image segmentation system and method based on a basic large model.

[0007] The technical solution adopted in the present invention is:

[0008] A craniopharynx tumor MRI image segmentation system based on a basic large model includes a 3D image encoder, a 3D hint encoder, and a 3D mask decoder. The system includes the following components:

[0009] Image Segmentation Inference Module (Infer Sequence): This module receives the pre-trained weights of the basic large model and 3D craniopharynx tumor image data, and uses the basic large model to infer the 3D craniopharynx tumor image data to generate a craniopharynx tumor segmentation mask.

[0010] Specifically, the image segmentation reasoning module implementation includes starting from the 3D image input in nii format, generating the craniopharynx tumor segmentation mask and saving it as a nii file.

[0011] 3D image encoder: used to process input 3D image data of different sizes and convert the input 3D image into image features represented by high-dimensional features;

[0012] Furthermore, the 3D image encoder uses ViT to extract information.

[0013] Specifically, the 3D image encoder is an image encoder module based on the 3D Vision Transformer (ViT), designed to process 3D image data (such as medical images). This module uses ViT to extract information, which enables the model to handle inputs of different sizes. ViT also includes a self-attention mechanism, which enables global perception. This module converts the input 3D image into a high-dimensional feature representation for subsequent tasks (such as segmentation and classification).

[0014] Interactive click generation module: used to simulate the user's interactive (click) operation on the 3D craniopharyngioma MRI image and generate interactive (click point) prompts for the segmentation task;

[0015] Specifically, click points are generated by analyzing the input image and the segmentation results of the previous round.

[0016] Predictor module: used to generate segmentation results by combining image embedding and click point prompts;

[0017] 3D Hint Encoder: It is used to encode the click point hints generated by simulating user input and the low-resolution segmentation mask of the predictor module into hint features in the 3D segmentation task, and map the hint features to the same high-dimensional space as the 3D image features; so that they can be input into the mask decoder together with the image embedding to generate the mask prediction results; that is, the 3D hint encoder encodes the click point information into features that the model can understand.

[0018] Furthermore, the 3D Hint Encoder encodes the hint information into sparse and dense embeddings. Sparse embeddings use 3D position encoding to represent subtle differences in 3D space, while dense embeddings are generated through 3D convolution. Sparse embeddings generate sparse embeddings based on the input point coordinates and encode them in combination with foreground / background labels; dense embeddings encode the input mask; if no mask is input, a default embedding is generated.

[0019] Specifically, the 3D hint encoder is used to embed the click point hints and masks generated by simulating user input into a high-dimensional space in the 3D segmentation task, so that they can be input into the mask decoder together with the image embedding to generate the mask prediction result. Sparse hints use 3D position encoding to represent the subtle differences in 3D space, while dense hints are processed through 3D convolution.

[0020] 3D mask decoder: used to extract the contextual relationship between image features and hint features to generate segmentation masks and predict mask quality scores; convert high-dimensional features back to segmentation results in 3D space and mark the target area in the image.

[0021] Specifically, the 3D mask decoder uses a Transformer-based architecture to process 3D image embedding and cue features through a multi-layer interactive attention mechanism. That is, the contextual relationship between the image and the cue is extracted through the Transformer, and the output of the Transformer is converted into the final mask through deconvolution and multi-layer perceptron.

[0022] Specifically, the interactive click generation module (Click Method), 3D prompt encoder (3D Prompt Encoder) and predictor module (Predictor) together constitute the simulated interaction module. These modules work together to form the entire interactive segmentation workflow.

[0023] The method for segmenting a craniopharyngeal duct tumor magnetic resonance image based on a basic large model comprises the following steps:

[0024] Step 1: Loading a pre-trained basic large model based on the obtained basic large model weight parameters; the basic large model includes a 3D image encoder, a 3D hint encoder, and a 3D mask decoder;

[0025] Step 2: Load the 3D image data and preprocess it (e.g., normalize, crop, or pad the image to a target size);

[0026] Step 3: Use the pre-trained basic large model combined with the simulated click points generated by the interactive click generation module to perform multiple rounds of iterative segmentation reasoning on the image features to generate the final segmentation result;

[0027] Step 4: resize the segmentation mask of the final segmentation result to the size of the original 3D image, and crop, fill, or restore the mask size to ensure that the segmentation result is aligned with the original image;

[0028] Step 5: Save the generated segmentation mask as a file in the specified format (nii.gz format), and record the click points and segmentation results. Specifically, in the set output path, save the inferred image, segmentation mask, click point location information, etc.

[0029] Furthermore, in step 1, ViT is used to extract three-dimensional image data of different sizes.

[0030] Furthermore, in step 3, a sliding window strategy is used to divide the 3D image into blocks, and segmentation reasoning is performed on each window to obtain the segmentation result, avoiding memory overload while ensuring that each area is covered by the model.

[0031] Furthermore, in step 3, in the first round of segmentation reasoning, initial click points are generated based on specified features of the three-dimensional image; the specified features are brightness threshold and image structure; the second round starts based on the segmentation results of the previous round, and the interactive click generation module automatically generates new click points in the misclassified areas of the foreground and background (misclassification of foreground and background) by comparing the segmentation results of the previous round with the original three-dimensional image.

[0032] Specifically, in the first round of reasoning, there are no segmentation results from the previous round, so click points can only be generated based on certain features of the image (such as brightness threshold, image structure, etc.), and do not depend on the segmentation results of the previous round. This step is equivalent to simulating the user's initial click. After the first segmentation result is obtained, the click points for the next round can be generated based on this segmentation result. The interactive click generation module automatically generates new click points in the error area (incorrect classification of foreground and background) by comparing the segmentation results and the image. This process is similar to manually adding prompt points to these areas after the user discovers an error. In each subsequent round, the generation and segmentation of click points are carried out step by step. Use the newly generated click points for the next round of reasoning to obtain more accurate segmentation results.

[0033] Furthermore, in each round of segmentation inference in step 3, the 3D hint encoder is called to encode the click point hints and masks generated by the simulated user input into hint features, and the hint features are mapped to the same high-dimensional space as the 3D image features; then the 3D mask decoder module is called to decode the segmentation mask of the segmentation result based on the image features from the 3D image encoder module and the hint features generated by the 3D hint encoder module.

[0034] Furthermore, step 3 specifically includes the following steps:

[0035] Step 3-1, using a 3D image encoder to convert the input 3D image into image features represented by high-dimensional features;

[0036] Step 3-2: Determine whether the first segmentation reasoning is performed; if so, use the interactive click generation module to generate the initial click point; otherwise, the interactive click generation module generates a new click point based on the segmentation results of the previous round;

[0037] Step 3-3, combining image features and click point hints to predict the segmentation results after simulated click correction, and generating a corrected low-resolution segmentation mask;

[0038] Step 3-4: Encode the click point hints and low-resolution segmentation masks generated by the simulated user input as hint features, and map the hint features to the same high-dimensional space as the three-dimensional image features;

[0039] Steps 3-5: Extract the contextual relationship between image features and hint features to generate a segmentation mask, and predict the mask quality score to form the segmentation result of the corresponding round; convert the high-dimensional features back to the segmentation result in three-dimensional space and mark the target area in the image;

[0040] Step 3-6: Repeat steps 3-2 through 3-5 until the specified number of iterations is reached. The final segmentation result is output as the final segmentation result. Specifically, this section is the core logic of the image segmentation inference module and consists of multiple iterative steps. Each iteration generates new click points based on the previous segmentation result and updates the segmentation result.

[0041] Furthermore, in step 3-2, new click points are generated in the misclassified areas of the foreground and background, specifically including the following steps:

[0042] Step 3-2-1, generate a pseudo-negative mask: perform threshold processing on the input 3D image to generate a potential foreground region, and perform a binarization operation on the previous round of segmentation results to generate a foreground mask predicted by the model; by comparing the potential foreground region with the foreground mask, identify the potential error region in the segmentation result that is mistakenly considered to be the foreground as a pseudo-negative region;

[0043] Specifically, the false negative mask refers to the area that is considered to be the foreground in the image but is incorrectly predicted as the background in the segmentation result. The identification of this area is the key to generating click points.

[0044] Step 3-2-2, click point generation: Generate click points for correcting segmentation results based on the pseudo-negative mask;

[0045] Step 3-2-3, output the three-dimensional coordinates of the click point and the corresponding label.

[0046] Furthermore, after the click point is generated in step 3-2-2, the three-dimensional coordinates (d, h, w) of the click point are generated through logical operations, representing the position in the depth, width and height directions respectively; this precise three-dimensional coordinate can help the model better focus on the three-dimensional area that needs to be corrected. When the false negative area does not exist (for example, the segmentation result of the model is relatively consistent with the expectation), a simplified prompt information generation strategy is used to generate the click point, and the prompt information is generated by randomly selecting a point outside the segmented area as the click point. That is, the module will randomly generate a click point in the image, usually as a negative sample (background). This strategy ensures that the model can still receive prompt information even if there is no obvious error area.

[0047] Specifically, in some cases, the module can bypass complex thresholding operations and instead generate an uncertainty region directly based on the previous segmentation results. This strategy randomly selects points outside the segmented region as click points to generate prompt information. This mechanism is simple and effective, making it suitable for scenarios where prompt generation is required quickly.

[0048] Furthermore, the 3D coordinates of the clicked points in step 3-2-3 are used to identify the region of interest or the segmentation uncertainty region; the corresponding labels are used to indicate whether the pixel is foreground or background; foreground indicates that the pixel belongs to the region of interest, while background indicates that the pixel does not belong to the foreground. These clicked points and labels are used as input to the model's next round of segmentation tasks, providing hints for improving the current segmentation.

[0049] Furthermore, step 3-3 specifically includes the following steps:

[0050] In step 3-3-1, the preprocessed 3D image is tensorized to generate an image in PyTorch format. This is used to obtain an embedded representation of the currently set image, improving computational efficiency, incorporating global information for accurate segmentation, and supporting subsequent tasks or multiple interactive adjustments. The shape of the embedded representation is typically 1xCxHxW, where C is the embedding dimension and H and W are the spatial dimensions of the embedding.

[0051] Step 3-3-2: Call the 3D image encoder to generate the feature embedding of the image in PyTorch format

[0052] Step 3-3-3, save the original size and input size of the 3D image for subsequent mask upsampling and restoration to the original size;

[0053] Step 3-3-4, perform mask prediction based on the click points and low-resolution mask hint information generated by the simulation of the current round; that is, convert the generated click points into sparse and dense embedding representations through the 3D hint encoder; use the 3D mask decoder to generate low-resolution masks and IoU prediction values; upsample the low-resolution mask to the original image size.

[0054] Step 3-3-5, obtain the embedded representation of the currently set image, combine global information for accurate segmentation, and support subsequent tasks or multiple interactive adjustments; the shape of the embedded representation is 1xCxHxW, where C is the embedding dimension, and H and W are the spatial dimensions of the embedding.

[0055] The present invention adopts the above technical solution. The basic large model is trained using craniopharynx tumor magnetic resonance imaging data, and the obtained pre-trained model has strong performance in craniopharynx tumor image segmentation. Secondly, the different components of the basic large model jointly participate in the processing of three-dimensional images, which can directly integrate three-dimensional spatial information and perform well in three-dimensional medical image segmentation. In addition, the basic large model generates foreground or background click points to simulate the interactive scene of how users click to prompt the target area of ​​the model, reducing the dependence on manual click points, avoiding the time-consuming and labor-intensive problem of manually generating a large number of click points and the difficulty in ensuring consistency, speeding up the development and tuning of the model, and having good interactivity. This patent designs a pre-trained basic large model specifically for craniopharynx tumor segmentation, which can solve the problems of low model interactivity and poor performance when facing three-dimensional medical images during segmentation tasks, and improves the accuracy of craniopharynx tumor segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0057] Figure 1 Schematic diagram of the process of the magnetic resonance image segmentation method of craniopharyngeal duct tumor based on the basic large model of the present invention;

[0058] Figure 2 A schematic diagram of the basic large model architecture. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0060] like Figure 1 As shown in FIG2 , the present invention discloses a craniopharynx tumor magnetic resonance image segmentation system based on a basic large model, wherein the basic large model includes a 3D image encoder, a 3D hint encoder, and a 3D mask decoder; the system includes the following parts:

[0061] Image Segmentation Inference Module (Infer Sequence): This module receives the pre-trained weights of the basic large model and 3D craniopharynx tumor image data, and uses the basic large model to infer the 3D craniopharynx tumor image data to generate a craniopharynx tumor segmentation mask.

[0062] Specifically, the image segmentation reasoning module implementation includes starting from the 3D image input in nii format, generating the craniopharynx tumor segmentation mask and saving it as a nii file.

[0063] 3D image encoder: used to process input 3D image data of different sizes and convert the input 3D image into image features represented by high-dimensional features;

[0064] Furthermore, the 3D image encoder uses ViT to extract information.

[0065] Specifically, the 3D image encoder is an image encoder module based on the 3D Vision Transformer (ViT), designed to process 3D image data (such as medical images). This module uses ViT to extract information, which enables the model to handle inputs of different sizes. ViT also includes a self-attention mechanism, which enables global perception. This module converts the input 3D image into a high-dimensional feature representation for subsequent tasks (such as segmentation and classification).

[0066] Interactive click generation module: used to simulate the user's interactive (click) operation on the 3D craniopharyngioma MRI image and generate interactive (click point) prompts for the segmentation task;

[0067] Specifically, click points are generated by analyzing the input image and the segmentation results of the previous round.

[0068] Predictor module: used to generate segmentation results by combining image embedding and click point prompts;

[0069] 3D Hint Encoder: It is used to encode the click point hints and masks generated by simulated user input into hint features in the 3D segmentation task, and map the hint features to the same high-dimensional space as the 3D image features; so that they can be input into the mask decoder together with the image embedding to generate the mask prediction results; that is, the 3D hint encoder encodes the click point information into features that the model can understand.

[0070] Furthermore, the 3D Hint Encoder encodes the hint information into sparse and dense embeddings. Sparse embeddings use 3D position encoding to represent subtle differences in 3D space, while dense embeddings are generated through 3D convolution. Sparse embeddings generate sparse embeddings based on the input point coordinates and encode them in combination with foreground / background labels; dense embeddings encode the input mask; if no mask is input, a default embedding is generated.

[0071] Specifically, the 3D hint encoder is used to embed the click point hints and masks generated by simulating user input into a high-dimensional space in the 3D segmentation task, so that they can be input into the mask decoder together with the image embedding to generate the mask prediction result. Sparse hints use 3D position encoding to represent the subtle differences in 3D space, while dense hints are processed through 3D convolution.

[0072] 3D mask decoder: used to extract the contextual relationship between image features and hint features to generate segmentation masks and predict mask quality scores; convert high-dimensional features back to segmentation results in 3D space and mark the target area in the image.

[0073] Specifically, the 3D mask decoder uses a Transformer-based architecture to process 3D image embedding and cue features through a multi-layer interactive attention mechanism. That is, the contextual relationship between the image and the cue is extracted through the Transformer, and the output of the Transformer is converted into the final mask through deconvolution and multi-layer perceptron.

[0074] Specifically, the interactive click generation module (Click Method), 3D prompt encoder (3D Prompt Encoder) and predictor module (Predictor) together constitute the simulated interaction module. These modules work together to form the entire interactive segmentation workflow.

[0075] It should be noted that the image segmentation inference module generates a segmentation mask based on the weights of the pre-trained model, generating a preliminary image segmentation result. The predictor module generates a segmentation result by further modifying the low-resolution segmentation result obtained by combining the simulated click point interaction information generated by the primary segmentation result with the extracted image features. The 3D hint encoder processes the low-resolution segmentation result to obtain hint features. Furthermore, the 3D mask decoder generates a segmentation mask, which represents the final segmentation result generated based on these hint features.

[0076] The method for segmenting a craniopharyngeal duct tumor magnetic resonance image based on a basic large model comprises the following steps:

[0077] Step 1: Loading a pre-trained basic large model based on the obtained basic large model weight parameters; the basic large model includes a 3D image encoder, a 3D hint encoder, and a 3D mask decoder;

[0078] Step 2: Load the 3D image data and preprocess it (e.g., normalize, crop, or pad the image to a target size);

[0079] Step 3: Use the pre-trained basic model combined with the simulated click points generated by the interactive click generation module to perform multiple rounds of iterative segmentation reasoning on the image features to generate the final segmentation result;

[0080] Step 4: resize the segmentation mask of the final segmentation result to the size of the original 3D image, and crop, fill, or restore the mask size to ensure that the segmentation result is aligned with the original image;

[0081] Step 5: Save the generated segmentation mask as a file in the specified format (nii.gz format), and record the click points and segmentation results. Specifically, in the set output path, save the inferred image, segmentation mask, click point location information, etc.

[0082] Furthermore, in step 1, ViT is used to extract three-dimensional image data of different sizes.

[0083] Furthermore, in step 3, a sliding window strategy is used to divide the 3D image into blocks, and segmentation reasoning is performed on each window to obtain the segmentation result, avoiding memory overload while ensuring that each area is covered by the model.

[0084] Furthermore, in step 3, in the first round of segmentation reasoning, initial click points are generated based on specified features of the three-dimensional image; the specified features are brightness threshold and image structure; the second round starts based on the segmentation results of the previous round, and the interactive click generation module automatically generates new click points in the misclassified areas of the foreground and background (misclassification of foreground and background) by comparing the segmentation results of the previous round with the original three-dimensional image.

[0085] Specifically, in the first round of reasoning, there are no segmentation results from the previous round, so click points can only be generated based on certain features of the image (such as brightness threshold, image structure, etc.), and do not depend on the segmentation results of the previous round. This step is equivalent to simulating the user's initial click. After the first segmentation result is obtained, the click points for the next round can be generated based on this segmentation result. The interactive click generation module automatically generates new click points in the error area (incorrect classification of foreground and background) by comparing the segmentation results and the image. This process is similar to manually adding prompt points to these areas after the user discovers an error. In each subsequent round, the generation and segmentation of click points are carried out step by step. Use the newly generated click points for the next round of reasoning to obtain more accurate segmentation results.

[0086] Furthermore, in each round of segmentation inference in step 3, the 3D hint encoder is called to encode the click point hints and masks generated by the simulated user input into hint features, and the hint features are mapped to the same high-dimensional space as the 3D image features; then the 3D mask decoder module is called to decode the segmentation mask of the segmentation result based on the image features from the 3D image encoder module and the hint features generated by the 3D hint encoder module.

[0087] Furthermore, step 3 specifically includes the following steps:

[0088] Step 3-1, using a 3D image encoder to convert the input 3D image into image features represented by high-dimensional features;

[0089] Step 3-2: Determine whether the first segmentation reasoning is performed; if so, use the interactive click generation module to generate the initial click point; otherwise, the interactive click generation module generates a new click point based on the segmentation results of the previous round;

[0090] Step 3-3, combining image features and click point hints to predict the segmentation results after simulated click correction, and generating a corrected low-resolution segmentation mask;

[0091] Step 3-4: Encode the click point hints and low-resolution segmentation masks generated by the simulated user input as hint features, and map the hint features to the same high-dimensional space as the three-dimensional image features;

[0092] Steps 3-5: Extract the contextual relationship between image features and hint features to generate a segmentation mask, and predict the mask quality score to form the segmentation result of the corresponding round; convert the high-dimensional features back to the segmentation result in three-dimensional space and mark the target area in the image;

[0093] Step 3-6: Repeat steps 3-2 through 3-5 until the specified number of iterations is reached. The final segmentation result is output as the final segmentation result. Specifically, this section is the core logic of the image segmentation inference module and consists of multiple iterative steps. Each iteration generates new click points based on the previous segmentation result and updates the segmentation result.

[0094] Furthermore, in step 3-2, new click points are generated in the misclassified areas of the foreground and background, specifically including the following steps:

[0095] Step 3-2-1, generate a pseudo-negative mask: perform threshold processing on the input 3D image to generate a potential foreground region, and perform a binarization operation on the previous round of segmentation results to generate a foreground mask predicted by the model; by comparing the potential foreground region with the foreground mask, identify the potential error region in the segmentation result that is mistakenly considered to be the foreground as a pseudo-negative region;

[0096] Specifically, the false negative mask refers to the area that is considered to be the foreground in the image but is incorrectly predicted as the background in the segmentation result. The identification of this area is the key to generating click points.

[0097] Step 3-2-2, click point generation: Generate click points for correcting segmentation results based on the pseudo-negative mask;

[0098] Step 3-2-3, output the three-dimensional coordinates of the click point and the corresponding label.

[0099] Furthermore, after the click point is generated in step 3-2-2, the three-dimensional coordinates (d, h, w) of the click point are generated through logical operations, representing the position in the depth, width and height directions respectively; this precise three-dimensional coordinate can help the model better focus on the three-dimensional area that needs to be corrected. When the false negative area does not exist (for example, the segmentation result of the model is relatively consistent with the expectation), a simplified prompt information generation strategy is used to generate the click point, and the prompt information is generated by randomly selecting a point outside the segmented area as the click point. That is, the module will randomly generate a click point in the image, usually as a negative sample (background). This strategy ensures that the model can still receive prompt information even if there is no obvious error area.

[0100] Specifically, in some cases, the module can bypass complex thresholding operations and instead generate an uncertainty region directly based on the previous segmentation results. This strategy randomly selects points outside the segmented region as click points to generate prompt information. This mechanism is simple and effective, making it suitable for scenarios where prompt generation is required quickly.

[0101] Furthermore, the 3D coordinates of the clicked points in step 3-2-3 are used to identify the region of interest or the segmentation uncertainty region; the corresponding labels are used to indicate whether the pixel is foreground or background; foreground indicates that the pixel belongs to the region of interest, while background indicates that the pixel does not belong to the foreground. These clicked points and labels are used as input to the model's next round of segmentation tasks, providing hints for improving the current segmentation.

[0102] Furthermore, step 3-3 specifically includes the following steps:

[0103] In step 3-3-1, the preprocessed 3D image is tensorized to generate an image in PyTorch format. This is used to obtain an embedded representation of the currently set image, improving computational efficiency, incorporating global information for accurate segmentation, and supporting subsequent tasks or multiple interactive adjustments. The shape of the embedded representation is typically 1xCxHxW, where C is the embedding dimension and H and W are the spatial dimensions of the embedding.

[0104] Step 3-3-2: Call the 3D image encoder to generate the feature embedding of the image in PyTorch format

[0105] Step 3-3-3, save the original size and input size of the 3D image for subsequent mask upsampling and restoration to the original size;

[0106] Step 3-3-4, perform mask prediction based on the click points and low-resolution mask hint information generated by the simulation of the current round; that is, convert the generated click points into sparse and dense embedding representations through the 3D hint encoder; use the 3D mask decoder to generate low-resolution masks and IoU prediction values; upsample the low-resolution mask to the original image size.

[0107] Step 3-3-5, obtain the embedded representation of the currently set image, combine global information for accurate segmentation, and support subsequent tasks or multiple interactive adjustments; the shape of the embedded representation is 1xCxHxW, where C is the embedding dimension, and H and W are the spatial dimensions of the embedding.

[0108] The principle of the present invention is described in detail below:

[0109] The basic large model adopted by the present invention includes a 3D image encoder, a 3D hint encoder and a 3D mask decoder.

[0110] 1) 3D image encoder module The 3D module is a 3D ViT encoder, which first divides the 3D image into multiple patches, and then captures the dependencies between patches through a series of Transformer blocks. The window mechanism and attention mechanism help the model to perform efficient calculations on large-scale inputs. Finally, the feature map is further processed and adjusted to output a 3D feature map suitable for downstream tasks (such as segmentation, detection, etc.). This module uses a 3D convolution with a kernel size of (16, 16, 16) to embed patches; paired with a learnable 3D absolute position encoding obtained by naturally expanding the 2D absolute position (PE) by an extra dimension; the embedding of the patch will be input into the 3D attention module. For the 3D attention block, the 3D relative position encoding is incorporated into the Multi-Head Self Attention (MHSA) module, enabling it to directly capture spatial details. The 3D image encoder module mainly includes the following contents:

[0111] 1.1) Embedding 3D Patches: The input 3D image is divided into several 3D patches, each of which is mapped to an embedding space. Patch size and stride are controlled through convolution, and the dimensions of the final output feature map are rearranged into [B, D, H, W, C], representing batch, depth, height, width, and number of channels.

[0112] 1.2) Embedding Position Information: Since the Transformer has no inherent perception of the spatial position of the input, explicit position information must be added. This module defines a 5D position embedding parameter matrix corresponding to the patch size of the 3D image, encoding the spatial position information of each patch in the image through absolute position embedding.

[0113] 1.3) Self-Attention: The 3D image encoder module relies on the self-attention mechanism to handle the global relationship between patches, which is the core of the Transformer. The attention mechanism calculates the similarity between different parts of the input patch, allowing the model to focus on the most important parts. The formula of the self-attention mechanism is:

[0114]

[0115] Where: Q is the query matrix, generated from the prompt features. K and V are the key and value matrices, respectively, generated from the image features. k is the dimension of the key and is used for scaling. The softmax function ensures the normalization of the attention weights.

[0116] 1.4) Transformer Architecture: Responsible for performing self-attention calculations on patch representations and capturing dependencies between patches at each layer. Each block contains a standard Transformer architecture, including a self-attention mechanism (Attention) and a multi-layer perceptron (MLP).

[0117] The self-attention mechanism consists of the following modules: Multi-Head Attention: Multiple attention heads calculate attention weights in parallel to capture the correlation between patches. Each attention head is responsible for a different feature space, thereby enhancing the model's expressiveness. Relative Positional Encoding: The model incorporates relative position offset information when calculating attention, allowing the model to perceive the relative distance between patches.

[0118] The Multi-Layer Perceptron (MLP Block) is used to perform nonlinear transformations on the features of each patch. The MLP Block contains two layers of linear transformations and activation functions to further improve the expressiveness of the model.

[0119] 1.5) Feature Processing Module (Neck): The high-dimensional features output by the Transformer are processed by the convolutional layers in the Neck section to adjust the number of channels. Spatial convolution is then applied to the feature map to further process the patch output. A Conv3D layer is first used to reduce the feature dimensionality to the specified number of channels. LayerNorm3D is then used to normalize the features of each channel to prevent vanishing or exploding gradients in deep networks. Finally, another Conv3D and LayerNorm3D layer are used to further process the features, outputting the final 3D feature map.

[0120] 2) The 3D Hint Encoder module is a 3D hint encoder that embeds hints (such as click points and masks) that simulate user input in 3D segmentation tasks into a high-dimensional space so that they can be input into the mask decoder along with the image embedding to generate mask prediction results. It processes multiple hints and encodes this information into sparse and dense embeddings, helping the model better understand user hints and image features. This module mainly includes:

[0121] 2.1) Hint Input: Hints are provided to the 3D craniopharyngioma MRI images through click points, bounding boxes, etc. Click points identify the region of interest or excluded regions (positive clicks and negative clicks). The core part is to convert the click point hints into feature representations that can be understood by the Transformer. The position of the click point in the 3D image can be expressed as (d, h, w), which represents the coordinates of the click point in the depth, width, and height directions. The formula for hint embedding is:

[0122] p i =W prompt ·δ (d,h,w) +b prompt ;

[0123] Where: p i is the embedding representation of the click point; W prompt is the weight matrix of the cue embedding; δ (d,h,w) It is the position representation of the click point in three-dimensional space (the network is similar to One-Hot encoding, indicating the specific location of the click); b prompt is the bias term.

[0124] 2.2) Feature Fusion: The 3D Cue Encoder module combines simulated user click point cues with generated image features to guide the model to adjust segmentation boundaries based on the cues. Multiple cues (e.g., click points, boxes, masks) are encoded to generate sparse and dense embeddings, respectively. These embeddings are fed into the segmentation model's decoder for mask prediction.

[0125] Sparse embedding: Generates sparse embeddings based on the input point coordinates and encodes them in combination with foreground / background labels.

[0126] Dense embedding: Encodes the input mask. If no mask is input, a default embedding is generated.

[0127] 2.3) Spatial Mapping: The 3D cue encoder module is responsible for mapping these cues to the same space as the 3D image features. This requires considering the location of the click point cue in the 3D craniopharyngioma MRI image data and generating a feature representation associated with each patch. Generate a random 3D position embedding for encoding the input 3D coordinates. By applying a Gaussian random matrix to the coordinates, the 3D coordinates are mapped to a higher-dimensional feature space. Given the input 3D coordinates, the position embedding is calculated as follows:

[0128] a) Linear transformation: coords transformed =coords·W; where coords is the input coordinates and W is a Gaussian random matrix used to map the three-dimensional coordinates to a high-dimensional space.

[0129] b) Sine-cosine encoding: The linearly transformed coordinates are encoded using sine and cosine functions:

[0130] PE sin =sin(2π·coords transformed );

[0131] PE cos =cos(2π·coords transformed );

[0132] c) Concatenation results: Concatenate the sine and cosine results together to form a complete embedding representation:

[0133] PE(coords)=[sin(2π·coords transformed ),cos(2π·coords transformed )].

[0134] 3) The 3D mask decoder module generates segmentation masks from image and cue embeddings using a bidirectional Transformer and a multi-layer perceptron. Its core mechanism is to extract the contextual relationship between the image and the cue through the Transformer and convert the Transformer output into the final mask through deconvolution and a multi-layer perceptron. This design enables the model to efficiently process 3D data and generate high-quality segmentation masks based on different cues.

[0135] The main task is to decode the final segmentation mask based on the image features from the 3D image encoder module and the hint features generated by the 3D hint encoder module. This module is responsible for converting the high-dimensional features back to the segmentation results in 3D space and marking the target area in the image. This module mainly includes:

[0136] 3.1) 3D Data Input Processing: 3D image input is converted into embedded features through a convolutional layer. These features preserve the image's 3D spatial structure. The image embedding has the dimensions (B, C, D, H, W), where B is the batch size, C is the number of channels, and D, H, and W are the depth, height, and width, respectively. This dimensionality demonstrates that the model preserves the image's depth information while processing the features of each 3D volume pixel.

[0137] 3.2) Transformer Extension for 3D Data: The 3D Bidirectional Transformer is an extension of the standard Transformer for 3D space, capable of processing tokens embedded in 3D images. It flattens the 3D image into a sequence of tokens and uses an attention mechanism to capture long-range dependencies between each volume pixel.

[0138] In 3D data, due to the greater number of volumetric pixels and the presence of depth information, the standard 2D self-attention mechanism has difficulty capturing these 3D dependencies. However, the 3D bidirectional Transformer can effectively process the interactions between volumetric pixels in 3D space, thereby capturing more complex spatial relationships.

[0139] 3.3) 3D Segmentation Mask Generation: The 3D Mask Decoder module upsamples the low-resolution mask output by the Transformer back to the original resolution through 3D deconvolution (ConvTranspose). This 3D upsampling is performed in depth, width, and height, resulting in a higher resolution segmentation mask in 3D space.

[0140] The upsampled 3D mask has dimensions (B, D, H, W), allowing the model to accurately segment every volumetric pixel in the 3D image. This 3D feature map-based processing ensures that the model can generate accurate segmentation results in 3D space.

[0141] 3.4) Interaction of Sparse and Dense Embeddings: Cues are converted into sparse and dense embeddings, which interact with the image embedding via a Transformer. Sparse embeddings are generated from click points and bounding boxes, representing regions of interest explicitly indicated by the user. These embeddings interact with the image embedding via self-attention and cross-attention layers, allowing the model to focus on the corresponding regions in the image based on these cues. Dense embeddings are generated from masks, representing the model's understanding of the entire image. By combining dense mask embeddings with image embeddings, the model can combine global information and local cues for more accurate segmentation.

[0142] 3.5) Interaction of Transformer Attention Mechanism: In the 3D bidirectional Transformer, the model first allows the cue features (click points and bounding boxes) to interact with each other, and then allows the cue features to interact with the image embedding. This interaction mechanism can model the cue information in both the global and local scope of the image, ensuring that the cue information can guide the model to generate more accurate segmentation masks.

[0143] 4) The 3D Prompt Encoder module, along with the ClickMethod module and the Predictor module, form the simulated interaction component. The ClickMethod module simulates user interaction to generate click point information; the 3D Prompt Encoder module encodes click information to generate model features; and the Predictor module combines image features and prompt information to generate segmentation results. These modules work together to form the entire simulated interactive segmentation workflow.

[0144] 4.1) The interactive click generation module's primary function is to provide interactive prompts for the segmentation task by generating click points, simulating user clicks on 3D craniopharyngioma MRI images. This module generates prompt points based on the model's previous segmentation results and input image information. These prompt points can be used in interactive segmentation tasks to simulate user clicks and improve segmentation results. The module's input data is a 3D image with a shape of (B, D, H, W), where B represents the batch size, D represents the depth, H represents the height, and W represents the width; the segmentation result prev_seg is the result predicted by the model in the previous segmentation round; and the input image img_3D is the 3D image data currently being used for segmentation.

[0145] Cue point generation: The module generates click points by analyzing the input image and the previous segmentation results. The goal of cue point generation is to locate segmentation errors or uncertain areas, and use these cue points to guide the model to improve the performance in the next segmentation round. The specific steps of cue point generation in this module include:

[0146] S1. Generate False Negative Mask: False negative mask refers to the area that is considered as foreground in the image but is incorrectly predicted as background in the segmentation result. Identifying this area is the key to generating click points. Step by step:

[0147] a) Generate possible regions representing the foreground by thresholding the input image img_3D. Typically, a threshold is used to determine which voxels (volume pixels) are likely to belong to the foreground.

[0148] b) Generate the foreground mask predicted by the model by binarizing the segmentation result of the previous round.

[0149] c) Perform comparisons through logical operations to identify those areas that were considered foreground in the input image but were predicted as background in the segmentation result. These false negative areas represent potential errors in the segmentation result.

[0150] S2. Click point generation: Based on the pseudo-negative mask, click points are generated to correct the segmentation results. This is divided into:

[0151] a) 3D coordinates (d, h, w) of the click point: After a click point is generated, logical operations are performed to generate the 3D coordinates (d, h, w) of the click point, representing its position in depth, width, and height, respectively. These precise 3D coordinates help the model better focus on the 3D area requiring correction.

[0152] b) Random click point generation: If a false negative region does not exist (for example, the model's segmentation results are consistent with expectations), the module will randomly generate a click point in the image, usually as a negative sample (background). This strategy ensures that the model can still receive cues even if there are no obvious false positives.

[0153] S3. Simplified Hint Generation Strategy: In some cases, the module can bypass complex thresholding and instead generate an uncertainty region directly based on the previous segmentation results. This strategy generates a hint by randomly selecting a point outside the segmented region as the click point. This simple and effective mechanism is suitable for scenarios requiring fast hint generation.

[0154] This module outputs: a) 3D coordinates of click points (batch_points): generated click point coordinates, used to identify regions of interest or segment uncertain regions. b) corresponding labels (batch_labels): labels of click points, used to indicate whether the point is foreground (1) or background (0). Foreground indicates that the point belongs to the region of interest (usually the lesion area that needs to be segmented), while background indicates that the point does not belong to the foreground. These click points and labels will be used as input for the next round of segmentation tasks of the model, providing hints for improving the current segmentation.

[0155] 4.2) The predictor module is the core component that actually performs inference. It is responsible for processing image features and encoded cue features (such as click points) to generate the final segmentation mask. This module inputs data into the 3D image encoder and 3D cue encoder, processes the features through these modules, and finally generates the prediction result. The click point information of the simulated user interaction prompt will affect the model's craniopharyngioma segmentation results in real time. This module can dynamically adjust based on the input and quickly respond to click point information during the segmentation process. This module mainly includes:

[0156] S1. Image settings:

[0157] a) Image format conversion: Convert the image format to RGB or BGR.

[0158] b) Image transformation: Scale the image to the size expected by the model.

[0159] c) Image tensor quantization: Convert the numpy array to PyTorch's Tensor format for compatibility with the model.

[0160] S2.Torch version image setup: used to receive images converted to PyTorch format and calculate their embeddings. The key steps are as follows:

[0161] a) Generate image embedding: Call the 3D image encoder to generate feature embedding of the image.

[0162] b) Record the original size and input size: The original size and input size of the image are stored for subsequent mask upsampling and restoration to the original size.

[0163] S3. Mask Prediction: Receives the simulated click point and low-resolution mask hints and uses them to predict the mask. Specific implementation steps include: Coordinate Transformation: transforming the click point information to the model input size based on the input image size; further calculating the mask; Post-Processing: restoring the predicted mask to the original image size and returning the mask, predicted IoU, and low-resolution mask.

[0164] S4.PyTorch mask prediction: This is the core part of the model for actual mask prediction, processing the prompt information after coordinate transformation, calling various components of the model for segmentation prediction, and outputting low-resolution segmentation results. Main steps:

[0165] a) Embedding Hints: The generated click points are converted into sparse and dense embedding representations through the 3D Hint Encoder. b) Mask Decoding: The model’s 3D Mask Decoder is used to generate low-resolution masks and IoU predictions. c) Post-processing: The low-resolution masks are upsampled to the original image size.

[0166] S5. Obtain Image Embedding: This step obtains an embedded representation of the current image, improving computational efficiency, incorporating global information for accurate segmentation, and supporting subsequent tasks or multiple interactive adjustments. The shape of the embedded representation is typically 1xCxHxW, where C is the embedding dimension, and H and W are the spatial dimensions of the embedding.

[0167] 5) The image segmentation inference module implements inference on 3D craniopharynx duct tumor imaging data using a pre-trained basic large model. This module uses the pre-trained model in combination with simulated click points to generate the final segmentation results, including the entire inference process from inputting a 3D image in NII format to generating a segmentation mask and saving it as an NII file. This module supports multiple click interactions, with its core purpose being to improve segmentation accuracy through a small number of interactive prompts. Each click further optimizes the model's segmentation results. The specific process is as follows:

[0168] a) Input processing: This module obtains various parameters from user input, including the path of test data, the path of model weights, whether to skip existing inference results, etc.

[0169] b) Data loading: Load 3D image data and perform preprocessing operations (such as normalization, cropping, or padding the image to the target size). Each image is loaded as a 3D tensor. The batch_size is set to 1, processing one 3D image at a time.

[0170] ToCanonical function: Converts the image direction to a standardized direction to ensure that all images follow the same coordinate system standard for subsequent processing.

[0171] CropOrPad function: Ensures that each input image is cropped or padded to the specified target size of 128x128x128, unifies the size of the input image, and avoids the model being unable to process due to inconsistent image sizes.

[0172] 5.3) Model loading: Load the pre-trained basic large model.

[0173] 5.4) Simulating Interaction to Generate Click Points: The interactive click generation module is called to generate click points based on the image and the model's previous segmentation results. Click points serve as user interactions in subsequent reasoning, similar to simulating a user labeling the foreground and background on an image. The interactive click generation module generates prompt information as follows:

[0174] a) Generating Initial Click Points: In the first round of inference, before the segmentation results from the previous round are available, click points are generated based solely on certain image features (such as brightness thresholds and image structure) and are independent of the previous round's segmentation results. This step simulates the user's initial click.

[0175] b) Generate new click points for the next round: After obtaining the first segmentation result, the next round of click points can be generated based on this segmentation result. The interactive click generation module automatically generates new click points in error areas (foreground and background misclassification) by comparing the segmentation result with the image. This process is similar to manually adding cue points in these areas after a user discovers an error.

[0176] 5.5) Sliding Window Inference: Use a sliding window strategy to divide the image into blocks and run model inference on each window to avoid memory overload while ensuring that every area is covered by the model.

[0177] a) Initialized Inference: The model performs its first inference using the initially generated click points to obtain a preliminary segmentation result. This segmentation result is generated based on the initial click point hints and the pre-trained model's understanding of the image.

[0178] b) Subsequent reasoning and segmentation: The generation and segmentation of click points in each round are performed step by step. The newly generated click points are used for the next round of reasoning to obtain more accurate segmentation results.

[0179] c) Multi-round interactive reasoning: This part is the core logic of the image segmentation reasoning module and consists of multiple iterative steps. Each round of iteration generates new click points based on the segmentation results of the previous round and updates the segmentation results.

[0180] 5.6) Mask processing: The segmentation mask generated after inference is adjusted to the size of the original image and further cropped, padded, or restored to ensure that the segmentation result is aligned with the original image.

[0181] 5.7) Saving Results: After inference is complete, the generated segmentation mask is saved as a nii.gz file, along with the click points and segmentation results. The inferred image, segmentation mask, and click point locations are stored in the output directory.

[0182] This invention utilizes craniopharyngioma MRI data to design a basic large-scale model with pre-trained weights. This model can perform craniopharyngioma segmentation tasks in clinical practice, enhance neurosurgeons' personalized understanding of craniopharyngioma in patients, and serve as an adjunct to treatment. The invention has three major features: 1. Using craniopharyngioma MRI data for training, the resulting pre-trained model demonstrates strong performance in craniopharyngioma image segmentation; 2. Multiple modules jointly process three-dimensional images, directly integrating three-dimensional spatial information and excelling in three-dimensional medical image segmentation; 3. Simulated interaction is generated based on click point information from craniopharyngioma images, improving the accuracy of individualized segmentation for craniopharyngioma patients. Using the pre-trained large-scale model and interactive module of the present invention, users can conveniently use brain MRI images to accurately segment craniopharyngioma images. This basic large-scale model generates simulated interactive information, features multiple modules for processing interactive information, and an overall three-dimensional structural design. Fine-tuned using craniopharyngioma MRI data, it can accurately segment craniopharyngioma.

[0183] The present invention adopts the above technical solution and has the following technical features: 1. The pre-trained model is fine-tuned using craniopharyngioma magnetic resonance imaging data, which improves the stability of the model and obtains a pre-trained large model for craniopharyngioma, making the model segmentation of craniopharyngioma more accurate and efficient. 2. The basic large model of the present invention uses the three-dimensional structure of the three-dimensional image encoder, the three-dimensional prompt encoder and the three-dimensional mask decoder to directly capture spatial information. The three-dimensional position encoding of different components is used to directly integrate three-dimensional spatial information, and it performs excellently in three-dimensional medical image segmentation. 3. The interactive functions of the interactive click generation module, the three-dimensional prompt encoder module, and the predictor module introduce manual supervision in the segmentation process, generate and process prompt information that simulates user interaction, reduce dependence on manual click points, avoid the time-consuming and labor-intensive problem of manually generating a large number of click points and the difficulty in ensuring consistency, accelerate the development and tuning process of the model, and make this basic large model have powerful interactive functions, which can handle complex three-dimensional craniopharyngioma magnetic resonance imaging image segmentation tasks in an individualized and targeted manner, and improve segmentation quality.

[0184] The present invention generates simulated interactive information through a basic large model, which has multiple interactive modules and an overall three-dimensional structural design. Based on these two features, the pre-trained model is fine-tuned using craniopharyngioma magnetic resonance imaging data to make the segmentation results more reliable, which helps to provide better individualized treatment for craniopharyngioma patients.

[0185] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

Claims

1. A craniopharynx tumor MRI image segmentation system based on a large basic model, comprising a 3D image encoder, a 3D cue encoder, and a 3D mask decoder; characterized by: The system consists of the following parts: Image segmentation inference module: This module receives the pre-trained weights of the basic large model and 3D craniopharynx tumor image data, and uses the basic large model to perform multiple rounds of iterative segmentation inference on the 3D craniopharynx tumor image data to generate a craniopharynx tumor segmentation mask. 3D image encoder: used to process input 3D image data of different sizes and convert the input 3D image into image features represented by high-dimensional features; Interactive click generation module: used to simulate the user's click operation on the 3D craniopharyngioma MRI image, and generate click point prompts for the segmentation task based on the previous round of segmentation results and image features; Predictor module: used to combine image features and click point hints to predict the segmentation results after simulated click correction and generate the corrected low-resolution segmentation mask; 3D Hint Encoder: used to encode the click point hints generated by simulated user input and the low-resolution segmentation mask of the predictor module into hint features in the 3D segmentation task, and map the hint features to the same high-dimensional space as the 3D image features; 3D mask decoder: used to extract the contextual relationship between image features and hint features to generate segmentation masks, and predict the mask quality score to form the segmentation result of the corresponding round; convert high-dimensional features back to segmentation results in 3D space and mark the target area in the image.

2. The craniopharynx tumor magnetic resonance image segmentation system based on the basic large model according to claim 1 is characterized by: The 3D hint encoder encodes hint information into sparse embeddings and dense embeddings. The sparse embeddings use 3D position encoding to represent the subtle differences in 3D space, while the dense embeddings are generated through 3D convolution. Sparse embedding generates sparse embeddings based on the input point coordinates and encodes them in combination with foreground and background labels; dense embedding encodes the input mask; When no mask is input, a default embedding is generated.

3. The craniopharynx tumor magnetic resonance image segmentation system based on the basic large model according to claim 1 is characterized by: The 3D mask decoder uses a Transformer-based architecture to process 3D image embedding and hint features through a multi-layer interactive attention mechanism; that is, the contextual relationship between the image and the hint is extracted through the Transformer, and the output of the Transformer is converted into the final mask through deconvolution and multi-layer perceptron.

4. A method for craniopharynx duct tumor MRI image segmentation based on a basic large model, comprising: The method comprises the following steps: Step 1: Loading a pre-trained basic large model based on the obtained basic large model weight parameters; the basic large model includes a 3D image encoder, a 3D hint encoder, and a 3D mask decoder; Step 2: Loading and preprocessing 3D image data; Step 3: Use the pre-trained basic model and the simulated click points generated by the interactive click generation module to perform multiple rounds of iterative segmentation reasoning on the image features to generate the final segmentation result. Step 3 specifically includes the following steps: Step 3-1, using a 3D image encoder to convert the input 3D image into image features represented by high-dimensional features; Step 3-2: Determine whether the first segmentation reasoning is performed; if so, use the interactive click generation module to generate the initial click point; otherwise, the interactive click generation module generates a new click point based on the segmentation results of the previous round; Step 3-3, combining image features and click point hints to predict the segmentation results after simulated click correction, and generating a corrected low-resolution segmentation mask; Step 3-4: Encode the click point hints and low-resolution segmentation masks generated by the simulated user input as hint features, and map the hint features to the same high-dimensional space as the three-dimensional image features; Steps 3-5: Extract the contextual relationship between image features and hint features to generate a segmentation mask, and predict the mask quality score to form the segmentation result of the corresponding round; convert the high-dimensional features back to the segmentation result in three-dimensional space and mark the target area in the image; Step 3-6: Repeat steps 3-2 to 3-5 until the specified number of iterations is reached, and output the segmentation result of the last round as the final segmentation result; Step 4: resize the segmentation mask of the final segmentation result to the size of the original 3D image, and crop, fill, or restore the mask size to ensure that the segmentation result is aligned with the original image; Step 5: Save the generated segmentation mask as a file in the specified format, and record the click points and segmentation results.

5. The method for craniopharynx duct tumor magnetic resonance image segmentation based on a basic large model according to claim 4, characterized in that: In step 3, a sliding window strategy is used to divide the 3D image into blocks, and segmentation reasoning is performed on each window to obtain the segmentation result.

6. The method for craniopharynx duct tumor magnetic resonance image segmentation based on a basic large model according to claim 4, characterized in that: In step 3-2, in the first round of segmentation reasoning, initial click points are generated based on the specified features of the 3D image; The specified features are brightness threshold and image structure. Starting from the second round, based on the segmentation results of the previous round, the interactive click generation module automatically generates new click points in the misclassified areas of the foreground and background by comparing the segmentation results of the previous round with the original three-dimensional image.

7. The method for craniopharynx duct tumor magnetic resonance image segmentation based on a basic large model according to claim 4, characterized in that: In step 3-2, new click points are generated in the misclassified areas of the foreground and background, which specifically include the following steps: Step 3-2-1, generate a pseudo-negative mask: perform threshold processing on the input 3D image to generate a potential foreground region, and perform a binarization operation on the previous round of segmentation results to generate a foreground mask predicted by the model; by comparing the potential foreground region with the foreground mask, identify the potential error region in the segmentation result that is mistakenly considered to be the foreground as a pseudo-negative region; Step 3-2-2, click point generation: Generate click points for correcting segmentation results based on the pseudo-negative mask; Step 3-2-3, output the three-dimensional coordinates of the click point and the corresponding label.

8. The method for craniopharynx duct tumor magnetic resonance image segmentation based on a basic large model according to claim 7, characterized in that: After the click point is generated in step 3-2-2, the three-dimensional coordinates (d, h, w) of the click point are generated through logical operations, representing the position in the depth, width and height directions respectively; when the false negative area does not exist, a simplified prompt information generation strategy is used to generate the click point, and prompt information is generated by randomly selecting a point outside the segmented area as the click point.

9. The method for craniopharynx duct tumor magnetic resonance image segmentation based on a basic large model according to claim 7, characterized in that: The three-dimensional coordinates of the click point in step 3-2-3 are used to identify the region of interest or segment the uncertain region; the corresponding label is used to indicate whether the pixel point is foreground or background; foreground means that the pixel point belongs to the region of interest, while background means that the pixel point does not belong to the foreground.

10. The method for craniopharynx duct tumor magnetic resonance image segmentation based on a basic large model according to claim 4, characterized in that: Step 3-3 specifically includes the following steps: Step 3-3-1, after preprocessing the 3D image, perform image tensorization to generate an image in PyTorch format; Step 3-3-2: Call the 3D image encoder to generate the feature embedding of the image in PyTorch format Step 3-3-3, save the original size and input size of the 3D image for subsequent mask upsampling and restoration to the original size; Step 3-3-4, perform mask prediction based on the click points and low-resolution mask hint information generated by the current round of simulation; that is, convert the generated click points into sparse and dense embedding representations through the 3D hint encoder; use the 3D mask decoder to generate low-resolution masks and IoU prediction values; upsample the low-resolution masks to the original image size; Step 3-3-5, obtain the embedded representation of the currently set image, combine global information for accurate segmentation, and support subsequent tasks or multiple interactive adjustments; the shape of the embedded representation is 1xCxHxW, where C is the embedding dimension, and H and W are the spatial dimensions of the embedding.

Images