An automated segmentation and IDH genotyping method for multi-task gliomas based on SAM

Abstract

This application provides a multi-task automatic glioma segmentation and IDH genotyping method based on SAM, achieving synergistic optimization of segmentation and genotyping tasks. The method includes: acquiring a multimodal MRI image dataset of gliomas; preprocessing the dataset to obtain a preprocessed standardized dataset, and dividing the standardized dataset into a training set and a test set according to a preset ratio; establishing a SAM-based multi-task automatic glioma segmentation and IDH genotyping model, the model including: a four-branch image encoder, a feature fusion module, a cue encoder, a mask decoder, and an IDH classifier; inputting the training set into the model for training, setting a joint loss function, and optimizing the model parameters through the joint loss function to obtain an optimized model; inputting the test set into the optimized model for segmentation, IDH genotyping, and prediction, obtaining the tumor segmentation accuracy and IDH genotyping accuracy of the optimized model.

Description

Technical Field

[0001] This application relates to the field of digital image processing, specifically to a method, system, device, and storage medium for automatic segmentation of gliomas and IDH genotyping based on SAM (Self-Aided Aesthetic Model). Background Technology

[0002] Gliomas are the most common primary tumors of the central nervous system. Pathologically, they are classified into two main categories according to the 2016 WHO criteria: low-grade (LGG, grades II and III) and high-grade (HGG, grade IV). Studies have found that the mutation status of the IDH gene plays a crucial role in assessing glioma prognosis, especially in low-grade gliomas, where IDH mutations often predict more favorable clinical outcomes. However, current glioma segmentation primarily relies on manual annotation by experienced neuroradiologists, a process that is not only subjective but also time-consuming and labor-intensive.

[0003] MRI, as a non-invasive imaging technique, plays a crucial role in the diagnosis and treatment planning of gliomas. Numerous studies have shown that quantitative features from MRI can reflect the histological and genetic information of gliomas; therefore, radiomics or radiogenomics methods are widely used for tumor grading and genotyping. However, these methods typically require a series of complex processes, including image acquisition, preprocessing, tumor segmentation, feature extraction, feature screening, and classification modeling, and the extracted image features have limitations in reproducibility and robustness.

[0004] In recent years, deep learning-based automated segmentation methods have made significant progress in the field of medical imaging, especially tumor segmentation techniques implemented using convolutional neural networks (such as U-Net, V-Net, and DMFNet), which have provided effective solutions for glioma segmentation. However, these methods are mostly single-task models (unlike the multi-task modality of this application—precise tumor segmentation and genotyping), primarily focusing on the segmentation task and failing to fully consider the intrinsic correlation between segmentation results and IDH genotyping. Furthermore, these traditional segmentation networks still lack versatility in adapting to different lesion types and multimodal data, making it difficult to meet actual clinical needs.

[0005] Based on this, the present invention proposes a multi-task automatic segmentation and IDH genotyping method for gliomas based on SAM, aiming to achieve synergistic optimization of automatic segmentation and IDH genotyping of gliomas. Summary of the Invention

[0006] This application provides a multi-task automatic segmentation and IDH genotyping method for gliomas based on SAM, which achieves synergistic optimization of segmentation and genotyping tasks and improves the efficiency of accurate diagnosis and treatment planning for gliomas.

[0007] To achieve the above objectives, this application adopts the following technical solution:

[0008] In a first aspect, this application provides a method for automated segmentation of gliomas and genotyping of IDH based on SAM (Self-Action-Assisted Aneurysm Injection), the method comprising:

[0009] Obtain a multimodal MRI image dataset of gliomas, wherein the dataset includes multiple samples, and each sample includes four modal images;

[0010] The dataset is preprocessed to obtain a preprocessed standardized dataset, and then the standardized dataset is divided into a training set and a test set according to a preset ratio.

[0011] A model for automatic segmentation of gliomas and genotyping of IDH based on SAM was established. The model includes: a four-branch image encoder, a feature fusion module, a cue encoder, a mask decoder, and an IDH classifier.

[0012] The four-branch image encoder is used to generate features for four modal images of each sample;

[0013] The feature fusion module is used to fuse the features of the four modal images of each sample to generate an image embedding;

[0014] The cue encoder includes a point cue encoder and a mask cue encoder. The point cue encoder encodes the spatial location of the input point cue and maps the cue to a 384-dimensional feature space through a point embedding layer to generate a point cue embedding. The mask cue encoder reduces the dimensionality of the input mask cue to 384-dimensional features through two mask downsampling modules using 3D convolution, LayerNorm3d, and GELU activation functions to generate a mask cue embedding. The cue encoder then generates a cue embedding based on the point cue embedding and the mask cue embedding. The input point cue is determined by the model based on any point in the sample when processing each sample, and the input mask cue is determined based on the mask output generated by the mask decoder.

[0015] A mask decoder is used to generate a mask output based on image embedding and cue embedding; an IDH classifier is used to generate gene classification results based on image embedding and mask output.

[0016] The training set is input into the model for training, and a joint loss function is set. The parameters of the model are optimized through the joint loss function to obtain the optimized model.

[0017] The test set is input into the optimized model for segmentation, IDH typing, and prediction to obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

[0018] Secondly, this application provides a multi-task automated glioma segmentation and IDH genotyping system based on SAM, the system comprising:

[0019] The acquisition module is used to acquire a multimodal MRI image dataset of gliomas, wherein the dataset includes multiple samples, and each sample includes four modal images;

[0020] The processing module is used to preprocess the dataset to obtain a preprocessed standardized dataset, and divide the standardized dataset into a training set and a test set according to a preset ratio.

[0021] The model building module is used to establish a multi-task automatic glioma segmentation and IDH genotyping model based on SAM. The model includes: a four-branch image encoder, a feature fusion module, a cue encoder, a mask decoder, and an IDH classifier. The four-branch image encoder generates features from four modalities for each sample. The feature fusion module fuses the features from the four modalities for each sample to generate an image embedding. The cue encoder includes a point cue encoder and a mask cue encoder. The point cue encoder encodes the spatial location of the input point cue and maps the cue to a 384-dimensional feature space through a point embedding layer to generate point cue. The embedding and masking cue encoders process the input masking cue through two masking downsampling modules, using 3D convolution, LayerNorm3d, and GELU activation functions to reduce the dimensionality to 384 dimensions, generating masking cue embeddings. The cue encoder then generates cue embeddings based on the point cue embeddings and masking cue embeddings. The input point cue is determined by the model processing each sample based on any point within the sample, while the input masking cue is determined based on the masking output generated by the masking decoder. The masking decoder generates masking outputs based on the image embeddings and cue embeddings. The IDH classifier generates gene classification results based on the image embeddings and masking outputs.

[0022] The model optimization module is used to input the training set into the model for training, set the joint loss function, optimize the model parameters through the joint loss function, and obtain the optimized model.

[0023] The evaluation module is used to input the test set into the optimized model for segmentation, IDH typing, and prediction, and to obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

[0024] Thirdly, a SAM-based multi-task automatic glioma segmentation and IDH genotyping device is provided, which includes a module for performing the method described in the first aspect.

[0025] In one possible design, the SAM-based multi-task glioma automatic segmentation and IDH genotyping device of the third aspect may further include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the SAM-based multi-task glioma automatic segmentation and IDH genotyping device of the third aspect and other devices.

[0026] In one possible design, the SAM-based multi-task glioma auto-segmentation and IDH genotyping device of the third aspect may further include a memory. This memory may be integrated with the processor or disposed separately. The memory may be used to store the instructions involved in the method of the first aspect.

[0027] Fourthly, a SAM-based multi-task glioma automatic segmentation and IDH genotyping device is provided. This SAM-based multi-task glioma automatic segmentation and IDH genotyping device includes: a processor coupled to a memory, the processor executing instructions stored in the memory to cause the SAM-based multi-task glioma automatic segmentation and IDH genotyping device to perform the method of the first aspect.

[0028] In one possible design, the SAM-based multi-task glioma automatic segmentation and IDH genotyping device of the fourth aspect may further include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the SAM-based multi-task glioma automatic segmentation and IDH genotyping device of the fourth aspect and other devices.

[0029] Fifthly, a SAM-based multi-task glioma automatic segmentation and IDH genotyping device is provided, comprising: a processor and a memory; the memory is used to store instructions, which, when executed by the processor, cause the SAM-based multi-task glioma automatic segmentation and IDH genotyping device to perform the method of the first aspect.

[0030] In one possible design, the SAM-based multi-task glioma automatic segmentation and IDH genotyping device of the fifth aspect may further include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used to communicate between the SAM-based multi-task glioma automatic segmentation and IDH genotyping device of the fifth aspect and other devices.

[0031] In a sixth aspect, a computer-readable storage medium is provided, the computer-readable storage medium including storage of a computer program or instructions, which, when executed, cause the SAM-based multi-task glioma automatic segmentation and IDH genotyping method of the first aspect to be executed.

[0032] In this embodiment, the SAM-based multi-task automatic glioma segmentation and IDH genotyping method provided by this invention employs a four-branch input architecture. For T1, T1ce, Flair, and T2 MRI image data, image embeddings are independently extracted using a shared-weight image encoder. Subsequently, a feature fusion module integrates the embedding features from each modality into a unified multimodal fusion feature. This fusion feature, along with the cue embedding generated by the cue encoder, is input to a mask decoder to generate a refined tumor segmentation mask. Simultaneously, an IDH classifier performs binary classification prediction of IDH genostatus. The model undergoes end-to-end joint training, adaptively balancing segmentation and genotyping losses based on a learnable log-variance parameter joint loss function, effectively improving the model's stability and robustness.

[0033] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating the SAM-based multi-task glioma automatic segmentation and IDH genotyping method provided in this application embodiment;

[0036] Figure 2 A schematic diagram of the structure of the SAM-based multi-task glioma automatic segmentation and IDH genotyping model provided in the embodiments of this application;

[0037] Figure 3 Schematic diagram of the structure of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device provided in the embodiments of this application. Figure 1 ;

[0038] Figure 4 Schematic diagram of the structure of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device provided in the embodiments of this application. Figure 2 . Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. At the same time, in the description of the embodiments of this application, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0040] Figure 1 This is a flowchart illustrating the SAM-based multi-task glioma automatic segmentation and IDH genotyping method provided in this application embodiment.

[0041] The workflow of this SAM-based multi-task automated glioma segmentation and IDH genotyping method is as follows:

[0042] Step S101: Obtain a multimodal MRI image dataset of glioma, wherein the dataset includes multiple samples, and each sample includes four modal images.

[0043] The dataset can be represented as: Where, N L x represents the total number of samples. (i) The multimodal MRI images represent the i-th sample, where each sample includes four multimodal MRI images: FLAIR modality, T1 modality, T1ce modality, and T2 modality, g (i) y represents the ground truth label of the tumor region in the i-th sample. (i) ∈{0,1} represents the IDH gene mutation status of the i-th sample, where y (i) =0 represents the wild type of IDH, y (i) =1 represents the IDH mutant type.

[0044] The dataset in this application can be obtained from historical TCIA image archives and TCGA gene archives, or from certain medical institutions, without any specific restrictions.

[0045] Step S102: Preprocess the dataset to obtain a preprocessed standardized dataset, and divide the standardized dataset into a training set and a test set according to a preset ratio.

[0046] For dataset D L The samples in the dataset are normalized at the voxel level. The normalization formula is as follows:

[0047]

[0048] Where U(x) is the set of foreground voxels for each sample in the dataset, and U(μ) and U(σ) are the mean and standard deviation of the foreground voxels in the dataset, respectively. Norm (x) is the set of foreground voxels for each sample in the standardized dataset.

[0049] Furthermore, the Z-score normalization method described above is only one example. Linear normalization (Min-Max normalization) can also be used to map voxel values ​​to a specified interval, usually [0,1] or [-1,1], depending on the actual situation, and no restrictions are imposed here.

[0050] It should also be noted that the standardized dataset is divided into training and testing sets according to a preset ratio. This preset ratio can be 8:2, 7:3, or other ratios, depending on the total sample size or actual circumstances; no specific restrictions are imposed here.

[0051] Step S103: Establish a multi-task automatic segmentation model for gliomas and IDH genotyping based on SAM. The model includes: a four-branch image encoder, a feature fusion module, a cue encoder, a mask decoder, and an IDH classifier.

[0052] For details, please refer to... Figure 2 Understanding is key; the model will now be explained in detail:

[0053] (1) Four-branch image encoder

[0054] The four-branch image encoder is used to generate features for each sample's four modal images. Specifically:

[0055] The four branch image encoders have the same structure and share weights. Each branch image encoder processes the four modal images of each sample and generates the features of the four modal images of each sample.

[0056] Each branch image encoder includes an image embedding module, a position encoding module, and an attention mechanism module;

[0057] The image embedding module uses a 3D convolutional layer with a kernel size of 16×16×16 to convert the four modal images of each sample into 384-channel features;

[0058] The location encoding module introduces 3D spatial location information into the 384-channel features;

[0059] The attention mechanism module contains multiple Transformer self-attention mechanism blocks. It uses query, key, and value to perform linear transformation on the 384-channel features and calculates the global context dependency to generate features for each sample's four modal images that fuse local and global semantic information.

[0060] It's worth noting that employing a four-branch image encoder, with all four branches using the same structure and shared weights, significantly reduces the number of model parameters, thereby lowering computational overhead and storage requirements, and improving model efficiency. Furthermore, each branch specifically processes an image of a particular modality, ensuring independent encoding of features for each modality. This also allows for flexible handling of differences between images of different modalities, enhancing the model's adaptability to diverse modal information. Moreover, the identical encoder structure makes model training more efficient, avoiding redundant learning tasks, promoting rapid model convergence, and contributing to improved generalization ability.

[0061] (2) Feature fusion module

[0062] The feature fusion module is used to fuse the features of the four modal images of each sample to generate an image embedding.

[0063] (3) Prompt encoder

[0064] The cue encoder is used to generate cue embeddings. Specifically:

[0065] Cue encoders include dot cue encoders and mask cue encoders;

[0066] The point hint encoder encodes the spatial location of the input point hints and maps the hints to a 384-dimensional feature space through a point embedding layer to generate point hint embeddings. The input point hints are determined by the model based on any point in the sample when processing each sample.

[0067] The mask hint encoder reduces the dimensionality of the input mask hint to 384-dimensional features through two mask downsampling modules using 3D convolution, LayerNorm3d, and GELU activation functions to generate a mask hint embedding. The input mask hint is determined based on the mask output generated by the mask decoder.

[0068] The cue encoder generates cue embeddings based on dot cue embeddings and mask cue embeddings.

[0069] (4) Mask Decoder

[0070] The mask decoder is used to generate masked output based on image embedding and cue embedding.

[0071] Specifically as follows:

[0072] The mask decoder includes a bidirectional interaction module, an upsampling module, and a multilayer sensing module;

[0073] The bidirectional interaction module, built on the Transformer structure, consists of multiple bidirectional attention blocks. Each attention block simultaneously employs self-attention and cross-attention mechanisms to perform bidirectional information interaction and global context modeling for image embedding and cue embedding. Furthermore, it optimizes feature representation through LayerNorm and MLP.

[0074] The upsampling module employs multiple deconvolutional layers, LayerNorm3d, and GELU to progressively improve the spatial resolution of low-resolution fused features.

[0075] The multilayer sensing module dynamically generates weights for each category through the network and is equipped with an IoU prediction head to generate a mask output.

[0076] (5) IDH classifier

[0077] The IDH classifier is used to generate gene classification results based on image embedding and mask output. Specifically:

[0078] The IDH classifier performs 3D average pooling and max pooling on the image embedding and mask output, respectively. The pooled features are concatenated and then input into a fully connected layer to generate gene classification results, which include IDH wild type and IDH mutant type.

[0079] Step S104: Input the training set into the model for training, set the joint loss function, optimize the model parameters through the joint loss function, and obtain the optimized model.

[0080] The training set is input into the model for training. The following example illustrates the model's processing by training any sample from the training set:

[0081] Step 1: Sample R is processed in four branch image encoders to process four multimodal MRI images (FLAIR modality, T1 modality, T1ce modality, T2 modality) respectively, to obtain the features of the four modal images of the generated sample R;

[0082] Step 2: The feature fusion module fuses the features of the four modal images of sample R to generate an image embedding;

[0083] Step 3: The prompt encoder obtains the first prompt based on any point in the sample R, and then obtains the first prompt embedding based on the first prompt.

[0084] Step 4: The mask decoder generates a first mask output based on the image embedding and the first cue embedding, and converts it into the first mask cue corresponding to the first mask output;

[0085] Step 5: The cue encoder obtains a second cue based on the first mask cue and any other point in sample R, and generates the second cue embedding;

[0086] Step 6: The mask decoder generates a second mask output based on the image embedding and the second cue embedding, and converts it into the second mask cue corresponding to the second mask output;

[0087] It should be noted that steps five and six can be executed multiple times, depending on the hyperparameters set manually. In this example, the second mask output is used as the final mask output to segment the sample R.

[0088] Step 7: The IDH classifier generates gene classification results based on the second mask output and image embedding.

[0089] It should also be noted that before generating the gene classification results based on the second mask output and image embedding in step seven, the second mask output and image embedding need to be normalized to make them on the same dimension.

[0090] Then, based on the segmentation results and gene classification results of sample R, the model parameters are optimized using a joint loss function. The next sample is then processed, and the corresponding joint loss function value is obtained. The model parameters are then optimized again until all samples in the training set have been processed.

[0091] It should be noted that before inputting the training set into the model for training, data augmentation processing can be performed on the training set, including operations such as random rotation, random noise addition, random cropping, and random flipping. After that, the data-augmented training set is input into the model for training, and the specific training method is the same as described above, which will not be elaborated here.

[0092] Furthermore, the aforementioned joint loss function includes the segmentation loss function L. seg and classification loss function L idh ;

[0093] Segmentation loss function L seg for:

[0094]

[0095] Where, λ dice and λ ce These represent the weight coefficients of the Dice loss function and the cross-entropy loss function of the model, respectively. and Let represent the Dice loss function and cross-entropy loss function for the i-th sample, and M represent the total number of samples in the training set.

[0096] The Dice loss function is:

[0097]

[0098] The cross-entropy loss function is:

[0099]

[0100] in, This represents the predicted label of the nth voxel in the i-th sample, as predicted by the model. Let ξ represent the true segmentation label of the nth voxel in the i-th sample. The dataset contains the true label of each sample. ξ and ζ are smoothing constants added to the numerator and denominator, respectively, to prevent division by zero errors and stabilize training. N represents the number of voxels in the i-th sample.

[0101] Classification loss function L idh for:

[0102]

[0103] Where M represents the total number of samples in the training set, y i ∈{0,1} represents the true IDH mutation label of the i-th sample, y i A value of "0" indicates that the i-th sample is wild-type, y i A value of "1" indicates that the i-th sample is a mutant, p i p represents the probability that the model predicts the i-th sample to be a mutant. i The value range is (0,1).

[0104] The joint loss function is:

[0105]

[0106] Where, var seg and var idh For the learnable log-variance parameter corresponding to segmentation and fractal tasks, var seg and var idh It is a tensor initialized to 0, which is then updated adaptively through iterative steps during the training phase.

[0107] L Combine By applying exponentially decaying weights to the loss of each task and adding corresponding uncertainty terms, the goal of dynamically balancing the losses of different tasks during training is achieved, thereby improving the overall performance and robustness of the model.

[0108] It is understandable that by introducing a learnable log-variance parameter, the segmentation loss and IDH genotyping loss can be adaptively weighted to achieve a dynamic balance between the two tasks. This design promotes complementary learning between the segmentation and genotyping tasks, effectively improving the accuracy of glioma segmentation and IDH genotyping, and providing a solid technical guarantee for overall diagnostic performance.

[0109] Step S105: Input the test set into the optimized model for segmentation, IDH typing, and prediction to obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

[0110] For example, this embodiment uses the BraTS2020 public dataset, which contains 148 patient samples. 118 of these samples were used to train the model, and 30 samples were used as an independent test set for model evaluation. The Dice score was used to evaluate the model's segmentation performance across the entire tumor region. The Dice score ranges from 0 to 1; a higher Dice score indicates better segmentation. The IDH classification accuracy was used to evaluate the model's IDH genotyping accuracy. The IDH classification accuracy ranges from 0 to 1; a higher IDH classification accuracy indicates better IDH genotyping. The results are shown in Table 1.

[0111] Experimental results Dice Score IDH classification accuracy Training set results 0.97 0.98 Test set results 0.96 0.85

[0112] Table 1

[0113] In summary, the SAM-based multi-task automatic glioma segmentation and IDH genotyping method provided in this invention employs a four-branch input architecture. For T1, T1ce, Flair, and T2 MRI image data, image embeddings are independently extracted using a shared-weight image encoder. Subsequently, a feature fusion module integrates the embedding features from each modality into a unified multimodal fusion feature. This fusion feature, along with the cue embedding generated by the cue encoder, is input to a mask decoder to generate a refined tumor segmentation mask. Simultaneously, an IDH classifier performs binary classification prediction of IDH genostatus. The model undergoes end-to-end joint training, adaptively balancing segmentation and genotyping losses based on a learnable log-variance parameter joint loss function, effectively improving the model's stability and robustness.

[0114] The above combination Figures 1-2 This application provides a detailed description of the SAM-based automatic segmentation and IDH genotyping method and application for gliomas and genotyping based on multi-tasks, as provided in the embodiments of this application. The following details the system for implementing the SAM-based automatic segmentation and IDH genotyping system for gliomas and genotyping provided in the embodiments of this application.

[0115] The system specifically includes: an acquisition module, a processing module, a model building module, a model optimization module, and an evaluation module, as shown below.

[0116] The acquisition module is used to acquire a multimodal MRI image dataset of gliomas, wherein the dataset includes multiple samples, and each sample includes four modal images;

[0117] The processing module is used to preprocess the dataset to obtain a preprocessed standardized dataset, and divide the standardized dataset into a training set and a test set according to a preset ratio.

[0118] The model building module is used to establish a multi-task automatic glioma segmentation and IDH genotyping model based on SAM. The model includes: a four-branch image encoder, a feature fusion module, a cue encoder, a mask decoder, and an IDH classifier. The four-branch image encoder generates features from four modalities for each sample. The feature fusion module fuses the features from the four modalities for each sample to generate an image embedding. The cue encoder includes a point cue encoder and a mask cue encoder. The point cue encoder encodes the spatial location of the input point cue and maps the cue to a 384-dimensional feature space through a point embedding layer to generate point cue. The embedding and masking cue encoders process the input masking cue through two masking downsampling modules, using 3D convolution, LayerNorm3d, and GELU activation functions to reduce the dimensionality to 384 dimensions, generating masking cue embeddings. The cue encoder then generates cue embeddings based on the point cue embeddings and masking cue embeddings. The input point cue is determined by the model processing each sample based on any point within the sample, while the input masking cue is determined based on the masking output generated by the masking decoder. The masking decoder generates masking outputs based on the image embeddings and cue embeddings. The IDH classifier generates gene classification results based on the image embeddings and masking outputs.

[0119] The model optimization module is used to input the training set into the model for training, set the joint loss function, optimize the model parameters through the joint loss function, and obtain the optimized model.

[0120] The evaluation module is used to input the test set into the optimized model for segmentation, IDH typing, and prediction, and to obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

[0121] Furthermore, the specific implementation of the above system is basically similar to the method implementation, so the description is relatively simple. For relevant details, please refer to the description of the method implementation. Moreover, it should be noted that in the various modules of the system of this application, the components are logically divided according to the functions they are to perform. However, this application is not limited to this and can re-divide or combine the components as needed.

[0122] The above describes the SAM-based multi-task glioma automatic segmentation and IDH genotyping method and system provided in the embodiments of this application. The following, in conjunction with... Figures 3-4 This document provides a detailed description of the SAM-based multi-task automatic glioma segmentation and IDH genotyping device used to execute the embodiments provided in this application.

[0123] Figure 3 This is a schematic diagram of the structure of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device provided in the embodiments of this application. Figure 1 For example, such as Figure 3 As shown, the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 300 includes: a transceiver module 301 and a processing module 302. For ease of explanation, Figure 3 Only the main components of this SAM-based multi-task glioma automatic segmentation and IDH genotyping device are shown.

[0124] The transceiver module 301 is used to perform the transceiver function of the above-mentioned SAM-based multi-task glioma automatic segmentation and IDH genotyping method, and the processing module 302 is used to perform other functions of the above-mentioned SAM-based multi-task glioma automatic segmentation and IDH genotyping method besides the transceiver function.

[0125] Optionally, the transceiver module 301 may include a sending module ( Figure 3 (not shown in the image) and receiving module ( Figure 3 (Not shown in the image). The transmitting module is used to implement the transmitting function of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 300, and the receiving module is used to implement the receiving function of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 300.

[0126] Optionally, the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 300 may also include a storage module ( Figure 3(Not shown in the image), the storage module stores programs or instructions. When the processing module 302 executes the program or instructions, the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 300 can perform the SAM-based multi-task glioma automatic segmentation and IDH genotyping method in the embodiments of this application.

[0127] The following is combined with Figure 4 A detailed description of each component of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400 is provided below:

[0128] The processor 401 is the control center of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400. It can be a single processor or a collective term for multiple processing elements. For example, the processor 401 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0129] Optionally, the processor 401 can execute various functions of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400 by running or executing software programs stored in the memory 402 and calling data stored in the memory 402, such as executing the SAM-based multi-task glioma automatic segmentation and IDH genotyping method in the embodiments of this application.

[0130] In a specific implementation, as one example, processor 401 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0131] In a specific implementation, as one example, the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400 may also include multiple processors, for example... Figure 4The processors 401 and 404 are shown in the diagram. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). The memory 402 is used to store the software program executing the scheme of this application, and its execution is controlled by the processor 401. Specific implementation methods can be found in the above method embodiments, and will not be repeated here.

[0132] Optionally, the memory 402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 402 may be integrated with the processor 401 or exist independently, and may be connected via an interface circuit between the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400. Figure 4 (Not shown in the image) is coupled to processor 401, and this embodiment of the application does not specifically limit this.

[0133] Transceiver 403 is used for communication with other communication devices. For example, in the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400, which is the first device, transceiver 403 can be used to communicate with a second device or a third device.

[0134] Alternatively, transceiver 403 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0135] Optionally, the transceiver 403 can be integrated with the processor 401 or exist independently, and can be connected to the interface circuit of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400. Figure 4(Not shown in the image) is coupled to processor 401, and this embodiment of the application does not specifically limit this.

[0136] Understandable Figure 4 The structure of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400 shown in the figure does not constitute a limitation on the SAM-based multi-task glioma automatic segmentation and IDH genotyping device. The actual SAM-based multi-task glioma automatic segmentation and IDH genotyping device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0137] Furthermore, the technical effects of the SAM-based multi-task glioma automatic segmentation and IDH genotyping device 400 can be referred to the technical effects of the methods described in the above-described method embodiments, and will not be repeated here.

[0138] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0139] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0140] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

Claims

1. A SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method, characterized in that, The method comprises: acquiring a brain glioma multi-modal MRI image data set, wherein the data set comprises a plurality of samples, and each sample comprises four modal images; preprocessing the data set to obtain a preprocessed standardized data set, and dividing the standardized data set into a training set and a test set according to a preset ratio; establishing a SAM-based multi-task brain glioma automatic segmentation and IDH genotyping model, wherein the model comprises four branch image encoders, a feature fusion module, a prompt encoder, a mask decoder and an IDH classifier; the four branch image encoders are used to generate features of the four modal images of each sample; the feature fusion module is used to perform feature fusion on the features of the four modal images of each sample to generate image embeddings; the prompt encoder comprises a point prompt encoder and a mask prompt encoder, the point prompt encoder encodes the spatial position of the input point prompt, and maps the prompt to a 384-dimensional feature space through a point embedding layer to generate a point prompt embedding, the mask prompt encoder reduces the dimension of the input mask prompt to a 384-dimensional feature through a 3D convolution, a LayerNorm3d and a GELU activation function through two mask down-sampling modules to generate a mask prompt embedding, and the prompt encoder generates a prompt embedding based on the point prompt embedding and the mask prompt embedding, wherein the input point prompt is determined based on any point in the sample when the model processes each sample, and the input mask prompt is determined based on the mask output generated by the mask decoder; the mask decoder is used to generate a mask output based on the image embedding and the prompt embedding; the IDH classifier is used to generate a gene classification result based on the image embedding and the mask output; the training set is input into the model for training, a joint loss function is set, the parameters of the model are optimized through the joint loss function, and an optimized model is obtained; the test set is input into the optimized model for segmentation, IDH typing and prediction, and the tumor segmentation accuracy and IDH typing accuracy of the optimized model are obtained.

2. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that, The preprocessing of the data set to obtain a preprocessed standardized data set comprises: performing voxel-level normalization on the samples in the data set, and the normalization formula is: wherein U(x) is the set of foreground voxels of each sample in the dataset, U(μ) and U(σ) are the mean and standard deviation of foreground voxels of the dataset, respectively, and U Norm (x) is the set of foreground voxels of each sample in the normalized dataset.

3. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that, the four branch image encoders are used to generate features of the four modal images of each sample, comprising: the four branch image encoders have the same structure and share the same weight, and each branch image encoder processes the four modal images of each sample to generate features of the four modal images of each sample; each branch image encoder comprises an image embedding module, a position encoding module and an attention mechanism module; the image embedding module adopts a three-dimensional convolution layer with a convolution kernel size of 16x16x16 to convert the four modal images of each sample into 384-channel features; the position encoding module introduces 3D spatial position information into the 384-channel features; The attention mechanism module comprises a plurality of Transformer self-attention mechanism blocks, linearly transforms the 384-channel features by using query, key and value, and calculates global context dependency to generate features of four modality images of each sample fused with local and global semantic information.

4. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that, The mask decoder is configured to generate a mask output based on the image embedding and the prompt embedding, including: The mask decoder comprises a bidirectional interaction module, an upsampling module and a multi-layer perception module. The bidirectional interaction module is constructed based on a Transformer structure and comprises a plurality of bidirectional attention blocks, each of which adopts self-attention and cross-attention mechanisms to perform bidirectional information interaction and global context modeling on the image embedding and the prompt embedding, and further optimizes feature expression through LayerNorm and MLP. The upsampling module adopts a plurality of deconvolution layers, LayerNorm3d and GELU to gradually improve the spatial resolution of the low-resolution fused features. The multi-layer perception module dynamically generates class weights through the network and is equipped with an IoU prediction head to generate the mask output.

5. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that, The IDH classifier is configured to generate a gene classification result based on the image embedding and the mask output, including: The IDH classifier performs 3D average pooling and maximum pooling on the image embedding and the mask output respectively, concatenates the pooled features and inputs them into a fully connected layer to generate the gene classification result, wherein the gene classification result includes IDH wild type and IDH mutant type.

6. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that, The joint loss function comprises a segmentation loss function L seg and a classification loss function L idh ; The segmentation loss function L seg is: Wherein, the λ dice and the λ ce respectively represent the weight coefficients of the Dice loss function and the cross-entropy loss function of the model, the and the represent the Dice loss function and the cross-entropy loss function of the i-th sample, and the M represents the total number of samples in the training set. The Dice loss function is: The cross-entropy loss function is: wherein the represents a predicted label of the n-th voxel in the i-th sample predicted by the model, the represents a true segmentation label of the n-th voxel in the i-th sample, the data set containing the true label of each sample, the ξ and the η being smoothing constants added to the numerator and the denominator, respectively, for preventing division by zero error and stabilizing training, and the N representing the number of voxels of the i-th sample; The classification loss function L idh is: Where M represents the total number of samples in the training set, and y i ∈{0,1} represents the true IDH mutation label of the i-th sample, where y i A value of "0" indicates that the i-th sample is wild-type, and the y i A "1" indicates that the i-th sample is a mutant, and p i p represents the probability that the model predicts the i-th sample to be a mutant. i The range of values ​​for is (0,1); The joint loss function is: wherein the var seg and the var idh are learnable log-variance parameters corresponding to the splitting and typing tasks, respectively, and the var seg and the var idh are tensors initialized to 0 and updated adaptively through iterations during the training phase.

7. A SAM-based multi-task brain glioma automatic segmentation and IDH genotyping system, characterized in that, The system comprises: An acquisition module configured to acquire a brain glioma multi-modal MRI image dataset, wherein the dataset comprises a plurality of samples, and each sample comprises four modality images; A processing module configured to pre-process the dataset to obtain a pre-processed standardized dataset, and divide the standardized dataset into a training set and a test set according to a preset ratio. A model construction module is configured to establish a SAM-based multi-task brain glioma automatic segmentation and IDH genotyping model, wherein the model comprises four branch image encoders, a feature fusion module, a prompt encoder, a mask decoder, and an IDH classifier; the four branch image encoders are configured to generate features of four modality images of each sample; the feature fusion module is configured to perform feature fusion on the features of the four modality images of each sample to generate image embeddings; the prompt encoder comprises a point prompt encoder and a mask prompt encoder; the point prompt encoder encodes the spatial position of an input point prompt and maps the prompt to a 384-dimensional feature space through a point embedding layer to generate a point prompt embedding; the mask prompt encoder processes an input mask prompt through two mask down-sampling modules to reduce the dimension to 384-dimensional features using 3D convolution, LayerNorm3d, and a GELU activation function to generate a mask prompt embedding; the prompt encoder generates a prompt embedding based on the point prompt embedding and the mask prompt embedding; the input point prompt is determined based on any point in the sample when the model processes each sample, and the input mask prompt is determined based on a mask output generated by the mask decoder; the mask decoder is configured to generate a mask output based on the image embedding and the prompt embedding; and the IDH classifier is configured to generate a gene classification result based on the image embedding and the mask output. A model optimization module is configured to input the training set into the model for training, set a joint loss function, optimize the parameters of the model through the joint loss function, and obtain an optimized model. An evaluation module is configured to input the test set into the optimized model for segmentation, IDH typing, and prediction, and obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

8. A device for SAM-based automatic segmentation of multi-task brain glioma and IDH genotyping, characterized in that, The apparatus comprises modules for performing the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a computer program or instructions that, when executed, cause the method of any one of claims 1-6 to be performed.

Images