Multi-task brain glioma automatic segmentation and IDH genotyping method based on SAM

Abstract

The invention provides a multi-task brain glioma automatic segmentation and IDH genetic typing method based on SAM. Cooperative optimization of segmentation and typing tasks is achieved. The method comprises the following steps: acquiring a brain glioma multi-mode MRI image data set; preprocessing the data set to obtain a preprocessed standardized data set, and dividing the standardized data set into a training set and a test set according to a preset proportion; establishing a multi-task brain glioma automatic segmentation and IDH genotyping model based on SAM, wherein the model comprises a four-branch image encoder, a feature fusion module, a prompt encoder, a mask decoder and an IDH classifier; inputting the training set into the model for training, setting a joint loss function, and optimizing parameters of the model through the joint loss function to obtain an optimized model; and inputting the test set into the optimized model for segmentation and IDH typing and prediction to obtain the tumor segmentation precision and IDH typing accuracy of the optimized model.

Description

Technical Field

[0001] The present application relates to the field of digital image processing, and in particular to a SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method, system, device and storage medium. Background Art

[0002] Gliomas are the most common primary tumors of the central nervous system. Pathologically classified according to the 2016 WHO criteria, they can be divided into two categories: low-grade (LGG, grades II and III) and high-grade (HGG, grade IV). Studies have found that the mutation status of the IDH gene plays a key role in assessing glioma prognosis. In particular, in low-grade gliomas, IDH mutations often predict better clinical outcomes. However, current glioma segmentation relies primarily on manual annotation by experienced neuroradiologists, a process that is subjective, time-consuming, and labor-intensive.

[0003] As a non-invasive imaging technology, MRI plays a vital role in the diagnosis and treatment planning of brain gliomas. Numerous studies have shown that quantitative MRI features can reflect the histological and genetic information of gliomas. Therefore, imaging genomics or imaging genomics methods are widely used to determine tumor grade and genotype. However, these methods usually require a series of complex processes such as image acquisition, preprocessing, tumor segmentation, feature extraction, feature screening, and classification modeling. In addition, the extracted imaging features have certain limitations in reproducibility and robustness.

[0004] In recent years, deep learning-based automated segmentation methods have made significant progress in the field of medical imaging. In particular, tumor segmentation techniques implemented using convolutional neural networks (such as U-Net, V-Net, and DMFNet) have provided effective solutions for glioma segmentation. However, these methods are mostly single-task models (as opposed to the multi-task modality of this application—precise tumor segmentation and genotyping)—and primarily focus on the segmentation task, failing to fully consider the inherent correlation between segmentation results and IDH genotyping. Furthermore, these traditional segmentation networks lack versatility in adapting to diverse lesion types and multimodal data, making them difficult to meet actual clinical needs.

[0005] Based on this, the present invention proposes a multi-task SAM-based automatic glioma segmentation and IDH genotyping method, aiming to achieve the coordinated optimization of automatic glioma segmentation and IDH genotyping. Summary of the Invention

[0006] The embodiment of the present application provides a multi-task SAM-based automatic glioma segmentation and IDH genotyping method to achieve collaborative optimization of segmentation and typing tasks, and improve the efficiency of accurate diagnosis and treatment planning of gliomas.

[0007] To achieve the above objectives, this application adopts the following technical solutions: In a first aspect, the present application provides a multi-task SAM-based method for automatic brain glioma segmentation and IDH genotyping, the method comprising: Acquire a multimodal MRI image dataset of brain glioma, wherein the dataset includes multiple samples, and each sample includes four modality images; Preprocess the data set to obtain a preprocessed standardized data set, and divide the standardized data set into a training set and a test set according to a preset ratio; A SAM-based multi-task model for automatic glioma segmentation and IDH genotyping was established. The model includes: a four-branch image encoder, a feature fusion module, a hint encoder, a mask decoder, and an IDH classifier. The four-branch image encoder is used to generate features of the four modal images of each sample; the feature fusion module is used to fuse the features of the four modal images of each sample to generate image embedding; the prompt encoder is used to generate prompt embedding; the mask decoder is used to generate mask output based on the image embedding and prompt embedding; the IDH classifier is used to generate gene classification results based on the image embedding and mask output; Input the training set into the model for training, set the joint loss function, optimize the model parameters through the joint loss function, and obtain the optimized model; The test set was input into the optimized model for segmentation, IDH typing and prediction to obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

[0008] In a second aspect, the present application provides a multi-task brain glioma automatic segmentation and IDH genotyping system based on SAM, the system comprising: An acquisition module is used to acquire a multimodal MRI image dataset of a glioma, wherein the dataset includes multiple samples, and each sample includes four modal images; The processing module is used to preprocess the data set to obtain a preprocessed standardized data set, and divide the standardized data set into a training set and a test set according to a preset ratio; A model building module is used to establish a SAM-based multi-task model for automatic glioma segmentation and IDH genotyping, wherein the model includes: a four-branch image encoder, a feature fusion module, a prompt encoder, a mask decoder, and an IDH classifier; the four-branch image encoder is used to generate features of the four modal images of each sample; the feature fusion module is used to fuse the features of the four modal images of each sample to generate an image embedding; the prompt encoder is used to generate a prompt embedding; the mask decoder is used to generate a mask output based on the image embedding and the prompt embedding; the IDH classifier is used to generate a gene classification result based on the image embedding and the mask output; The model optimization module is used to input the training set into the model for training, set the joint loss function, optimize the model parameters through the joint loss function, and obtain the optimized model; The evaluation module is used to input the test set into the optimized model for segmentation, IDH typing and prediction, and obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

[0009] In a third aspect, a SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device is provided, wherein the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device includes a module for executing the method of the first aspect.

[0010] In one possible design, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping apparatus of the third aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping apparatus of the third aspect to communicate with other devices.

[0011] In one possible design, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping apparatus of the third aspect may further include a memory. The memory may be integrated with the processor or provided separately. The memory may be used to store instructions involved in the method of the first aspect.

[0012] In a fourth aspect, a SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device is provided. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device includes a processor coupled to a memory, the processor being configured to execute instructions stored in the memory to cause the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device to perform the method of the first aspect.

[0013] In one possible design, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping apparatus of the fourth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping apparatus of the fourth aspect to communicate with other devices.

[0014] In the fifth aspect, a SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device is provided, comprising: a processor and a memory; the memory is used to store instructions, and when the processor executes the instructions, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device performs the method of the first aspect.

[0015] In one possible design, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping apparatus of the fifth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping apparatus of the fifth aspect to communicate with other devices.

[0016] In a sixth aspect, a computer-readable storage medium is provided, which includes a computer program or instruction stored therein. When the computer program or instruction is executed, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method of the first aspect is executed.

[0017] In an embodiment of the present application, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method provided by the present invention adopts a four-branch input architecture. For the four modal MRI image data of T1, T1ce, Flair and T2, the image embedding is independently extracted by the image encoder with shared weights; then, the embedding features of each modality are integrated into a unified multimodal fusion feature using a feature fusion module. The fusion feature and the hint embedding generated by the hint encoder are input to the mask decoder together to generate a fine tumor segmentation mask, and at the same time, the binary classification prediction of the IDH gene status is realized through the IDH classifier. The model is trained end-to-end jointly, based on a joint loss function with a learnable logarithmic variance parameter, to adaptively balance the segmentation and typing losses, effectively improving the stability and robustness of the model.

[0018] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of the process of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method provided in the embodiments of the present application; Figure 2 Schematic diagram of the structure of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping model provided in the embodiments of the present application; Figure 3 Schematic diagram of the structure of the multi-task brain glioma automatic segmentation and IDH genotyping device based on SAM provided in the embodiment of the present application Figure 1 ; Figure 4 Schematic diagram of the structure of the multi-task brain glioma automatic segmentation and IDH genotyping device based on SAM provided in the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0022] Figure 1 Schematic diagram of the process of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method provided in the embodiment of the present application.

[0023] The workflow of this SAM-based multi-task glioma automatic segmentation and IDH genotyping method is as follows: Step S101: Acquire a multimodal MRI image dataset of glioma, wherein the dataset includes multiple samples, and each sample includes four modality images.

[0024] Among them, the dataset can represent: ,in, represents the total number of samples, Representative Multimodal MRI images of samples, where each sample includes four multimodal MRI images, namely FLAIR mode, T1 mode, T1ce mode, and T2 mode. Representative The ground truth of the tumor region in each sample. Representative IDH gene mutation status of samples, among which Represents IDH wild type, Represents IDH mutant type.

[0025] Among them, the datasets in this application can be obtained through the historical TCIA image archive and TCGA gene archive, or certain medical institutions, without specific restrictions.

[0026] Step S102 , preprocessing the data set to obtain a preprocessed standardized data set, and dividing the standardized data set into a training set and a test set according to a preset ratio.

[0027] For the dataset The samples in are normalized at the voxel level, and the normalization formula is:

[0028] in, is the set of foreground voxels for each sample in the dataset, and are the mean and standard deviation of the foreground voxels in the dataset, is the set of foreground voxels for each sample in the normalized dataset.

[0029] In addition, the above Z-score normalization method is only an example. Linear normalization (Min-Max normalization) can also be used to map voxel values ​​to a specified interval, usually [0, 1] or [-1, 1]. The specific method depends on the actual situation and is not limited here.

[0030] It should also be noted that the standardized dataset is divided into a training set and a test set according to a preset ratio. The preset ratio can be 8:2, 7:3, or other ratios, which are determined based on the total sample size or actual situation and are not specifically limited here.

[0031] Step S103 , establishing a SAM-based multi-task model for automatic glioma segmentation and IDH genotyping, wherein the model includes: a four-branch image encoder, a feature fusion module, a hint encoder, a mask decoder, and an IDH classifier.

[0032] For details, please refer to Figure 2 To understand, the following is a detailed introduction to the model: (1) Four-branch image encoder The four-branch image encoder is used to generate the features of the four modal images for each sample. The details are as follows: The four branch image encoders have the same structure and shared weights. Each branch image encoder processes the four modal images of each sample separately and generates features of the four modal images of each sample; Each branch image encoder includes an image embedding module, a position encoding module, and an attention mechanism module; The image embedding module uses a 3D convolutional layer with a convolution kernel size of 16×16×16 to convert the four modality images of each sample into 384-channel features; Position encoding module, which introduces 3D spatial position information into the 384-channel features; The attention mechanism module contains multiple Transformer self-attention mechanism blocks, which uses query, key, and value to perform linear transformation on 384-channel features and calculate global context dependencies to generate features of the four modal images of each sample that fuse local and global semantic information.

[0033] It should be noted that the use of a four-branch image encoder, with the same structure and shared weights across all four branches, significantly reduces the number of model parameters, thereby lowering computational overhead and storage requirements and improving model efficiency. Furthermore, each branch specifically processes images of a single modality, ensuring independent encoding of each modality's features while flexibly handling the differences between images of different modalities and improving the model's adaptability to information from different modalities. Furthermore, the identical encoder structure makes the model training process more efficient, avoids redundant learning tasks, promotes rapid model convergence, and helps improve the model's generalization capabilities.

[0034] (2) Feature fusion module The feature fusion module is used to fuse the features of the four modal images of each sample to generate image embedding.

[0035] (3) Prompt encoder The prompt encoder is used to generate prompt embeddings. The details are as follows: The cue encoder includes a point cue encoder and a mask cue encoder; The point cue encoder encodes the spatial position of the input point cue and maps the cue to a 384-dimensional feature space through the point embedding layer to generate a point cue embedding, where the input point cue is determined based on any point in the sample when the model processes each sample; The mask hint encoder takes the input mask hint and reduces it to 384-dimensional features through two mask downsampling modules using 3D convolution, LayerNorm3d and GELU activation functions to generate a mask hint embedding. The input mask hint is determined based on the mask output generated by the mask encoder. The hint encoder generates hint embeddings based on the point hint embeddings and the mask hint embeddings.

[0036] (4) Mask decoder The mask decoder is used to generate mask output based on the image embedding and hint embedding.

[0037] The details are as follows: The mask decoder includes a bidirectional interaction module, an upsampling module, and a multi-layer perception module; The bidirectional interaction module is built on the Transformer structure and consists of multiple bidirectional attention blocks. Each attention block uses both self-attention and cross-attention mechanisms to perform bidirectional information interaction and global context modeling between image embedding and cue embedding, and further optimizes feature expression through LayerNorm and MLP. The upsampling module uses multiple deconvolution layers, LayerNorm3d, and GELU to gradually improve the spatial resolution of low-resolution fused features; The multi-layer perception module dynamically generates weights for each category through the network and is equipped with an IoU prediction head to generate mask output.

[0038] (5) IDH classifier The IDH classifier is used to generate gene classification results based on image embedding and mask output. The details are as follows: The IDH classifier performs 3D average pooling and maximum pooling on the image embedding and mask output, respectively, concatenates the pooled features and inputs them into the fully connected layer to generate gene classification results, where the gene classification results include IDH wild type and IDH mutant types.

[0039] In step S104, the training set is input into the model for training, and a joint loss function is set to optimize the parameters of the model through the joint loss function to obtain an optimized model.

[0040] The training set is input into the model for training. The following describes the model processing by taking any sample in the training set as an example: Step 1: Sample R is processed separately in four branch image encoders on four multimodal MRI images (FLAIR mode, T1 mode, T1ce mode, T2 mode) to obtain the features of the four modal images of sample R; Step 2: The feature fusion module fuses the features of the four modal images of sample R to generate image embedding; Step 3: The prompt encoder obtains a first prompt based on any point in the sample R, and then obtains a first prompt embedding based on the first prompt; Step 4: The mask encoder generates a first mask output based on the image embedding and the first hint embedding, and converts it into a first mask hint corresponding to the first mask output; Step 5: The prompt encoder generates a second prompt embedding based on the first mask prompt and another arbitrary point in the sample R; Step 6: The mask encoder generates a second mask output based on the image embedding and the second hint embedding, and converts it into a second mask hint corresponding to the second mask output; It should be noted that steps 5 and 6 can be performed multiple times, depending on the artificially set hyperparameters. In this example, the second mask output is determined as the final mask output to segment the sample R.

[0041] Step 7: The IDH classifier generates gene classification results based on the second mask output and image embedding.

[0042] It should also be noted that before generating the gene classification result based on the second mask output and the image embedding in step 7, the second mask output and the image embedding need to be normalized so that they are in the same dimension.

[0043] Afterwards, based on the segmentation results of sample R and the gene classification results, the model parameters are optimized through the joint loss function, and then the next sample is processed, and the corresponding joint loss function value is obtained, and the model parameters are optimized again until the samples in the above training set are processed.

[0044] It should be noted that before the training set is input into the model for training, data augmentation processing can be performed on the training set, including random rotation, random noise addition, random cropping, and random flipping. The data-enhanced training set is then input into the model for training. The specific training form is the same as above and will not be described here one by one.

[0045] In addition, the above joint loss function includes the segmentation loss function And the classification loss function ; Segmentation loss function for:

[0046] in, and Represent the weight coefficients of the Dice loss function and the cross entropy loss function of the model, and represents the Dice loss function and cross entropy loss function of the i-th sample, and M represents the total number of samples in the training set.

[0047] The Dice loss function is: , The cross entropy loss function is: , in, The model predicts the In the sample The predicted label of the voxel, Indicates the In the sample The true segmentation label of each voxel, the dataset contains the true label of each sample, and are smoothing constants added to the numerator and denominator, respectively, to prevent division by zero errors and stabilize training. Indicates the The number of voxels in a sample.

[0048] Classification loss function for: , Where M represents the total number of samples in the training set, represents the true IDH mutation label of the i-th sample, "0" indicates that the i-th sample is wild type, "1" indicates that the i-th sample is a mutant type. Indicates the probability that the model predicts the i-th sample as a mutant type, The value range is .

[0049] The joint loss function is: , in, and is the learnable log-variance parameter corresponding to the segmentation and classification tasks, and It is a tensor initialized to 0 and updated adaptively through iterations during the training phase.

[0050] By exponentially weighting the loss of each task and adding the corresponding uncertainty term, the goal of dynamically balancing the losses of different tasks during training is achieved, thereby improving the overall performance and robustness of the model.

[0051] It can be understood that by introducing a learnable logarithmic variance parameter, the segmentation loss and IDH typing loss are adaptively weighted, achieving a dynamic balance between the two tasks. This design promotes complementary learning between the segmentation and typing tasks, effectively improving the accuracy of brain glioma segmentation and IDH genotyping, and providing a solid technical guarantee for overall diagnostic performance.

[0052] Step S105: Input the test set into the optimized model for segmentation, IDH typing, and prediction to obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

[0053] For example, this embodiment uses the BraTS2020 public dataset, which contains 148 patient samples. 118 of these samples were used to train the model, and 30 of these samples were used as independent test sets for model evaluation. The Dice score indicator was used to evaluate the segmentation performance of the model in the entire tumor area. The Dice Score value ranges from 0 to 1. The larger the Dice score value, the better the segmentation effect. The IDH classification accuracy indicator was used to evaluate the IDH genotyping accuracy of the model. The IDH classification accuracy value ranges from 0 to 1. The larger the IDH classification accuracy value, the better the IDH genotyping effect. The results are shown in Table 1.

[0054]

[0055] Table 1 In summary, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method provided by the present invention adopts a four-branch input architecture. For the four modal MRI image data of T1, T1ce, Flair and T2, the image embedding is independently extracted through the image encoder with shared weights; then, the embedding features of each modality are integrated into a unified multimodal fusion feature using a feature fusion module. The fusion feature and the hint embedding generated by the hint encoder are input into the mask decoder together to generate a fine tumor segmentation mask, and at the same time, the binary classification prediction of the IDH gene status is realized through the IDH classifier. The model is trained end-to-end jointly, based on a joint loss function with learnable logarithmic variance parameters, to adaptively balance the segmentation and typing losses, effectively improving the stability and robustness of the model.

[0056] Combination of the above Figure 1-Figure 2 The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method and application provided in the embodiment of the present application are described in detail. The following details the multi-task brain glioma automatic segmentation and IDH genotyping system based on SAM provided in the embodiment of the present application.

[0057] The system specifically includes: an acquisition module, a processing module, a model building module, a model optimization module and an evaluation module, as shown below.

[0058] An acquisition module is used to acquire a multimodal MRI image dataset of a glioma, wherein the dataset includes multiple samples, and each sample includes four modal images; The processing module is used to preprocess the data set to obtain a preprocessed standardized data set, and divide the standardized data set into a training set and a test set according to a preset ratio; A model building module is used to establish a SAM-based multi-task model for automatic glioma segmentation and IDH genotyping, wherein the model includes: a four-branch image encoder, a feature fusion module, a prompt encoder, a mask decoder, and an IDH classifier; the four-branch image encoder is used to generate features of the four modal images of each sample; the feature fusion module is used to fuse the features of the four modal images of each sample to generate an image embedding; the prompt encoder is used to generate a prompt embedding; the mask decoder is used to generate a mask output based on the image embedding and the prompt embedding; the IDH classifier is used to generate a gene classification result based on the image embedding and the mask output; The model optimization module is used to input the training set into the model for training, set the joint loss function, optimize the model parameters through the joint loss function, and obtain the optimized model; The evaluation module is used to input the test set into the optimized model for segmentation, IDH typing and prediction, and obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

[0059] In addition, for the above-mentioned specific implementation of the system, since it is basically similar to the method implementation, the description is relatively simple, and the relevant parts can be referred to the partial description of the method implementation. Moreover, it should be noted that in each module of the system of the present application, the components therein are logically divided according to the functions to be implemented, but the present application is not limited thereto, and the components can be re-divided or combined as needed.

[0060] The above describes the multi-task brain glioma automatic segmentation and IDH genotyping method and system based on SAM provided by the embodiment of the present application. Figure 3-Figure 4 The invention provides a detailed description of the multi-task automatic brain glioma segmentation and IDH genotyping device based on SAM provided in the embodiments of the present application.

[0061] Figure 3 This is a schematic diagram of the structure of the multi-task brain glioma automatic segmentation and IDH genotyping device based on SAM provided in the embodiment of the present application. Figure 1 For example, Figure 3 As shown, the multi-task brain glioma automatic segmentation and IDH genotyping device 300 based on SAM includes: a transceiver module 301 and a processing module 302. For ease of description, Figure 3 Only the main components of the SAM-based multi-task automatic glioma segmentation and IDH genotyping device are shown.

[0062] Among them, the transceiver module 301 is used to perform the transceiver function of the above-mentioned SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method, and the processing module 302 is used to perform other functions of the above-mentioned SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method except the transceiver function.

[0063] Optionally, the transceiver module 301 may include a sending module ( Figure 3 Not shown) and the receiving module ( Figure 3 (not shown). The sending module is used to implement the sending function of the multi-task brain glioma automatic segmentation and IDH genotyping device 300 based on SAM, and the receiving module is used to implement the receiving function of the multi-task brain glioma automatic segmentation and IDH genotyping device 300 based on SAM.

[0064] Optionally, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device 300 may further include a storage module ( Figure 3 (not shown) The storage module stores a program or instruction. When the processing module 302 executes the program or instruction, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device 300 can perform the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to the embodiment of the present application.

[0065] The following combination Figure 4 Each component of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device 400 is introduced in detail: The processor 401 is the control center of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device 400, and can be a single processor or a collective term for multiple processing elements. For example, the processor 401 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0066] Optionally, the processor 401 can perform various functions of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device 400 by running or executing a software program stored in the memory 402 and calling data stored in the memory 402, such as executing the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method in the embodiment of the present application.

[0067] In a specific implementation, as an embodiment, the processor 401 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0068] In a specific implementation, as an embodiment, the multi-task brain glioma automatic segmentation and IDH genotyping device 400 based on SAM may also include multiple processors, such as Figure 4 4 and 5. The processor 401 and processor 404 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). The memory 402 is used to store the software program for executing the solution of the present application, and the execution is controlled by the processor 401. The specific implementation method can be referred to the above method embodiment and will not be repeated here.

[0069] Alternatively, the memory 402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 402 may be integrated with the processor 401 or may exist independently and interface with the IDH genotyping device 400 through the SAM-based multi-task brain glioma automatic segmentation interface circuit ( Figure 4 (not shown) is coupled to the processor 401, which is not specifically limited in this embodiment of the present application.

[0070] The transceiver 403 is used for communicating with other communication devices. For example, the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device 400 is a first device, and the transceiver 403 can be used to communicate with a second device or a third device.

[0071] Optionally, the transceiver 403 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0072] Optionally, the transceiver 403 can be integrated with the processor 401 or can exist independently and interface with the IDH genotyping device 400 through a multi-task brain glioma automatic segmentation based on SAM ( Figure 4 (not shown) is coupled to the processor 401, which is not specifically limited in this embodiment of the present application.

[0073] It is understandable that Figure 4 The structure of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device 400 shown in the figure does not constitute a limitation of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device. The actual SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0074] In addition, the technical effects of the SAM-based multi-task brain glioma automatic segmentation and IDH genotyping device 400 can refer to the technical effects of the methods described in the above method embodiments, and will not be repeated here.

[0075] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), but may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0076] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0077] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in the embodiments of this application are fully or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

Claims

1. A multi-task SAM-based method for automatic brain glioma segmentation and IDH genotyping, characterized by: The method comprises: Acquire a multimodal MRI image dataset of brain glioma, wherein the dataset includes multiple samples, and each sample includes four modality images; Preprocessing the data set to obtain a preprocessed standardized data set, and dividing the standardized data set into a training set and a test set according to a preset ratio; Establish a SAM-based multi-task model for automatic glioma segmentation and IDH genotyping, wherein the model includes: a four-branch image encoder, a feature fusion module, a hint encoder, a mask decoder, and an IDH classifier; The four-branch image encoder is used to generate features of the four modal images of each sample; the feature fusion module is used to fuse the features of the four modal images of each sample to generate an image embedding; the prompt encoder is used to generate a prompt embedding; the mask decoder is used to generate a mask output based on the image embedding and the prompt embedding; the IDH classifier is used to generate a gene classification result based on the image embedding and the mask output; Inputting the training set into the model for training, setting a joint loss function, optimizing the parameters of the model by the joint loss function, and obtaining an optimized model; The test set is input into the optimized model for segmentation, IDH typing and prediction to obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

2. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that: The preprocessing of the data set to obtain a preprocessed standardized data set includes: The samples in the dataset are normalized at the voxel level, and the normalization formula is: ， Among them, the is the set of foreground voxels of each sample in the dataset, and stated are the mean and standard deviation of the foreground voxels in the dataset, respectively. is the set of foreground voxels of each sample in the normalized dataset.

3. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that: The four-branch image encoder is used to generate features of four modal images for each sample, including: The four branch image encoders have the same structure and share weights. Each branch image encoder processes the four modal images of each sample respectively to generate features of the four modal images of each sample. Each branch image encoder includes an image embedding module, a position encoding module, and an attention mechanism module; The image embedding module uses a three-dimensional convolutional layer with a convolution kernel size of 16×16×16 to convert the four modality images of each sample into 384-channel features; The position encoding module introduces 3D spatial position information into the 384-channel features; The attention mechanism module includes multiple Transformer self-attention mechanism blocks, which use queries, keys, and values ​​to linearly transform the 384-channel features and calculate global context dependencies to generate features of the four modal images of each sample that fuse local and global semantic information.

4. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that: The prompt encoder is used to generate prompt embeddings, including: The cue encoder includes a point cue encoder and a mask cue encoder; The point cue encoder encodes the spatial position of the input point cue and maps the cue to a 384-dimensional feature space through a point embedding layer to generate a point cue embedding, wherein the input point cue is determined based on any point in the sample when the model processes each sample; The mask hint encoder generates a mask hint embedding by reducing the input mask hint to 384-dimensional features using two mask downsampling modules, using 3D convolution, LayerNorm3d and GELU activation functions, wherein the input mask hint is determined based on the mask output generated by the mask encoder; The hint encoder generates a hint embedding based on the point hint embedding and the mask hint embedding.

5. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that: The mask decoder is configured to generate a mask output based on the image embedding and the hint embedding, comprising: The mask decoder includes a bidirectional interaction module, an upsampling module and a multi-layer perception module; The bidirectional interaction module is built based on the Transformer structure and consists of multiple bidirectional attention blocks. Each attention block uses both self-attention and cross-attention mechanisms to perform bidirectional information interaction and global context modeling on the image embedding and the prompt embedding, and further optimizes feature expression through LayerNorm and MLP. The upsampling module uses multiple deconvolution layers, LayerNorm3d and GELU to gradually improve the spatial resolution of the low-resolution fusion features; The multi-layer perception module dynamically generates weights for each category through the network and is equipped with an IoU prediction head to generate mask output.

6. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that: The IDH classifier is used to generate a gene classification result based on the image embedding and the mask output, including: The IDH classifier performs 3D average pooling and maximum pooling on the image embedding and the mask output, respectively, splices the pooled features and inputs them into a fully connected layer to generate a gene classification result, wherein the gene classification result includes IDH wild type and IDH mutant type.

7. The SAM-based multi-task brain glioma automatic segmentation and IDH genotyping method according to claim 1, characterized in that: The joint loss function includes the segmentation loss function And the classification loss function ; The segmentation loss function for: ； Among them, the and stated Represent the weight coefficients of the Dice loss function and the cross entropy loss function of the model respectively. and stated represents the Dice loss function and cross entropy loss function of the i-th sample, and M represents the total number of samples in the training set; The Dice loss function is: ， The cross entropy loss function is: ， Among them, the Indicates the number of samples predicted by the model The predicted labels of the voxels are Indicates the In the sample The true segmentation label of each voxel, the dataset contains the true label of each sample, With the are smoothing constants added to the numerator and denominator respectively, used to prevent division by zero errors and stabilize training. Indicates the The number of voxels in a sample; The classification loss function for: ， Wherein, M represents the total number of samples in the training set, represents the true IDH mutation label of the i-th sample, "0" indicates that the i-th sample is wild type, "1" indicates that the i-th sample is a mutant type. Represents the probability that the model predicts the i-th sample to be a mutant type, The value range is ; The joint loss function is: ， Among them, the and stated is the learnable logarithmic variance parameter corresponding to the segmentation and classification tasks, and stated It is a tensor initialized to 0 and updated adaptively through iterations during the training phase.

8. A multi-task brain glioma automatic segmentation and IDH genotyping system based on SAM, characterized by: The system comprises: An acquisition module is used to acquire a multimodal MRI image dataset of a glioma, wherein the dataset includes a plurality of samples, and each sample includes four modal images; A processing module is used to preprocess the data set to obtain a preprocessed standardized data set, and divide the standardized data set into a training set and a test set according to a preset ratio; A model building module is used to establish a SAM-based multi-task brain glioma automatic segmentation and IDH genotyping model, wherein the model includes: a four-branch image encoder, a feature fusion module, a prompt encoder, a mask decoder and an IDH classifier; the four-branch image encoder is used to generate features of four modal images of each sample; the feature fusion module is used to fuse the features of the four modal images of each sample to generate an image embedding; the prompt encoder is used to generate a prompt embedding; the mask decoder is used to generate a mask output based on the image embedding and the prompt embedding; the IDH classifier is used to generate a gene classification result based on the image embedding and the mask output; A model optimization module, configured to input the training set into the model for training, set a joint loss function, optimize the parameters of the model by the joint loss function, and obtain an optimized model; An evaluation module is used to input the test set into the optimized model for segmentation, IDH typing and prediction, and obtain the tumor segmentation accuracy and IDH typing accuracy of the optimized model.

9. A multi-task brain glioma automatic segmentation and IDH genotyping device based on SAM, characterized by: The apparatus comprises: a module for executing the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program or instructions, which, when executed, causes the method according to any one of claims 1 to 7 to be performed.

Images