Glioma multi-task diagnosis and segmentation method based on multi-modal MRI and graph learning

Abstract

This application provides a multi-task diagnostic and segmentation method for gliomas based on multimodal MRI and graph learning. The method includes: acquiring a multimodal MRI image dataset of gliomas; preprocessing the dataset and dividing it into training, validation, and test sets according to a preset ratio; constructing a multi-task diagnostic and segmentation model for gliomas, the model consisting of an encoder, a biomarker relationship graph learning module, a segmenter, and a classifier; inputting the training set into the model for training, constructing a joint loss function based on the true segmentation labels, true biomarker labels, and the predicted segmentation labels and predicted biomarker labels output by the model, optimizing the model parameters through this joint loss function to obtain the trained model; validating the trained model using the validation set, selecting the best-performing model, and then using it to process the test set, outputting the predicted segmentation labels and predicted biomarker labels corresponding to the samples in the test set.

Description

Technical Field

[0001] This application relates to the fields of medical image analysis and artificial intelligence technology, specifically to a method, system, device, and storage medium for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning. Background Technology

[0002] Gliomas are the most common primary malignant intracranial tumors in adults. Their high heterogeneity and diffuse invasive growth characteristics make clinical treatment challenging and lead to poor patient prognosis. According to the World Health Organization (WHO) classification of tumors of the central nervous system, gliomas are classified into grades I-IV based on their malignancy. High-grade gliomas (such as glioblastoma) have an extremely poor prognosis, with a median survival of less than 15 months. Recent studies have shown that molecular genetic characteristics, such as isocitrate dehydrogenase (IDH) mutations and 1p / 19q chromosomal co-deletion, are of crucial guiding significance in assessing the malignancy of gliomas, selecting treatment options, and predicting prognosis.

[0003] Magnetic resonance imaging (MRI) has become a core tool for diagnosing gliomas due to its non-invasive nature, high soft tissue resolution, and multimodal imaging capabilities. Conventional multimodal MRI sequences (such as T1, T1CE, T2, and FLAIR) can reveal the anatomical structure and pathological features of tumors from different angles. However, clinical diagnosis still heavily relies on the physician's subjective experience, presenting challenges such as low efficiency and poor consistency.

[0004] In recent years, deep learning technology, especially convolutional neural networks (CNN) and the Transformer architecture, has shown great potential in the automated segmentation and molecular subtyping prediction of gliomas. However, existing technical solutions generally have the following key limitations: (1) Task isolation: Most models focus only on a single task (such as segmentation or prediction of only one molecular marker), failing to effectively utilize the potential interdependencies and synergistic effects between different tasks. (2) Neglecting the correlation between biomarkers: Key biomarkers of gliomas (such as IDH status, 1p / 19q status, and tumor grade) have inherent hierarchical and correlational relationships in biology. Existing methods generally lack mechanisms for explicitly modeling such deep correlations. (3) Difficulty in handling label incompleteness: Obtaining comprehensive and accurate molecular pathology gold standard labels is costly, resulting in missing labels in training data. Existing models cannot efficiently and fully utilize such partially labeled data for training.

[0005] Therefore, based on the above problems, there is an urgent need for an innovative method that can integrate multimodal MRI information, explicitly model biomarker dependencies, and effectively and synergistically optimize segmentation and multiple molecular subtyping tasks. Summary of the Invention

[0006] This application provides a method, system, device, and storage medium for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning, achieving collaborative optimization of segmentation and diagnosis tasks.

[0007] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides a multi-task diagnosis and segmentation method for gliomas based on multimodal MRI and graph learning, the method comprising: Obtain a multimodal MRI image dataset of gliomas, which includes multiple samples, as well as the corresponding true segmentation labels and true biomarker labels for each sample; The dataset is preprocessed, and then divided into training, validation and test sets according to a preset ratio. A multi-task diagnostic and segmentation model for gliomas was constructed. The model includes: an encoder for extracting multi-scale deep image features and global semantic features from multimodal MRI images; a biomarker relationship graph learning module for constructing a graph structure with biomarker states as nodes and outputting association features between biomarkers through graph embedding and graph neural network learning; a segmenter for outputting predicted segmentation labels based on multi-scale deep image features and global semantic features; and a classifier for outputting predicted biomarker labels based on multi-scale deep image features, global semantic features, and association features. The training set is input into the model for training. Based on the real segmentation label, real biomarker label, predicted segmentation label and predicted biomarker label corresponding to each sample, a joint loss function is constructed. The parameters of the model are then optimized through the joint loss function to obtain the trained model. The trained model is validated using a validation set. The model with the best performance is selected, and then the model with the best performance is used to process the test set. The predicted segmentation label and predicted biomarker label corresponding to the samples in the test set are output.

[0008] Secondly, this application provides a multi-task diagnostic and segmentation system for gliomas based on multimodal MRI and graph learning, the system comprising: The data acquisition module is used to acquire a multimodal MRI image dataset of gliomas. The dataset includes multiple samples, as well as the real segmentation label and real biomarker label for each sample. The data processing module is used to preprocess the dataset and then divide the preprocessed dataset into training set, validation set and test set according to a preset ratio. The model building module is used to construct a multi-task diagnostic and segmentation model for gliomas. The model includes: an encoder for extracting multi-scale deep image features and global semantic features from multimodal MRI images; a biomarker relationship graph learning module for constructing a graph structure with biomarker states as nodes and outputting association features between biomarkers through graph embedding and graph neural network learning; a segmenter for outputting predicted segmentation labels based on multi-scale deep image features and global semantic features; and a classifier for outputting predicted biomarker labels based on multi-scale deep image features, global semantic features, and association features. The training module is used to input the training set into the model for training, and to construct a joint loss function based on the real segmentation label, real biomarker label, predicted segmentation label and predicted biomarker label corresponding to each sample. The parameters of the model are then optimized through the joint loss function to obtain the trained model. The prediction module is used to validate the trained model using the validation set, select the best-performing model, and then use the best-performing model to process the test set, outputting the predicted segmentation label and predicted biomarker label for the samples in the test set.

[0009] Thirdly, a multi-task diagnostic and segmentation device for gliomas based on multimodal MRI and graph learning is provided, the device including modules for performing the method of the first aspect described above.

[0010] In one possible design, the glioma multi-task diagnosis and segmentation device based on multimodal MRI and graph learning in the third aspect may further include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the glioma multi-task diagnosis and segmentation device based on multimodal MRI and graph learning in the third aspect and other devices.

[0011] In one possible design, the glioma multitasking diagnostic and segmentation device based on multimodal MRI and graph learning in the third aspect may further include a memory. This memory may be integrated with the processor or disposed separately. The memory may be used to store instructions related to the method of the first aspect.

[0012] Fourthly, a multi-task diagnostic and segmentation device for gliomas based on multimodal MRI and graph learning is provided. This multi-task diagnostic and segmentation device for gliomas based on multimodal MRI and graph learning includes: a processor coupled to a memory, the processor executing instructions stored in the memory to cause the multi-task diagnostic and segmentation device for gliomas based on multimodal MRI and graph learning to perform the method of the first aspect.

[0013] In one possible design, the glioma multi-task diagnosis and segmentation device based on multimodal MRI and graph learning in the fourth aspect may further include a transceiver. This transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the glioma multi-task diagnosis and segmentation device based on multimodal MRI and graph learning in the fourth aspect and other devices.

[0014] Fifthly, a multi-task diagnostic and segmentation apparatus for gliomas based on multimodal MRI and graph learning is provided, comprising: a processor and a memory; the memory is used to store instructions, which, when executed by the processor, cause the multi-task diagnostic and segmentation apparatus for gliomas based on multimodal MRI and graph learning to perform the method of the first aspect.

[0015] In one possible design, the glioma multi-task diagnosis and segmentation device based on multimodal MRI and graph learning in the fifth aspect may further include a transceiver. This transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used to communicate between the glioma multi-task diagnosis and segmentation device based on multimodal MRI and graph learning in the fifth aspect and other devices.

[0016] In a sixth aspect, a computer-readable storage medium is provided, the computer-readable storage medium including a computer program or instructions that, when executed, cause the multi-task diagnosis and segmentation method for glioma based on multimodal MRI and graph learning of the first aspect to be performed.

[0017] In this embodiment, the application achieves multi-task diagnosis and segmentation of gliomas through deep fusion learning of multimodal MRI images and biomarker relationship graphs. Specifically, it utilizes a Transformer to effectively extract deep image features and global semantic features from multimodal MRI, and transforms biomarker information into learnable graph embedding representations through biomarker relationship graph learning, thereby capturing their intrinsic correlation features. This organic combination of image features and biomarker correlation features enables the model to comprehensively understand tumor characteristics from both image and molecular dimensions. Furthermore, the application employs a channel attention mechanism classifier and a skip connection segmenter to ensure effective integration of features from different tasks and high-precision segmentation. Through a weighted summation multi-task loss function, the application balances the importance of different tasks, prompting the model to simultaneously optimize across multiple tasks such as tumor segmentation, glioma grading, IDH genotype, and 1p / 19q state prediction, significantly improving overall diagnostic performance. This application requires only a single model to handle multiple diagnostic tasks, simplifying clinical workflows and improving the comprehensiveness and accuracy of diagnosis, providing strong support for precision medicine for gliomas.

[0018] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the multi-task diagnosis and segmentation method for gliomas based on multimodal MRI and graph learning provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the missing modality brain tumor generation network model provided in the embodiments of this application; Figure 3 A schematic diagram of the structure of the multi-task diagnostic and segmentation device for gliomas based on multimodal MRI and graph learning provided in this application embodiment. Figure 1 ; Figure 4 A schematic diagram of the structure of the multi-task diagnostic and segmentation device for gliomas based on multimodal MRI and graph learning provided in this application embodiment. Figure 2 . Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. At the same time, in the description of the embodiments of this application, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0022] Figure 1 This is a flowchart illustrating the multi-task diagnosis and segmentation method for gliomas based on multimodal MRI and graph learning, provided in an embodiment of this application.

[0023] The workflow of this multi-task diagnostic and segmentation method for gliomas based on multimodal MRI and graph learning is as follows: Step S101: Obtain a multimodal MRI image dataset of glioma, wherein the dataset includes multiple samples, as well as the real segmentation label and real biomarker label corresponding to each sample.

[0024] For example, three datasets, UCSF-PDGM, TCGA and EGD, can be used to conduct experiments. All samples contain complete multimodal MRI image data, namely T1, T2, T1CE and FLAIR.

[0025] It should be noted that the real segmentation label for each sample in this application is used to mark the regions where different tumors are located. The real biomarker labels for each sample in this application include real labels for glioma grading, real labels for IDH gene typing, and real labels for 1p / 19q gene deletion status. Please refer to the following text for details.

[0026] Step S102: Preprocess the dataset, and then divide the preprocessed dataset into training set, validation set and test set according to a preset ratio.

[0027] In this embodiment of the application, the dataset is preprocessed, including: Each sample in the dataset includes four modal images (T1, T2, T1CE, FLAIR). Voxel-level normalization is performed on each of the four modal images in all samples of the dataset. The normalization formula is as follows: , in, Let v be the set of foreground voxels for the v-th modality image of each sample in the dataset. and Let be the mean and standard deviation of the foreground voxel of the v-th modality image for each sample in the dataset. This is the set of foreground voxels for the v-th modality image of each sample after preprocessing the dataset.

[0028] It should also be noted that this application only normalizes the foreground (non-zero) voxels of MRI, while the background (zero-value voxels) is explicitly preserved as zero.

[0029] In this embodiment of the application, the preprocessed dataset is divided into a training set, a validation set, and a test set according to a preset ratio, including: The true biomarker labels corresponding to each sample in the sample set include true labels for glioma grading, true labels for IDH genotyping, and true labels for 1p / 19q gene deletion status; The labeling status for glioma grading includes: missing, low-grade glioma, and high-grade glioma; The tag status for IDH genotyping includes: deletion, wild-type IDH, and mutant IDH. The label status of 1p / 19q gene deletion includes: deletion, 1p / 19q co-deletion, and 1p / 19q non-co-deletion; The training set, validation set, and test set all include a certain proportion of samples corresponding to three label states: glioma grade, IDH genotype, and 1p / 19q gene deletion state.

[0030] It is understandable that the training, validation, and test sets, in addition to the presence of missing biomarker states, also avoid the imbalance problem of both positive and negative samples. Specifically, the missing, positive, and negative samples for the three labeled states—low-grade glioma (negative sample), high-grade glioma (positive sample), IDH wild-type (negative sample), IDH mutant (positive sample), and 1p / 19q co-deletion (positive sample) and 1p / 19q non-co-deletion (negative sample)—are divided according to a certain ratio, such as 7:2:1. This ratio is only an example and can be adjusted according to actual circumstances. Table 1 below provides a detailed example.

[0031] Table 1 In this embodiment of the application, after dividing the preprocessed dataset into a training set, a validation set, and a test set, the method further includes: Data augmentation processing is performed on multimodal MRI images in the training set. This data augmentation processing includes random image rotation, random flipping, random cropping, and random noise addition, and may also include padding.

[0032] Specifically, this can be understood as: Padding: Zeros are added to the axial ends to the standard size, which is 128×128×128 in this specific example to ensure the consistency of the model input size; Random cropping: In the H×W×D space dimension, the cropping range is determined according to the sample image size, and 128×128×128 voxel blocks are randomly cropped; Random flipping: Three-dimensional rotation is performed along the axial, sagittal, and coronal directions with a probability of 0.5; Random noise addition: The grayscale value of the image is adjusted within ±10%, etc.

[0033] Step S103: Construct a multi-task diagnostic and segmentation model for gliomas. The model includes: an encoder for extracting multi-scale deep image features and global semantic features from multimodal MRI images; a biomarker relationship graph learning module for constructing a graph structure with biomarker states as nodes and outputting association features between biomarkers through graph embedding and graph neural network learning; a segmenter for outputting predicted segmentation labels based on multi-scale deep image features and global semantic features; and a classifier for outputting predicted biomarker labels based on multi-scale deep image features, global semantic features, and association features.

[0034] Among them, the multi-task diagnosis and segmentation model for gliomas can be combined with Figure 2The model comprises an encoder, a biomarker relationship graph learning module, a segmenter, and a classifier. The following is a detailed description of the model: (1) Encoder The encoder includes a convolutional neural network and a Transformer module; Convolutional neural networks, consisting of multiple cascaded convolutional layers, are used to downsample multimodal MRI images layer by layer and output multi-scale deep image features. The Transformer module receives multi-scale deep image features output from the last layer of the convolutional neural network. After linearly projecting the multi-scale deep image features, it inputs them into a structure consisting of at least two stacked Transformer coding layers. Each Transformer coding layer contains a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization to extract global semantic features across space.

[0035] Specifically, this can be understood as: The encoder starts with an initial convolutional layer that receives 4 input channels. It maps these channels to 16 output channels through a 3x3x3 convolutional kernel and is accompanied by a 0.2 Dropout operation to enhance the model's regularization ability, prevent overfitting, and transform the raw image data into a unified feature representation for subsequent model processing. Subsequently, multi-level downsampling is performed to capture more abstract and higher-level semantic information, enabling the network to capture image features from multiple scales while preserving rich details and edge information. In the final downsampling stage, batch normalization (BN) and the LeakyReLU activation function are applied to stabilize training and introduce non-linearity. Through a 3x3x3 convolutional layer, the 128-channel spatial features are transformed into high-dimensional channel features, which can be represented as: in, Represents the original multimodal MRI image. For CNN encoder The output feature map (i.e., multi-scale deep image features). For the parameters of the Kth layer CNN encoder network, It represents the number of layers in the CNN encoder; Compared to the local receptive field of CNNs, the Transformer compensates for this deficiency, enabling the model to understand global contextual information. Each Transformer layer consists of a multi-head self-attention mechanism (MHSA), a feedforward network (FFN), residual connections, and layer normalization (LN), and its internal operations are defined as follows: in, Indicates the first The output features of the layer Transformer This layer's self-attention module outputs the intermediate result after connecting it to the residual; This represents a multi-head self-attention mechanism used to capture global dependencies between features; This represents a feedforward network consisting of two fully connected layers and using the ReLU activation function. and The LayerNorm operation is used for stable training. Each layer contains two residual connection structures, improving the trainability and gradient propagation efficiency of deep networks. The query, key, and value in the self-attention mechanism are obtained through a linear transformation of the input features and used to calculate the attention weights.

[0036] The final global semantic features after four Transformer layers are represented as follows: in, This represents the output of different Transformer layers. This indicates function composition, where the output of the previous function serves as the input to the next function, executed sequentially from right to left. The final result is the global semantic features. It integrates dependency information across spatial ranges, providing a unified, highly expressive high-level feature for subsequent tumor segmentation and molecular subtyping tasks.

[0037] (2) Biomarker Relationship Map Learning Module The biomarker relationship graph learning module is used to construct a graph structure with biomarker states as nodes, and outputs the association features between biomarkers through graph embedding and graph neural network learning, including: Based on the real biomarker labels of all samples in the dataset, a graph structure with biomarker status as nodes is constructed; By statistically analyzing the co-occurrence frequency of different types of nodes in all samples, a co-occurrence matrix characterizing the strength of association between nodes is established. The GloVe model based on global word vectors is used to map discrete nodes in the co-occurrence matrix into low-dimensional continuous vectors to generate a node embedding matrix. The adjacency relationship of the graph is constructed based on the co-occurrence matrix, and then sparsification and normalization are performed to obtain the normalized adjacency matrix. The node embedding matrix and the normalized adjacency matrix are input into the graph convolutional network. Through multi-layer graph convolution operations, the network learns and outputs the association features between biomarkers.

[0038] Specifically, this can be understood as: First, all possible combinations of values ​​for the three biomarkers—glioma grade, IDH genotype, and 1p / 19q status—are defined as nodes in the graph, as follows: Glioma grading nodes: Grade=0 (LGG), Grade=1 (HGG); IDH genotype nodes: IDH=0 (wild type), IDH=1 (mutant type); p / 19q state nodes: 1p / 19q=0 (non-common missing), 1p / 19q=1 (common missing); These nodes together form the vertex set of the graph, with a total number of nodes C=6; The number of samples is set to N, and each sample has multiple labeling tasks T={t1,t2,…,tK}, where the set of values ​​for each task tK is... Where k is 6, and the value sets represent negative sample 0 and positive sample 1 respectively. Representing all possible label-value pairs, defined as: The total number of nodes is For each node Construct a co-occurrence matrix A vector of the number of times a node appears Mij ​​represents a node With nodes The number of times they co-occur in the same sample, provided that and From different tag tasks. This represents the total number of times node i appears in all samples. For each sample in the dataset... Extract the non-missing label items to form a set. Then, the following statistics were performed: For each Increment its occurrence count by one: for ,if Since they come from different tasks, the co-occurrence matrix is ​​updated as follows: in Represents a node The corresponding labeling task. The final matrix obtained. sum vector It is serialized and saved for subsequent tasks such as graph structure modeling or label dependency modeling.

[0039] To convert discrete biomarker nodes and their co-occurrence relationships into continuous, low-dimensional vector representations, this application uses the GloVe model for graph embedding learning. The co-occurrence matrix is converted into a label embedding matrix . In this example, the embedding dimension d is set to 128 and C is the number of label nodes, which is 6.

[0040] First, is converted into a sparse representation to improve training efficiency and memory utilization. The label node strings are mapped to indices, and a dictionary is defined: Glove is used to train the sparse co-occurrence matrix , and the goal is to minimize the following loss function: where, and are the main word and context word embeddings respectively, and are bias terms, is a weighting function: The embedding vectors of all nodes are obtained through iterative optimization. Among them, is a preset constant, and its specific value is adjusted according to the corpus size and task. After training, the main word embedding of the label nodes is extracted, that is: The key parameters of training include the learning rate of 0.05, the number of training epochs of 500, and the number of parallel threads of 4. Each row in matrix E represents the initial embedding vector of a biomarker node, which is used as the input for the subsequent graph neural network.

[0041] After obtaining the initial embeddings of the nodes, the graph neural network is used to further learn the deep association features between biomarker nodes. First, the adjacency matrix of the graph is constructed. According to the co-occurrence matrix M and the node frequency vector N, an initial weighted adjacency matrix is generated, and its element can be expressed as: Then, is sparsified: a prediction threshold t = 0.4 is set. When <t, it is set to 0, otherwise it is set to 1. Then normalization is performed, and the self-connection identity matrix I is added to ensure the coherence of the graph, and the final adjacency matrix is obtained. In the graph neural network, in order to perform graph convolution operations, the adjacency matrix Perform symmetric normalization. The normalized adjacency matrix. The calculation is as follows: Where S is a degree matrix, a diagonal matrix whose diagonal elements This embodiment employs a two-layer graph convolutional network structure. Graph convolutional layer +1 output feature It can be represented as: in, It is the first The input feature matrix of the layer, It is the first The learnable weight matrix of the layer, It is the activation function. Specifically, the first layer of the graph convolutional network maps the 128-dimensional node embedding features to a 64-dimensional hidden layer and uses the LeakyReLU activation function with a negative slope of 0.2. The second layer of the graph convolutional network maps the 64-dimensional hidden layer features to a 32-dimensional output space. Finally, the output of the GNN is a feature representation containing deep correlation information between biomarkers. These correlated features are then fused with image features to jointly guide the comprehensive diagnosis of gliomas.

[0042] (3) Divider The segmenter consists of: multiple sequentially connected upsampling and fusion modules, a 1x1x1 convolutional layer, a softmax activation function layer, and a classification output layer. Multiple upsampling and fusion modules connected in sequence are used to perform upsampling and fusion of multi-scale deep image features and global semantic features step by step. Each module upsamples the multi-scale deep image features and global semantic features and then fuses them with the feature map of the corresponding scale from the encoder through skip connections. The fusion result is then output to the next level module, thereby gradually restoring the spatial resolution and detail information of the feature map and finally outputting a high-resolution feature map. A 1x1x1 convolutional layer is used to map the channels of the high-resolution feature map to the preset segmentation categories, generating classification logits for each pixel corresponding to each category. The preset segmentation categories include intact tumor, tumor core, and enhanced tumor. The Softmax activation function layer is used to transform the classification logits of each pixel corresponding to each category into the probability distribution of each pixel in all categories, where the sum of the probabilities of each pixel in each category is 1; The classification output layer is used to assign the highest probability category label to each pixel according to the probability distribution, and then output the predicted segmentation label.

[0043] (4) Classifier A classifier is used to output predicted biomarker labels based on multi-scale deep image features, global semantic features, and association features, including: The classifier performs average pooling and max pooling operations on multi-scale deep image features and global semantic features, respectively. The feature vectors obtained after pooling are concatenated along the channel dimension to obtain the fused feature vector; The fused feature vector is concatenated and fused with the associated features to obtain the interactive features; The interactive features are input into a multilayer perceptron containing at least two fully connected layers for feature transformation and dimensionality reduction, and then output predicted biomarker labels.

[0044] It is understandable that the deepest feature map of a CNN encoder contains rich local and intermediate semantic information, and its shape is (B, (D, H, W), where B is the batch size, which is set to 4 in this specific example. =128 is the number of channels, and D, H, and W are the spatial dimensions, all of which are 16. The output of the Transformer module incorporates global context information and long-range dependencies, and its shape is (B, ...). (D′, H′, W′), where The embedding dimension of the Transformer is 128, and D′, H′, and W′ are the same spatial dimensions as the deep features of the CNN, all of which are 16. For deep features in CNN and Transformer global semantic features Average pooling and max pooling operations are performed separately. The pooling kernel size is set to 16 spatial dimensions and the stride is 1. All pooled feature maps are flattened to transform them from five-dimensional tensors to two-dimensional tensors, and their shape is changed to the number of channels B.

[0045] These four flattened feature vectors are concatenated along the channel dimension and multiplied with the graph features learned by the biomarker module. Finally, they are passed through a multilayer perceptron containing two fully connected layers for further feature transformation and dimensionality reduction, and then the predicted biomarker label is output.

[0046] Step S104: Input the training set into the model for training, and construct a joint loss function based on the real segmentation label, real biomarker label, predicted segmentation label and predicted biomarker label corresponding to each sample. Optimize the model parameters through the joint loss function to obtain the trained model.

[0047] In this embodiment, the joint loss function is composed of the segmentation loss function. Classification loss function of biomarkers constitute.

[0048] Segmentation loss function It is the sum of segmentation losses calculated separately for the three segmentation categories: intact tumor, tumor core, and enhancing tumor. Specifically: ; Where N represents the total number of samples, k These represent the three segmentation categories for brain tumor segmentation. Indicates the first The sample at the th Predicted segmentation labels for each segmentation category. Indicates the first The sample at the th The true segmentation labels for each segmentation category It represents the smoothing coefficient, avoiding division by zero errors and promoting convergence; Classification loss function of biomarkers Including glioma grading subclassification loss IDH genotyping subclassification loss 1p / 19q gene deletion state subclassification loss Specifically: , , and The calculation formula is the same, which is ; Where N represents the total number of samples, Indicates the first The true labels for glioma grade / IDH genotyping / 1p / 19q gene deletion status of each sample are respectively assigned values ​​belonging to {-1, 0, 1}. When, it indicates the first The label status for each sample's glioma grade / IDH genotyping / 1p / 19q gene deletion status is "deleted". , indicating the first The label status for each sample's glioma grading / IDH genotyping / 1p / 19q gene deletion status is one of the two states other than deletion. hour, The value is 0 when hour, The value is 1. Indicates the first Predictive labels for glioma grade / IDH genotyping / 1p / 19q gene deletion status in each sample. and These are preset values ​​based on the true label information of all samples, including glioma grade, IDH genotype, and 1p / 19q gene deletion status. It is a very small positive number used to avoid the denominator being zero.

[0049] also, , , Losses for three subcategories (Glioma grading) (IDH genotyping) The weighting coefficients for (1p / 19q gene deletion status) are used to adjust the relative importance of different biomarker prediction tasks in the total loss function. The three coefficients are non-negative real numbers and can be adaptively set through the uncertainty weighting method in multi-task learning.

[0050] Specifically, the model is trained end-to-end using the training set. During training, the model parameters are updated iteratively using the Adam optimizer and an adaptive learning rate, with an initial learning rate of 0.0002, based on the total loss. The training epochs are set to multiple epochs (e.g., 500 epochs). During validation, the model with the lowest loss is selected and saved as the final model.

[0051] Step S105: Validate the trained model using the validation set, select the best-performing model, and then use the best-performing model to process the test set, outputting the predicted segmentation label and predicted biomarker label corresponding to the samples in the test set.

[0052] The above describes the processing of a complete image. In addition, the following methods can also be used: Data augmentation is performed on the samples in the test set to generate augmented samples; The enhanced MRI image of the sample was segmented into multiple overlapping fixed-size sub-regions, and each fixed-size sub-region was input into the best-performing model for prediction, resulting in a predicted segmentation label and a predicted biomarker label for each fixed-size sub-region. The final predicted segmentation label and predicted biomarker label are obtained by stitching together the predicted segmentation labels of all fixed-size sub-regions and averaging the predicted biomarker labels of all fixed-size sub-regions.

[0053] In practical applications, the segmentation results are saved in the standard NIFTI format and can be visualized as color images to intuitively show the WT distribution of the tumor.

[0054] For classification tasks, the model records and calculates the following key evaluation metrics in detail: Confusion matrix: Used to show the number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN); Accuracy (ACC): Measures the proportion of predictions that a model correctly makes in all predictions; Sensitivity (SEN): Recall: Measures the model's ability to correctly identify positive cases; Specificity (SPE): Measures the model's ability to correctly identify negative cases; Area under the Receiver Operating Characteristic (AUC): This evaluates the overall discriminative ability of a binary classification model across all possible classification thresholds. The AUC value represents the area under the ROC curve, ranging from 0 to 1. A value closer to 1 indicates better performance. Table 2 shows the prediction performance of three classification tasks on the test set.

[0055] Table 2

[0056] This application underwent comprehensive testing on two glioma datasets from different sources, demonstrating its effectiveness in multi-task diagnosis. In IDH genotype prediction and glioma grading tasks, the model achieved excellent and stable performance on the test set, with AUC and ACC values ​​both exceeding 0.8. This demonstrates the synergistic advantages of this application's efficient extraction of multimodal image features through the CNN-Transformer architecture and its deep integration with biomarker relationship modeling. This fusion mechanism enables the model to comprehensively understand tumors from both image representation and molecular biological perspectives, improving diagnostic accuracy and generalization ability. For the tumor segmentation task, the Dice coefficient for complete tumor regions on the three multicenter datasets reached as high as 0.8333, indicating its strong ability to accurately locate tumor boundaries.

[0057] In summary, this application achieves multi-task diagnosis and segmentation of gliomas through deep fusion learning of multimodal MRI images and biomarker relationship graphs. Specifically, it utilizes a Transformer to effectively extract deep image features and global semantic features from multimodal MRI, and transforms biomarker information into learnable graph embedding representations through biomarker relationship graph learning, thereby capturing their intrinsic correlation features. This organic combination of image features and biomarker correlation features enables the model to comprehensively understand tumor characteristics from both image and molecular dimensions. Furthermore, this application employs a channel attention mechanism classifier and a skip connection segmenter to ensure effective integration of features from different tasks and high-precision segmentation. Through a weighted summation multi-task loss function, this application balances the importance of different tasks, prompting the model to simultaneously optimize across multiple tasks such as tumor segmentation, glioma grading, IDH genotype, and 1p / 19q state prediction, significantly improving overall diagnostic performance. This application requires only a single model to handle multiple diagnostic tasks, simplifying clinical workflows and improving the comprehensiveness and accuracy of diagnosis, providing strong support for precision medicine for gliomas.

[0058] The above combination Figures 1-2 This application describes the multi-task diagnosis and segmentation method for gliomas based on multimodal MRI and graph learning provided in the embodiments of this application. The following details the multi-task diagnosis and segmentation system for gliomas based on multimodal MRI and graph learning provided in the embodiments of this application.

[0059] The system specifically includes: a data acquisition module, a data processing module, a model building module, a training module, and a prediction module, as shown below.

[0060] The data acquisition module is used to acquire a multimodal MRI image dataset of gliomas. The dataset includes multiple samples, as well as the real segmentation label and real biomarker label for each sample. The data processing module is used to preprocess the dataset and then divide the preprocessed dataset into training set, validation set and test set according to a preset ratio. The model building module is used to construct a multi-task diagnostic and segmentation model for gliomas. The model includes: an encoder for extracting multi-scale deep image features and global semantic features from multimodal MRI images; a biomarker relationship graph learning module for constructing a graph structure with biomarker states as nodes and outputting association features between biomarkers through graph embedding and graph neural network learning; a segmenter for outputting predicted segmentation labels based on multi-scale deep image features and global semantic features; and a classifier for outputting predicted biomarker labels based on multi-scale deep image features, global semantic features, and association features. The training module is used to input the training set into the model for training, and to construct a joint loss function based on the real segmentation label, real biomarker label, predicted segmentation label and predicted biomarker label corresponding to each sample. The parameters of the model are then optimized through the joint loss function to obtain the trained model. The prediction module is used to validate the trained model using the validation set, select the best-performing model, and then use the best-performing model to process the test set, outputting the predicted segmentation label and predicted biomarker label for the samples in the test set.

[0061] Furthermore, the specific implementation of the above system is basically similar to the method implementation, so the description is relatively simple. For relevant details, please refer to the description of the method implementation. Moreover, it should be noted that in the various modules of the system of this application, the components are logically divided according to the functions they are to perform. However, this application is not limited to this and can re-divide or combine the components as needed.

[0062] The above describes the method and system for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning provided in the embodiments of this application. The following, in conjunction with... Figures 3-4 This document provides a detailed description of the multi-task diagnostic and segmentation device for gliomas based on multimodal MRI and graph learning, as provided in the embodiments of this application.

[0063] Figure 3 This is a schematic diagram of the structure of the multi-task diagnosis and segmentation device for gliomas based on multimodal MRI and graph learning provided in this application embodiment. Figure 1 For example, such as Figure 3 As shown, the multi-task diagnosis and segmentation device 300 for gliomas based on multimodal MRI and graph learning includes: a transceiver module 301 and a processing module 302. For ease of explanation, Figure 3 Only the main components of this multi-task diagnostic and segmentation device for gliomas based on multimodal MRI and graph learning are shown.

[0064] The transceiver module 301 is used to perform the transceiver function of the above-mentioned multi-task diagnosis and segmentation method for glioma based on multimodal MRI and graph learning, and the processing module 302 is used to perform other functions of the above-mentioned multi-task diagnosis and segmentation method for glioma based on multimodal MRI and graph learning, excluding the transceiver function.

[0065] Optionally, the transceiver module 301 may include a transmitting module ( Figure 3 (not shown in the image) and receiving module ( Figure 3(Not shown in the image). The transmitting module is used to implement the transmitting function of the multi-task diagnostic and segmentation device 300 for gliomas based on multimodal MRI and graph learning, and the receiving module is used to implement the receiving function of the multi-task diagnostic and segmentation device 300 for gliomas based on multimodal MRI and graph learning.

[0066] Optionally, the multi-task diagnostic and segmentation device 300 for gliomas based on multimodal MRI and graph learning may also include a storage module ( Figure 3 (Not shown in the image), the storage module stores programs or instructions. When the processing module 302 executes the program or instructions, the multi-modal MRI and graph learning-based glioma multi-task diagnosis and segmentation device 300 can perform the multi-modal MRI and graph learning-based glioma multi-task diagnosis and segmentation method in the embodiments of this application.

[0067] The following is combined with Figure 4 Each component of the multi-task diagnostic and segmentation device 400 for gliomas based on multimodal MRI and graph learning is described in detail: The processor 401 is the control center of the multi-task diagnostic and segmentation device 400 for gliomas based on multimodal MRI and graph learning. It can be a single processor or a collective term for multiple processing elements. For example, the processor 401 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0068] Optionally, the processor 401 can execute various functions of the multi-modal MRI and graph learning-based glioma multi-task diagnosis and segmentation device 400 by running or executing software programs stored in the memory 402 and calling data stored in the memory 402, such as executing the multi-modal MRI and graph learning-based glioma multi-task diagnosis and segmentation method in the embodiments of this application.

[0069] In a specific implementation, as one example, processor 401 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0070] In a specific implementation, as one example, the multi-task diagnosis and segmentation device 400 for gliomas based on multimodal MRI and graph learning may also include multiple processors, for example... Figure 4 The processors 401 and 404 are shown in the diagram. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). The memory 402 is used to store the software program executing the scheme of this application, and its execution is controlled by the processor 401. Specific implementation methods can be found in the above method embodiments, and will not be repeated here.

[0071] Optionally, the memory 402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 402 may be integrated with the processor 401 or exist independently, and may be connected via the interface circuit of the multi-modal MRI and graph learning-based glioma multitasking diagnosis and segmentation device 400. Figure 4 (Not shown in the image) is coupled to processor 401, but this embodiment does not specifically limit this.

[0072] Transceiver 403 is used for communication with other communication devices. For example, in a multi-modal MRI and graph learning-based multi-task diagnosis and segmentation device for gliomas 400, which is the first device, transceiver 403 can be used to communicate with a second device or a third device.

[0073] Alternatively, transceiver 403 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0074] Optionally, the transceiver 403 can be integrated with the processor 401 or exist independently, and can be connected to the interface circuit of the multi-task diagnosis and segmentation device for gliomas based on multimodal MRI and graph learning 400. Figure 4 (Not shown in the image) is coupled to processor 401, but this embodiment does not specifically limit this.

[0075] Understandable Figure 4 Figure 4 The structure of the multi-modal MRI and graph learning-based multi-task diagnostic and segmentation device 400 for gliomas shown in the figure does not constitute a limitation on the multi-modal MRI and graph learning-based multi-task diagnostic and segmentation device for gliomas. Actual multi-modal MRI and graph learning-based multi-task diagnostic and segmentation devices for gliomas may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0076] Furthermore, the technical effects of the multi-modal MRI and graph learning-based multi-task diagnostic and segmentation device 400 for gliomas can be referred to the technical effects of the methods described in the above-described embodiments, and will not be repeated here.

[0077] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0078] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0079] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

Claims

1. A multi-task diagnostic and segmentation method for gliomas based on multimodal MRI and graph learning, characterized in that, The method includes: A multimodal MRI image dataset of gliomas was obtained, wherein the dataset includes multiple samples, and the true segmentation label and true biomarker label corresponding to each sample; The dataset is preprocessed, and then divided into training set, validation set and test set according to a preset ratio. A multi-task diagnostic and segmentation model for gliomas is constructed, comprising: an encoder for extracting multi-scale deep image features and global semantic features from multimodal MRI images; a biomarker relationship graph learning module for constructing a graph structure with biomarker states as nodes and outputting association features between biomarkers through graph embedding and graph neural network learning; a segmenter for outputting predicted segmentation labels based on the multi-scale deep image features and the global semantic features; and a classifier for outputting predicted biomarker labels based on the multi-scale deep image features, the global semantic features, and the association features. The training set is input into the model for training. A joint loss function is constructed based on the real segmentation label, real biomarker label, predicted segmentation label, and predicted biomarker label corresponding to each sample. The parameters of the model are then optimized using the joint loss function to obtain the trained model. The trained model is validated using a validation set, the best-performing model is selected, and then the best-performing model is used to process the test set to output the predicted segmentation label and predicted biomarker label corresponding to the samples in the test set.

2. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 1, characterized in that, The preprocessing of the dataset includes: Each sample in the dataset includes four modal images. Voxel-level normalization is performed on each of the four modal images in all samples of the dataset. The normalization formula is as follows: ， Among them, the The set of foreground voxels for the v-th modality image of each sample in the dataset, wherein... and stated Let be the mean and standard deviation of the foreground voxel of the v-th modality image for each sample in the dataset. This refers to the set of foreground voxels for the vth modality image of each sample after preprocessing the dataset.

3. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 1, characterized in that, The step of dividing the preprocessed dataset into a training set, a validation set, and a test set according to a preset ratio includes: The true biomarker labels corresponding to each sample in the sample set include true labels for glioma grading, true labels for IDH gene typing, and true labels for 1p / 19q gene deletion status. The label status for glioma grading includes: missing, low-grade glioma, and high-grade glioma; The tag status for the IDH genotyping includes: deletion, wild-type IDH, and mutant IDH. The tag status of the 1p / 19q gene deletion status includes: deletion, 1p / 19q co-deletion, and 1p / 19q non-co-deletion; The training set, validation set, and test set all include a certain proportion of samples corresponding to three label states: glioma grade, IDH genotype, and 1p / 19q gene deletion state.

4. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 1, characterized in that, After dividing the preprocessed dataset into training, validation, and test sets, the method further includes: The multimodal MRI images in the training set are subjected to data augmentation processing, which includes random image rotation, random flipping, random cropping, and random addition of noise.

5. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 1, characterized in that, The encoder includes: Convolutional neural networks, consisting of multiple cascaded convolutional layers, are used to downsample multimodal MRI images layer by layer and output multi-scale deep image features. The Transformer module receives the multi-scale deep image features output from the last layer of the convolutional neural network, performs linear projection on the multi-scale deep image features, and inputs them into a structure composed of at least two stacked Transformer coding layers. Each Transformer coding layer includes a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization to extract global semantic features across space.

6. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 1, characterized in that, The biomarker relationship graph learning module is used to construct a graph structure with biomarker states as nodes, and outputs the association features between biomarkers through graph embedding and graph neural network learning, including: Based on the real biomarker labels of all samples in the dataset, a graph structure with biomarker status as nodes is constructed. By statistically analyzing the co-occurrence frequency of different types of nodes in all samples, a co-occurrence matrix characterizing the strength of association between nodes is established. The GloVe model based on global word vectors is used to map the discrete nodes in the co-occurrence matrix into low-dimensional continuous vectors to generate a node embedding matrix. Based on the co-occurrence matrix, the adjacency relationship of the graph is constructed, and then sparsification and normalization are performed to obtain the normalized adjacency matrix. The node embedding matrix and the normalized adjacency matrix are input into a graph convolutional network. Through multi-layer graph convolutional operations, the network learns and outputs the association features between biomarkers.

7. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 4, characterized in that, The divider includes: Multiple upsampling and fusion modules connected in sequence are used to perform upsampling and fusion of the multi-scale deep image features and the global semantic features step by step. Each module upsamples the multi-scale deep image features and the global semantic features, fuses them with the feature map of the corresponding scale from the encoder through skip connections, and outputs the fusion result to the next level module, thereby gradually restoring the spatial resolution and detail information of the feature map, and finally outputting a high-resolution feature map. A 1x1x1 convolutional layer is used to map the channels of the high-resolution feature map to preset segmentation categories, generating classification logits for each pixel corresponding to each category. The preset segmentation categories include intact tumor, tumor core, and enhanced tumor. The Softmax activation function layer is used to transform the classification logits of each pixel corresponding to each category into the probability distribution of each pixel in all categories, where the sum of the probabilities of each pixel in each category is 1; The classification output layer is used to assign the highest probability category label to each pixel according to the probability distribution, and then output the predicted segmentation label.

8. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 1, characterized in that, The classifier is used to output predicted biomarker labels based on the multi-scale deep image features, the global semantic features, and the association features, including: The classifier performs average pooling and max pooling operations on the multi-scale deep image features and the global semantic features, respectively. The feature vectors obtained after pooling are concatenated along the channel dimension to obtain the fused feature vector; The fused feature vector and the associated feature are concatenated and fused to obtain the interactive feature; The interactive features are input into a multilayer perceptron containing at least two fully connected layers for feature transformation and dimensionality reduction, and then output predicted biomarker labels.

9. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 3, characterized in that, The joint loss function is composed of the segmentation loss function. Classification loss function of biomarkers constitute; The segmentation loss function It is the sum of segmentation losses calculated separately for the three segmentation categories: intact tumor, tumor core, and enhancing tumor. Specifically: ； Where N represents the total number of samples, and k These represent the three segmentation categories for brain tumor segmentation. Indicates the first The sample at the th The predicted segmentation labels for each segmentation category, the Indicates the first The sample at the th The true segmentation labels of each segmentation category, the It represents the smoothing coefficient, avoiding division by zero errors and promoting convergence; The classification loss function of the biomarkers Including glioma grading subclassification loss IDH genotyping subclassification loss 1p / 19q gene deletion state subclassification loss Specifically: ， , and The calculation formula is the same, which is ; Wherein, N represents the total number of samples, and the Indicates the first The true labels for glioma grade / IDH genotyping / 1p / 19q gene deletion status of each sample are respectively assigned values ​​belonging to {-1, 0, 1}. When, it indicates the first The label status for each sample's glioma grade / IDH genotyping / 1p / 19q gene deletion status is "deleted". , indicating the first The label status for each sample's glioma grading / IDH genotyping / 1p / 19q gene deletion status is one of the two states other than deletion. hour, The value is 0 when hour, The value is 1, the Indicates the first Predictive labels for glioma grading / IDH genotyping / 1p / 19q gene deletion status in each sample, the aforementioned and These are preset values ​​based on the true label information of all samples' glioma grade / IDH genotyping / 1p / 19q gene deletion status. It is a very small positive number, used to avoid the denominator being zero. , , The three sub-classification losses are respectively preset. , , The weighting coefficients.

10. The method for multi-task diagnosis and segmentation of gliomas based on multimodal MRI and graph learning according to claim 3, characterized in that, The process of using the optimal performance model to process the test set and output the predicted segmentation labels and predicted biomarker labels corresponding to the samples in the test set includes: The samples in the test set are subjected to data augmentation to generate augmented samples; The enhanced MRI image of the sample is segmented into multiple overlapping fixed-size sub-regions, and each of the multiple fixed-size sub-regions is input into the best-performing model for prediction, to obtain the predicted segmentation label and predicted biomarker label for each fixed-size sub-region; The final predicted segmentation label and predicted biomarker label are obtained by stitching together the predicted segmentation labels of all fixed-size sub-regions and averaging the predicted biomarker labels of all fixed-size sub-regions.

Images