Glioma preoperative diagnosis method based on digital pathological image guided magnetic resonance imaging

Through digital pathological images, the magnetic resonance imaging is guided, deep learning models are constructed, and MRI and digital pathological image features are extracted, which solves the problem of non-invasive acquisition of glioma molecular markers before surgery, and achieves efficient and accurate multimolecular marker prediction, improving The reliability and accuracy of the diagnosis are achieved.

CN118982535AInactive Publication Date: 2024-11-19XI'AN POLYTECHNIC UNIVERSITY

Patent Information

Application Number
CN202411261036.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to obtain molecular marker information for glioma patients without invasiveness and convenience before surgery, and the existing MRI prediction accuracy and robustness are poor, making it impossible to achieve accurate preoperative diagnosis.

Method used

Through digital pathological images, we can obtain conventional preoperative MRI sequences and postoperative digital pathological images, extract MRI and digital pathological images, and build a preoperative diagnostic model of glioma based on deep learning to achieve independent prediction of preoperative MRI multiple Molecular markers.

Benefits of technology

It realizes non-invasive and accurate prediction of multimolecular markers before surgery, improves the reliability and accuracy of diagnosis, solves the problem of inability to obtain preoperative pathological images in clinical practice, and promotes the clinical transformation of imaging pathologic fusion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982535A_ABST
    Figure CN118982535A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of medical image processing, and particularly relates to a glioma preoperative diagnosis method based on digital pathological image guided magnetic resonance imaging. The glioma preoperative diagnosis method comprises the steps that a preoperative conventional magnetic resonance imaging sequence, a postoperative digital pathological image and various molecular marker tags of a glioma patient are obtained and arranged; carrying out image preprocessing, taking data with magnetic resonance imaging and digital pathological images as a training set, and taking patient data only with magnetic resonance imaging as a test set; extracting magnetic resonance imaging whole brain features and digital pathological image features, capturing an image-pathological feature relationship, and constructing a model; model performance is evaluated using prospective patient pre-operative magnetic resonance imaging data. According to the method, the preoperative magnetic resonance imaging can be formed to represent the postoperative pathological image feature map, the multi-molecular marker prediction based on the image pathology under the condition that no digital pathological image exists before the operation is achieved, intelligent and accurate diagnosis is achieved, and the problem that the glioma pathological image cannot be obtained before the operation during image pathology analysis in clinical practice is solved.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The invention belongs to the field of medical image processing, and in particular relates to a preoperative diagnosis method for glioma using magnetic resonance imaging guided by digital pathological images. [Background technology]

[0002] Glioma is one of the most common primary brain tumors. The significant differences in cell composition, gene expression and molecular characteristics in different regions of the tumor and its ability to spread rapidly and invade surrounding normal tissues lead to extremely poor prognosis, which seriously endangers human health. The clinical diagnosis of glioma mainly focuses on its classification, grading and molecular marker status. According to the fifth edition of the World Health Organization (WHO) Classification of Tumors of the Central Nervous System (WHO CNS5), molecular markers are of great significance in the study of the characteristics of glioma. These molecular changes are an important basis for determining the classification and grading of gliomas, and are also an important reference for diagnosis, treatment selection and prognosis evaluation. For example, O-6-methylguanine-DNA methyltransferase (MGMT) is still the only molecular marker for predicting the response to temozolomide treatment. The methylation status of the MGMT gene determines its expression level. Methylated MGMT gene will lead to downregulation of its expression, making tumor cells more sensitive to temozolomide. In clinical applications, temozolomide exerts its anti-tumor effect by spontaneously and rapidly degrading in vivo to generate active metabolites 5-(3-methyl-1-triazine)imidazole-4-carboxamide; Isocitrate dehydrogenase (IDH) gene is one of the important markers for distinguishing glioma subtypes, and the presence of IDH mutation can be used as a favorable prognostic marker. Telomerase reverse transcriptase (TERT), epidermal growth factor receptor (EGFR) and chromosomes 7 and 10 introduced by WHO CNS5 also provide important information for glioma diagnosis.

[0003] The detection of molecular markers through postoperative pathological tissue is the main way to diagnose gliomas and the gold standard for clinical diagnosis. However, the pathological tissue of gliomas can only be obtained through craniotomy. At present, the purpose of most craniotomies is to obtain tumor living tissue and evaluate and predict the treatment prognosis after detecting molecular information. At present, medical image analysis and recognition and pathological image analysis and recognition have benefited from the development of deep learning, and have had a positive impact on clinical decision-making in various diagnosis and treatment links before, during and after glioma surgery. However, there are very few deep learning models that can truly meet clinical transformation. The main reason is that some of the current research only uses deep learning technology for medical image analysis, and some are pure digital pathology calculations. Due to scale limitations, medical image analysis alone cannot fully reflect glioma tumor information, resulting in poor robustness of the model. In traditional computational pathology, breast cancer, lung cancer, etc. can be removed by puncture pathological tissue, which is less traumatic, while the pathological tissue of gliomas can only be obtained through craniotomy, resulting in the lag of such models. In addition, craniotomy is very traumatic and high-risk. Once postoperative complications occur, they will have a serious impact on the treatment prognosis. For patients with diffuse gliomas and tumors in poor locations, the condition may be further aggravated. In addition, the detection process of molecular markers is complicated, the detection cost is high, the application scenarios are limited, and it is difficult to popularize. Therefore, a method is needed to obtain these molecular marker information non-invasively, conveniently and accurately before surgery.

[0004] In the existing technology, a glioma molecular marker prediction model can be constructed through preoperative magnetic resonance imaging (MRI) image data, thereby promoting the clinical precision diagnosis of glioma. However, MRI cannot fully reflect the tumor microenvironment information. The prediction accuracy and robustness of the existing model are generally poor, and it is still difficult to achieve preoperative precision diagnosis and clinical transformation.

[0005] In the existing technology, imaging pathology omics based on multimodal fusion technology can also be used to construct a glioma molecular marker prediction model with high accuracy. However, paired MRI and digital pathology images are required for model training and practical applications. During preoperative prediction, digital pathology images paired with MRI cannot be obtained, which will cause the general multimodal imaging pathology omics model to fail.

[0006] In view of the above situation, clinical diagnosis urgently needs a technology that can non-invasively integrate macroscopic MRI information and microscopic pathological information before surgery and ensure the robustness of prediction. [Summary of the invention]

[0007] In order to solve the above problems, the present invention provides a preoperative diagnosis method for glioma using digital pathological images guided by magnetic resonance imaging.

[0008] The present invention provides a preoperative diagnosis method for glioma using digital pathological images guided by magnetic resonance imaging, and the technical solution is as follows:

[0009] The preoperative diagnosis method of glioma using digital pathology image-guided magnetic resonance imaging includes the following steps:

[0010] S1, obtain and organize 4 preoperative conventional MRI sequences, postoperative digital pathological images, and multiple molecular marker labels of glioma patients;

[0011] S2, preprocess the preoperative conventional MRI sequences and digital pathology images collected and sorted in S1 to obtain a data set, divide the patient data with both preoperative conventional MRI sequences and digital pathology images into a training set to train the model, and divide the data with only preoperative conventional MRI sequences into a test set to test the model performance;

[0012] S3, extract MRI whole brain image features and digital pathology image features, capture the nonlinear relationship between MRI whole brain image features and digital pathology image features, and use MRI whole brain image features to reconstruct digital pathology image features, build a digital pathology image-guided MRI preoperative diagnosis model for glioma, and realize preoperative MRI independent prediction of glioma multi-molecular markers. The model includes a feature extraction module and an image pathology feature fusion module;

[0013] S4, using the preprocessed data set obtained in S2, training and testing the digital pathology image-guided MRI glioma preoperative diagnosis model established in S3, to obtain the final digital pathology image-guided MRI glioma preoperative diagnosis model;

[0014] S5, inputting the multi-sequence MRI of prospective glioma patients into the digital pathology image-guided MRI preoperative diagnosis model of glioma obtained in step S4, and outputting the status of multiple molecular markers to obtain the prediction results of glioma multi-molecular markers, thereby assisting clinical accurate diagnosis;

[0015] In S2, N4 deviation correction, skull stripping, registration, and normalization were performed in sequence during the preprocessing of the conventional MRI sequence before surgery; tissue region segmentation and color correction were performed in sequence during the preprocessing of the digital pathology image.

[0016] Furthermore, the specific steps of N4 bias correction are as follows: first convert the MRI to the logarithmic domain, initialize the bias field to a field of all 1s, perform preliminary image estimation, and then perform non-parametric estimation on the low-frequency part, and fit the bias field by optimizing the B-spline low-frequency model; in the iterative optimization process, repeatedly optimize by updating the bias field estimate and the corrected image, and use the expectation maximization algorithm to ensure the uniformity of the intensity distribution; stop the iteration when the preset number of iterations or the change threshold is reached; finally, divide the original MRI image by the estimated bias field; skull stripping uses a deep learning model high-definition brain extraction tool to obtain a skull stripping image; registration is to register the skull stripping image to the T1 MNI1521mm standard template, and the registration uses the linear image registration tool of the functional magnetic resonance imaging software library. The interpolation method used for registration is trilinear interpolation, and the registration objective function is normalized mutual information; normalization uses Z-score normalization.

[0017] Furthermore, clustering constrained attention multi-instance learning is used for tissue region segmentation to automatically remove the background area of ​​digital pathology images, and a color normalization method is used for color correction.

[0018] Furthermore, in S3, feature extraction methods based on deep learning, such as CNN, self-supervised learning methods, and hybrid networks based on attention mechanisms, respectively extract MRI whole-brain image features and digital pathology image features, and when constructing a digital pathology image-guided MRI preoperative diagnosis model for glioma, feature extraction methods based on deep learning, such as CNN, self-supervised learning methods, and hybrid networks based on attention mechanisms, are used to extract semantic features from the reconstructed digital pathology image features and MRI whole-brain image features to predict multiple molecular markers.

[0019] Furthermore, in S3, feature extraction methods based on deep learning, such as CNN, self-supervised learning methods, and hybrid networks based on attention mechanisms, respectively extract MRI whole-brain image features and digital pathology image features, and encoders based on deep learning, such as CNN, self-supervised learning methods, and hybrid networks based on attention mechanisms, respectively extract semantic features from the reconstructed digital pathology image features and MRI whole-brain image features, and predict multiple molecular markers based on the extracted semantic features.

[0020] Furthermore, the feature extraction module includes feature extraction modules based on deep learning such as CNN, self-supervised learning methods, hybrid networks based on attention mechanisms, etc., which extract features from original whole-brain MRI and digital pathology images, and feature extraction modules based on deep learning such as CNN, self-supervised learning methods, hybrid networks based on attention mechanisms, etc.

[0021] Furthermore, the specific calculation process of the deep feature extraction module is to first learn the nonlinear relationship between MRI representation and digital pathological image representation through modal interactions such as contrastive learning and collaborative learning, then reconstruct the pathological features using the nonlinear relationship, and finally use the feature extractor to extract MRI deep features and reconstruct pathological features respectively.

[0022] Furthermore, in S3, modal interactions such as contrastive learning and collaborative learning are used to capture the nonlinear relationship between MRI whole-brain image features and digital pathology image features in the representation space of the two modalities.

[0023] Furthermore, in S4, model training uses modal interaction methods such as contrastive learning and collaborative learning to capture the nonlinear relationship between MRI whole-brain image features and digital pathology image features in the representation space of the two modalities, while MRI whole-brain images are used alone as model input during testing.

[0024] Furthermore, during the model training process, the Adam W optimizer was used to optimize the model parameters, and the warm up training strategy and one cycle learning rate decay were adopted. The loss function of the classification task was Focal Loss, and the probability values ​​of each molecular marker output by the model and the actual molecular marker labels were input into the loss function together. After repeated iterations, the model with the smallest loss value of the test set was taken as the final model.

[0025] Beneficial effects are as follows: the digital pathological image-guided MRI preoperative diagnosis method for glioma provided by the present invention effectively captures global multi-scale information through feature extraction methods based on deep learning such as CNN, self-supervised learning methods, hybrid networks based on attention mechanisms, etc. as encoders, and extracts multi-sequence MRI whole-brain image features and digital pathological image features of glioma patients. The nonlinear correlation information between the modalities is automatically learned in the representation space of the two modalities through modal interactions such as contrast learning and collaborative learning. Once these nonlinear correlation information are learned, a feature map of postoperative pathological images represented by preoperative MRI can be formed, which can realize the prediction of multi-molecular markers in the absence of digital pathological images before surgery, and solves the problem of being unable to obtain glioma pathological images before surgery in clinical practice, thereby promoting the method based on the fusion of image pathology to achieve clinical transformation. The present invention has high reliability, good accuracy, good practicality and can promote clinical transformation.

Brief Description of the Drawings

[0026] Figure 1 It is a flow chart of the preoperative diagnosis method of glioma using digital pathological image-guided magnetic resonance imaging (MRI) provided by the present invention;

[0027] Figure 2It is a framework diagram of the preoperative MRI independent prediction of glioma multi-molecular marker model provided by the present invention;

[0028] Figure 3 It is a schematic diagram of the structure of the spatial priors vision transformer (SP-ViT) feature extraction module provided by the present invention;

[0029] Figure 4 It is a schematic diagram of the image-pathology fusion structure provided by the present invention. [Specific implementation method]

[0030] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0031] A specific embodiment of a digital pathological image-guided magnetic resonance imaging (MRI) preoperative diagnosis method for glioma provided by the present invention is as follows:

[0032] The digital pathology image-guided magnetic resonance imaging method for preoperative diagnosis of glioma provided in this embodiment is as follows: Figure 1 , including the following steps:

[0033] S1, obtain and organize 4 preoperative conventional MRI sequences, postoperative digital pathology images and multiple molecular marker labels of glioma patients.

[0034] Among them, MRI includes T1-weighted imaging, T2-weighted imaging, enhanced T1-weighted imaging and fluid-attenuated inversion recovery imaging (FLAIR), and molecular markers include O6-Methylguanine-DNA Methyltransferase (MGMT), Isocitrate Dehydrogenase (IDH), Telomerase Reverse Transcriptase (TERT), Epidermal Growth Factor Receptor (EGFR) and +7 / -10;

[0035] S2, preprocesses the MRI and digital pathology images collected and organized in S1 to obtain patient data sets, divides the patient data with both MRI and digital pathology images into a training set to train the model, and divides the patient data with only MRI into a test set to test the model performance.

[0036] The MRI preprocessing process includes N4 deviation correction, skull stripping, registration and normalization.

[0037] N4 bias correction is based on nonparametric estimation and smooth regularization. The specific steps are: first convert MRI to the logarithmic domain, initialize the bias field to a field of all 1s, perform preliminary image estimation, and then perform nonparametric estimation on the low-frequency part, and fit the bias field by optimizing the B-spline low-frequency model; in the iterative optimization process, repeatedly optimize by updating the bias field estimate and correcting the image, and use the expectation maximization algorithm to ensure the uniformity of the intensity distribution; stop the iteration when the preset number of iterations or change threshold is reached. The intensity deviation and non-uniformity in the image can be effectively reduced through N4 bias correction.

[0038] The N4 bias-corrected images are then skull-stripped, which allows for a clearer view of the glioma and its surrounding tissue structures, while removing non-brain tissues. Skull-stripping uses a deep learning model high-definition brain extraction tool.

[0039] The registration is to register the skull-stripped images to the T1 MNI1521mm standard template. The linear image registration tool of the functional magnetic resonance imaging software library is used for registration. The interpolation method is trilinear difference, and the registration objective function is normalized mutual information.

[0040] Normalization is to perform Z-score normalization on the registered images, which is used to standardize the values ​​of different features or variables so that they have zero mean and unit variance, eliminating the scale differences between features. The principle of normalization is based on the Z-score in statistics, and the formula is expressed as:

[0041]

[0042] Where Z is the normalized grayscale value, x is the original grayscale value, μ is the mean grayscale value of the image, and σ is the standard deviation of the grayscale value of the image.

[0043] In the preprocessing of digital pathology images, tissue region segmentation and color correction are performed in sequence.

[0044] Tissue region segmentation is to extract the tissue region of interest from the background. In this embodiment, a clustering constrained attention multi-instance learning method is used to automatically segment the tissue region of the digital pathology image.

[0045] Color correction can effectively reduce the differences caused by scanning equipment, operators, dyeing and lighting. In this embodiment, a color standardization method that retains structural information is used for color correction, thereby effectively reducing the color differences between samples to improve the quality of subsequent feature extraction.

[0046] S3, extract MRI whole-brain features and digital pathology digital pathology image features, capture the relationship between image and digital pathology features, and build a digital pathology image-guided MRI preoperative diagnosis model for glioma.

[0047] The image classification model ViT (Vision Transformer) based on the Transformer architecture in this embodiment can also use feature extraction methods based on deep learning such as CNN, self-supervised learning methods and other feature extractors to extract MRI whole-brain image features and digital pathology image features, and use modal interaction methods such as contrast learning and collaborative learning to capture the relationship between image and digital pathology features, output image features and digital pathology reconstruction features, and then use feature extraction methods based on deep learning such as CNN, self-supervised learning methods, hybrid networks based on attention mechanisms, etc. to extract semantic features from the reconstructed digital pathology image features and MRI whole-brain image features to predict multiple molecular markers.

[0048] CNN-based feature extraction involves the following computational processes:

[0049] Convolution operation:

[0050]

[0051] The formula indicates that the convolution operation is performed at position (i, j), and features are extracted by performing dot multiplication and summation of the sliding window and the local area. In the formula, f(m, n) represents the pixel value or feature map of the input image, g(i, j) represents the weight of the convolution kernel, and (i, j) represents the position coordinates of the convolution result.

[0052] ReLU activation function:

[0053] ReLU(x)=max(0,x)

[0054] x represents the output after the convolution operation, and the ReLU function is used to introduce nonlinearity, so that the model can learn more complex features.

[0055] Max Pooling:

[0056] p(i,j)=max m,n {x(i+m,j+n)}

[0057] Average Pooling:

[0058]

[0059] Where x(i+m,j+n) represents the pixel value of the input feature map. R represents the size of the pooling window, and p(i,j) represents the feature value after pooling.

[0060] Self-supervised learning feature extraction involves the following computational processes:

[0061] Contrastive Loss:

[0062]

[0063] In the formula, sim(z i ,z j ) represents the embedding vector z i and z j The cosine similarity between τ represents the temperature parameter, which is used to control the smoothness of the distribution. [k≠i] Represents the indicator function, which is used to ignore self-comparison, L i,j Represents the loss function value, which is used to optimize the model so that the positive sample pairs are closer and the negative sample pairs are farther away.

[0064] In this embodiment, the digital pathology image-guided MRI glioma preoperative diagnosis model includes a feature extraction module and an image pathology feature fusion module. Figure 2 This is the architecture diagram of the preoperative MRI independent prediction of glioma multi-molecular markers model. The specific steps are: first, use the attention mechanism-based hybrid network SP-ViT (spatial prior Vision Transformer) to extract MRI whole-brain image features and digital pathology image features respectively, then use contrastive learning, collaborative learning and other modal interactions to capture the relationship between image and pathology features, output image features and digital pathology reconstruction features, and then use a 12-layer ViT encoder to obtain deeper semantic features from image features and digital pathology reconstruction features. Finally, predict the status of multi-molecular markers based on the extracted semantic features.

[0065] The feature extraction module includes the SP-ViT feature extraction module for extracting features from the original whole-brain MRI and digital pathology images and the ViT-based deep network feature extraction module. The output dimension of each layer of the ViT-based deep network feature extraction module is 768.

[0066] The SP-ViT feature extraction module replaces the self-attention in ViT with SP-SA (spatial prior enhanced self-attention), adds relative coordinates to the self-attention mechanism, and highlights spatial relationship groups such as relative position relationship, scale relationship, and direction relationship, which helps to analyze the entire medical image. Figure 3 Schematic diagram of the SP-ViT feature extraction module structure. The SP-ViT feature extraction module includes an SP-SA layer and an MLP (multi-layer perceptron) layer.

[0067] The processing process of the SP-SA layer is to add the relative coordinates of the medical image patch to the attention mechanism, highlighting the spatial relationship groups such as relative position relationship, scale relationship, and direction relationship, which helps to capture the relative spatial position information of the medical image and improve the prediction accuracy of downstream tasks. The specific process of the SP-SA layer is to first divide the medical image into an input sequence of patches. And input to the SP-SA layer, and output a new sequence of the same length The nonlinear relationship between X and Y is Where n represents the number of patches in the input sequence, A ij represents the attention weight of the i-th query vector to the j-th key vector, T is the transposed symbol, and W V is the projection matrix of the value matrix V. In addition, the present invention uses the spatial prior knowledge Ω ij The attention weights calculated for learning the query matrix Q and the key matrix K are then ij The formula is:

[0068]

[0069] In the formula, e ij Represents the attention score matrix of each Patch, Ω ij is the spatial prior knowledge, k is an ergodic variable, and all possible k are summed in the denominator to normalize the weights. ij The formula is:

[0070]

[0071] In the formula, and is the transposed input feature vector, T is the transposed symbol, and W Q is the projection matrix of the query matrix Q, W K is the projection matrix of the key matrix K, d k is the dimension of the key matrix K.

[0072] Ω ij The specific operation is: take the position of the query matrix Q as the reference point, and get the relative coordinates of the key matrix K Represents the set of real numbers, using a shared mapping f for all query matrices Q and key matrices K p , the formula of spatial prior knowledge is expressed as The purpose of spatial prior knowledge is to enable the model to automatically learn the required inductive bias, the shared mapping f p The MLP layer is used to map the relative coordinates, and the formula is expressed as A self-attention layer consists of multiple attention mechanisms, expressed as Head (attention head) represents capturing different feature relationships through different parameter matrices, and N represents the number of heads. It means that all the heads of the calculations are cascaded, and finally the final output is obtained through the linear transformation function φ. The processing process of the MLP layer is expressed by the following formula:

[0073] MLP(x)=FFN(x)=Linear2(GELU(Linear1(x)))

[0074] In the formula, x is the input feature, FFN is the feedforward neural network, the linear layer Linear1 expands the channel output by the multi-head attention mechanism by 4 times, the Gaussian error linear unit processing function GELU() performs nonlinear transformation on the feature, and then the linear layer Linear2 scales the channel to its original size. The formula of GELU() is expressed as:

[0075]

[0076] In the formula, x is the input feature, erf is the Gaussian error function, and the formula is expressed as:

[0077]

[0078] In the formula, t is the integral variable, x is the input feature, and the SP-ViT feature extraction module formula is expressed as:

[0079]

[0080] In the formula, is the output result of the SP-ViT layer, MLP is a multi-layer perceptron, LayerNorm is a layer normalization function, MultiAttn() represents a multi-head attention mechanism, and x is an input feature.

[0081] The difference between the ViT deep network feature extraction module and the SP-ViT feature extraction module is that there is no embedding space prior when calculating the attention weight. The attention weight B of the i-th query matrix to the j-th key matrix in the ViT deep network feature extraction module is ij The formula is:

[0082]

[0083] In the formula, k is the ergodic variable, and all possible k are summed in the denominator to normalize the weight. ij The formula is:

[0084]

[0085] B ijA in SP-ViT ij The attention score matrix e of each patch is ij Obtained by applying SoftMax operation.

[0086] Modality interaction captures the nonlinear relationship between images and pathologies in the representation space, e.g. Figure 4 Figure 1 is a schematic diagram of the image-pathology fusion structure, which can effectively use MRI features to represent digital pathology features. The specific process is to first obtain the output features of the last SP-ViT layer, and the whole brain image features extracted from MRI are recorded as the representation R i , the digital pathology image features mined from digital pathology images are denoted as representation P i , i represents the sample index. Then, these features are used to learn the nonlinear relationship between image and pathological features through modal interactions such as contrast learning and collaborative learning. Finally, this nonlinear relationship is run to reconstruct pathological features. The modal interaction captures the nonlinear relationship of features and is expressed as:

[0087] H(Z i )=GELU(WR i +VP i +b)

[0088] Where W and V are the learnable parameter matrices of MRI whole brain image features and digital pathology image features, b is the bias matrix, and Z i ′ represents the reconstruction of digital pathological features, and the formula is expressed as:

[0089] Z i ′=GELU([W′H(Z i ),V′H(Z i )]+b′)

[0090] Where W′, V′ and b′ are the image feature weight matrix, the reconstructed pathology feature weight matrix and the bias matrix, respectively.

[0091] S4, training and testing the established digital pathology image-guided MRI preoperative diagnosis model for glioma to obtain the final digital pathology image-guided MRI preoperative diagnosis model for glioma;

[0092] Among them, the model training adopts modal interaction methods such as contrastive learning and collaborative learning to capture nonlinear relationships in the representation space of the two modalities. When testing the model, this nonlinear relationship is used to infer the pathological features, that is, only the image is used as input to obtain a robust and generalizable digital pathology image-guided MRI glioma preoperative diagnosis model.

[0093] The model training process includes extracting MRI whole-brain image features and digital pathological image features through SP-ViT, and obtaining MRI features and reconstructing pathological features through modality interaction. The following objective function is used to maximize the correlation error between the potential features of the two modalities of imaging and pathology, and minimize the feature error between the two modalities of imaging and pathology, so as to better capture the nonlinear relationship between the two modalities. The objective function L Infer The formula is:

[0094] L Infer =L recon -λL corr

[0095] Where, L recon is the reconstruction error, λ is a hyperparameter used to control the trade-off between different terms in the loss function, and L corr represents the correlation error, L recon The formula is:

[0096]

[0097] In the formula, Z i =[H(R i ),H(P i )] is the latent representation of MRI whole brain image features and digital pathological image features extracted by SP-ViT, Z i ′ represents the reconstructed digital pathological features, N is the number of samples, L(Z i ,Z i ′) is the reconstruction loss of a single sample, L corr The formula is:

[0098]

[0099] Where N is the number of samples, i is the index of the sample, ranging from 1 to N, indicating the i-th sample currently being processed. i ) represents the MRI features extracted by the SP-ViT encoder, H(P i ) represents the digital pathology image features extracted by the SP-ViT encoder, and Represent the average values ​​of imaging features and pathological features, respectively.

[0100] The prediction results of each molecular marker output by the model and the actual molecular marker status are input into the loss function, and the model is optimized by repeatedly iteratively minimizing the loss function. The loss function Focal Loss of the classification task is expressed as follows:

[0101] FocalLoss = -α(1-p t )γ log(p t )

[0102] In the formula, α is the balance factor, which is used to balance the imbalance of positive and negative samples. It is set according to the ratio of positive and negative samples in the data set. α and γ are two adjustable hyperparameters. t is the predicted probability output by the model. The total loss of the model is L Total The formula is:

[0103]

[0104] Where, L Infer is the objective function for reconstructing pathological features, K represents K molecular marker classification tasks, l represents the loss of each classification task, σ Infer and σ are the learnable uncertainty task weights in the network, and i is the index of the sample. During training, σ is first initialized to 1 and adaptively updated during the model optimization phase.

[0105] During the training process, the Adam W optimizer is used to optimize the parameters of the model. The Adam W optimizer is an optimizer that combines Adam with the weight decay strategy. It also uses the Warm up training strategy and the One cycle learning rate decay. In this embodiment, the first three epochs are the Warm up stage. The initial momentum in this stage is set to 0.8. During normal training, the initial value of the learning rate is set to 0.0004, the weight decay is set to 0.001, the momentum is 0.9, and the total epoch during training is set to 300. The formula for One cycle learning rate decay is:

[0106]

[0107] In the formula, LR epoch is the learning rate size of the current epoch, base lr is the initial learning rate, epoch is the current training round, num_epoch is the total training round, the value of y1 determines the learning rate at the last epoch, and y2 is set to 1. When the model has the lowest loss in the test set, the model performs best at this time, and the model weight is saved as the optimal model weight for preoperative diagnosis of gliomas guided by digital pathology images in MRI.

[0108] S5, input the multi-sequence MRI of prospective glioma patients into the digital pathology image-guided MRI preoperative diagnosis model of glioma obtained in step S4, and output the status of multiple molecular markers to obtain the glioma multi-molecular marker prediction results, thereby assisting clinical diagnosis.

[0109] In order to illustrate the effectiveness of the method of the present invention, a private data set of 100 cases with both preoperative MRI and postoperative digital pathology was used to establish a model. The experimental results obtained in the task of predicting the MGMT methylation status of 40 prospective samples are shown in the following table. From the results, it can be seen that each evaluation index of the method of the present invention is better than that of the MRI and digital pathology single-modality models.

[0110]

[0111]

[0112] Note: MRI-magnetic resonance imaging, AUC-area under the receiver operating characteristic curve.

[0113] The digital pathological image-guided magnetic resonance imaging method for preoperative diagnosis of glioma provided by the present invention effectively captures global multi-scale information through ViT as an encoder, and extracts multi-sequence MRI whole-brain image features and digital pathological image features of glioma patients. The nonlinear correlation information between the modalities is automatically learned in the representation space of the two modalities through modal interactions such as contrast learning and collaborative learning. Once these nonlinear correlation information are learned, a preoperative MRI representation of postoperative pathological image feature maps can be formed, which can realize the prediction of multiple molecular markers in the absence of digital pathological images before surgery, and solves the problem of being unable to obtain glioma pathological images before surgery in clinical practice, thereby promoting the clinical transformation of the method based on the fusion of image pathology omics. The present invention has high reliability, good accuracy, good practicality and can promote clinical transformation.

Claims

1. A digital pathology image-guided magnetic resonance imaging method for preoperative diagnosis of glioma, characterized in that: The steps include: S1, obtain and organize 4 preoperative conventional MRI sequences, postoperative digital pathological images, and multiple molecular marker labels of glioma patients; S2, preprocess the preoperative conventional MRI sequences and digital pathology images collected and sorted in S1 to obtain a data set, divide the patient data with both preoperative conventional MRI sequences and digital pathology images into a training set to train the model, and divide the data with only preoperative conventional MRI sequences into a test set to test the model performance; S3, construct a preoperative diagnosis model for glioma guided by digital pathology images, which includes a feature extraction module and an image pathology feature fusion module. By capturing the nonlinear relationship between MRI whole-brain image features and digital pathology image features, preoperative MRI can independently predict multiple molecular markers of glioma. S4, using the preprocessed data set obtained in S2, training and testing the digital pathology image-guided MRI glioma preoperative diagnosis model established in S3, to obtain the final digital pathology image-guided MRI glioma preoperative diagnosis model; S5, inputting the multi-sequence MRI of prospective glioma patients into the digital pathology image-guided MRI preoperative diagnosis model of glioma obtained in step S4, and outputting the status of multiple molecular markers to obtain the prediction results of glioma multi-molecular markers, thereby assisting clinical accurate diagnosis; In S2, N4 deviation correction, skull stripping, registration, and normalization were performed in sequence during the preprocessing of the conventional MRI sequence before surgery; tissue region segmentation and color correction were performed in sequence during the preprocessing of the digital pathology image.

2. The method for preoperative diagnosis of glioma using digital pathological image-guided magnetic resonance imaging according to claim 1, characterized in that: The specific steps of N4 bias correction are: first convert MRI to the logarithmic domain, initialize the bias field to a field of all 1s, perform preliminary image estimation, and then perform non-parametric estimation of the low-frequency part to fit the bias field by optimizing the B-spline low-frequency model; During the iterative optimization process, optimization was repeated by updating the bias field estimate and the correction image, and the expectation maximization algorithm was used to ensure the uniformity of the intensity distribution. The iteration was stopped when the preset number of iterations or the change threshold was reached. The skull stripping image was obtained by using the deep learning model high-definition brain extraction tool. The skull stripping image was registered to the T1 MNI1521mm standard template using the linear image registration tool of the functional magnetic resonance imaging software library. The interpolation method used for registration was trilinear interpolation, and the registration objective function was normalized mutual information. Z-score normalization was used for normalization.

3. The digital pathology image-guided magnetic resonance imaging method for preoperative diagnosis of glioma is characterized by: In S3, clustering constrained attention multi-instance learning is used for tissue region segmentation to automatically remove the background area of ​​digital pathology images, and color normalization method is used for color correction.

4. The digital pathology image-guided magnetic resonance imaging method for preoperative diagnosis of glioma is characterized by: The extraction module includes a feature extraction module for extracting features from the original whole-brain MRI and digital pathology images and a feature extraction module for extracting features from the reconstructed digital pathology image features and MRI whole-brain image features. The specific calculation process of the feature extraction module for extracting features from the reconstructed digital pathology image features and MRI whole-brain image features is as follows: first, the nonlinear relationship between the MRI representation and the digital pathology image pathological representation is learned through modal interactions such as contrast learning and collaborative learning; the pathological features are reconstructed using the learned nonlinear relationship; and feature extraction methods based on deep learning, such as CNN, self-supervised learning methods, and feature extractors based on a hybrid network of an attention mechanism, are used to extract MRI deep features and reconstructed pathological features, respectively.

5. The digital pathology image-guided magnetic resonance imaging method for preoperative diagnosis of glioma is characterized in that, in S3, feature extraction methods based on deep learning such as convolutional neural networks (CNN), self-supervised learning methods, hybrid networks based on attention mechanisms, etc. are used to extract MRI whole-brain image features and digital pathology image features respectively; the relationship between image and digital pathology features is captured by modal interaction methods such as contrast learning and collaborative learning, and image features and digital pathology reconstruction features are output; feature extraction methods based on deep learning such as CNN, self-supervised learning methods, hybrid networks based on attention mechanisms, etc. are also used to extract semantic features from reconstructed digital pathology image features and MRI whole-brain image features; and clinical accurate diagnosis is achieved based on the extracted semantic features.

6. The digital pathology image-guided magnetic resonance imaging method for preoperative diagnosis of glioma is characterized by: In S4, model training uses modal interaction methods such as contrastive learning and collaborative learning to capture the nonlinear relationship between MRI whole-brain image features and digital pathology image features in the representation space of the two modalities, while MRI whole-brain images are used alone as model input during testing.

7. The digital pathology image-guided magnetic resonance imaging method for preoperative diagnosis of glioma is characterized in that During the model training process, the Adam W optimizer was used to optimize the model parameters, and the warm up training strategy and one-cycle learning rate decay were adopted. The loss function of the classification task was Focal Loss. The probability values ​​of each molecular marker output by the model and the actual molecular marker labels were input into the loss function together. After repeated iterations, the model with the smallest loss value of the test set was taken as the final model.

Citation Information

Patent Citations

  • Method for generating pathological image features by dynamically enhancing magnetic resonance image features

    CN116958094A

  • Multi-scale feature-based triple negative breast cancer immunophenotype prediction method and system

    CN117152506A

  • GBM patient prognosis evaluation method and system based on magnetic resonance-pathological image fusion

    CN118507057A

  • Systems and methods for deep orthogonal fusion for multimodal prognostic biomarker discovery

    EP4239647A1

  • Generating combined diagnostic imaging and pathology images

    WO2024121190A1

Cited By

  • CSF1R prediction method in brain glioma, storage medium and device

    CN120340839A

  • A method, storage medium and device for predicting CSF1R in brain glioma

    CN120340839B

  • Whole-brain voxel level-based double-encoder comparison method, device and program product

    CN120997590A