Tumor diagnosis method and system for generating molecular spatial distribution map based on MRI

Abstract

The invention discloses a tumor diagnosis method and system for generating a molecular spatial distribution diagram based on MRI (Magnetic Resonance Imaging), and belongs to the technical field of image data processing. The method comprises the following steps: constructing an MRI-pathological local matching data set; the Transform and the GAN are combined to construct a local MRI (Magnetic Resonance Imaging) to generate a small pathological graph model; and constructing and verifying a molecular spatial distribution diagram. According to the method, the histopathology is taken as a bridge, a two-layer mapping relation is constructed between the MRI and the molecular features, the global molecular features are estimated through local matching data, existing clinical resources are effectively utilized, and three-dimensional space distribution evaluation of the molecular features is indirectly achieved at low cost. And the constructed model can be used for pathological diagnosis in any region on the brain MRI sequence, so that the potential invasion region of the tumor can be explored, and reference is provided for the surgical resection range definition and treatment strategy planning of the glioma.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a tumor diagnosis method and system for generating a molecular spatial distribution map based on MRI. Background Art

[0002] Glioma is the most common primary brain tumor. According to the WHO standard, molecular pathological characteristics are used as the gold standard for glioma diagnosis, and glioma is graded in combination with histopathological morphological characteristics. The new standard allows similar treatment plans to be formulated for subtypes with similar biological characteristics to enhance efficacy and improve patient prognosis. However, the current pathological examination method still has defects.

[0003] First, molecular characteristics are obtained through pathological examination of surgically removed tumors, and molecular characteristics can only be obtained about one week after surgery, so the timeliness of diagnosis is poor; non-invasive diagnosis cannot be performed before surgery, which limits the planning of early precision treatment plans for gliomas. Secondly, gliomas have obvious intratumoral spatial heterogeneity, and there are large differences in the spatial distribution of molecular typing and microenvironment characteristics of individual tumors. Pathological examinations are performed on local areas of the tumor, which can only evaluate the local molecular characteristics of the tumor and cannot reflect the overall molecular spatial distribution of the tumor. Therefore, how to evaluate the distribution of its molecular characteristics from the overall space of the tumor before surgery is a key issue that needs to be solved in the precise diagnosis of gliomas.

[0004] Magnetic resonance imaging (MRI) can present the spatial distribution of gliomas and their adjacent areas non-invasively, and is an important tool for preoperative diagnosis of gliomas. Previous studies have found that the development of new imaging technologies based on MRI can map the microstructure of gliomas in the same dimension as MRI and improve the diagnostic effect of molecular classification. Artificial intelligence is currently the most effective technology in the field of medical image processing, and its performance in imaging and image post-processing is superior to traditional algorithms. Among them, generative artificial intelligence has a strong ability to create content, and can improve the information dimension through learning. It has excellent performance in MRI super-resolution, MRI-based cross-modal image generation, pathological image generation and segmentation. Through generative artificial intelligence technology, key MRI information related to glioma molecular classification is characterized, and semantic information of molecular features is generated in the MRI dimension, which is expected to provide quantitative analysis and visualization tools for preoperative evaluation of the spatial distribution of glioma molecules, and provide guidance for the selection of pathological sampling sites and accurate diagnosis of gliomas.

[0005] However, existing MRI and artificial intelligence technologies have the following shortcomings in tumor diagnosis: 1. The existing MRI-based molecular diagnosis model only learns the mapping relationship between the macroscopic information of tumors on MRI and the molecular features. The information difference between the two is large, and the diagnostic accuracy needs to be improved.

[0006] 2. The existing scheme for predicting local molecular classification of glioma based on global tumor information of MRI does not spatially correspond the information of two dimensions, MRI and pathology, and lacks reliability and clinical interpretability.

[0007] 3. Artificial intelligence algorithms can identify tumor microenvironment information related to molecular typing in histopathological images, greatly improving the diagnostic efficiency and accuracy of gliomas, and providing guidance for the accurate diagnosis of gliomas. However, due to the limitations of tissue sampling, artificial intelligence research based on histopathological images can only predict local molecular typing and cannot evaluate the molecular spatial distribution of the entire tumor. Summary of the invention

[0008] In view of the difficulty of evaluating the spatial distribution of molecules before surgery in the accurate diagnosis of glioma in the prior art, the present invention provides a tumor diagnosis method and system based on MRI to generate a molecular spatial distribution map. The key technical problem of fine-grained cross-modal image generation is solved by Transformer and GAN, and the molecular spatial distribution map of glioma is constructed by generating tissue pathology map as a bridge, and verified in a prospective data set. This technology is expected to provide clinical guidance for the preoperative molecular pathology evaluation of glioma and the selection of pathological tissue sampling areas, while improving the accuracy of molecular diagnosis of glioma.

[0009] The technical solution proposed in the present invention is a method for generating a molecular spatial distribution map based on preoperative MRI. Using tissue pathology as a bridge, the mapping relationship between MRI and molecular pathology is learned through local matching data of MRI and pathology, and the molecular spatial distribution map of the entire tumor on MRI is evaluated based on this mapping relationship.

[0010] In order to achieve the above technical objectives, the technical solution implemented by the present invention is: The first aspect of the present invention provides a tumor diagnosis method based on MRI to generate a molecular spatial distribution map, comprising the following steps: S1, Construction of MRI-pathology local matching dataset; The data sources of the dataset are MRI sequences, histopathological images and molecular pathological test results of tumor patients; including prospective data of recording the coordinates of the center point of tissue sampling through intraoperative navigation, as well as retrospective data; local spatial matching of the MRI local area at the tissue sampling site with the histopathological images and molecular features.

[0011] In the prospective dataset, the local MRI containing the center point of tissue sampling was cropped out to form an accurate matching dataset with the pathology; in the retrospective dataset, the possible areas of tissue sampling were outlined through clinical records and expert judgment to construct a fuzzy matching dataset.

[0012] The tissue pathology images are divided into several small pathology images to form a small pathology image dataset for the next step of pathology diagnosis model training. At the same time, the MRI of the tissue sampling site is cropped into several local areas by sliding a sliding window voxel by voxel. The possibility of generating high-resolution molecular spatial distribution maps is explored while expanding the dataset, and the influence of the sliding window size on the generation effect is evaluated in the modeling.

[0013] S2, construction and validation of “MRI-histopathology” and “histopathology-molecular features” mapping relationships; The pathological sub-images obtained by cropping the tissue pathology images in S1 are used to train the pathological diagnosis model with molecular features as the prediction target, and the mapping relationship between tissue pathology and molecular features is constructed; the tissue pathology images are cut into several pathological sub-images, and the pathological diagnosis model is trained based on a weakly supervised method to predict the molecular typing and grading of the tumor, and the model is used to screen out pathological sub-images with a high probability of correct diagnosis as the gold standard for MRI-generated pathological images; The MRI is cropped, and the multi-sequence features of the cropped local MRI area are extracted as the input of the GAN model. The information interaction relationship between the MRI sequences is learned through the self-attention mechanism of the Transformer generator to generate pathological thumbnails and construct a mapping relationship between MRI and tissue pathology. In the embodiment of the present invention, the MRI is cropped by a sliding window operation.

[0014] The GAN model uses a dual discriminator structure. One discriminator is a pathology diagnosis model, which is used to determine whether the generated pathology image has the diagnostic ability of the real image. The other discriminator is used to identify the authenticity of the generated pathology image. Both use the Transformer encoder as the core network structure. The generator and the authenticity discriminator are gradually optimized through alternating training. During the training process, the local MRI area and the pathology small image are randomly matched to expand the training sample size.

[0015] The local regions of MRI in multiple sequences are converted into feature matrices, and the self-attention mechanism of Transformer is used to mine the structural interaction information between sequences for image generation, and the input information is refined by alternating up and down sampling.

[0016] S3, constructs a full-volume molecular spatial distribution map with the same resolution as MRI: The global MRI is cropped into sub-regions with the same input size as the GAN model, and histopathology maps are generated respectively. The pathology diagnosis model is used to predict the molecular features of the histopathology maps, and finally they are combined into a full-volume molecular spatial distribution map.

[0017] The MRI area to be diagnosed is divided into sub-areas voxel by voxel using a sliding window, and a tissue pathology map is generated for each sub-area, and the molecular typing and grading of each tissue pathology map are obtained through a pathological diagnosis model. The diagnostic results are mapped onto the MRI according to the coordinates of the center point of the MRI sub-area, which is the molecular spatial distribution map constructed by the present invention. The map can characterize the tumor tissue grade and molecular spatial feature distribution at the same resolution as MRI.

[0018] In one embodiment of the present invention, the tumor is a brain tumor, and further, the tumor is a glioma.

[0019] Based on MRI-pathology local matching data set, the present invention first builds a model to learn the mapping relationship between local MRI region and pathology, and then the global MRI is subjected to sub-region cropping according to the same input size, and the molecular features are predicted respectively, and finally combined into a full-volume molecular space distribution map. The size of the local MRI region will affect the resolution and accuracy of the molecular space distribution map. In order to construct a molecular space distribution map with the same resolution as MRI, the most direct method is to set the MRI region size of the model input to a single voxel, but this will cause difficulties in modeling, and it is also impossible to analyze the relationship between it and the surrounding voxels, and the generated results lack reliability, and abnormal values ​​are prone to occur. Expanding the MRI region size can reduce the difficulty of modeling and improve the reliability of the results, but it will cause the resolution of the molecular space distribution map to decrease. Therefore, how to construct a high-precision full-volume molecular space distribution map with the same resolution as MRI is a key problem to be solved by the present invention.

[0020] To solve this problem, the present invention uses sliding window operation to expand the size of the analysis area and the sample size, and cuts the sub-area by the sliding step of a single voxel, so as to maintain the same generation resolution as MRI. At the same time, using tissue pathology as a bridge, the diagnostic information is integrated and refined by first increasing and then decreasing the amount of information, improving the molecular prediction accuracy under small-size sliding windows and reducing the model's requirements for sliding window size.

[0021] Since it is impossible to perform pathological examination of the entire tumor volume for glioma, the present invention learns the mapping relationship between local MRI and pathology based on the MRI-pathology local matching data set. In view of the limitation of the amount of local MRI information and the potential of tissue pathology in the molecular typing and grading of glioma, the present invention first generates a tissue pathology image based on local MRI, expands the local MRI information, and then uses the generated tissue pathology image to predict the molecular typing. The input of the generative model is a small-sized MRI local area corresponding to the pathological sampling area, and the traditional convolutional neural network model is difficult to effectively capture image features; the generated target tissue pathology image is a high-throughput microstructure imaging under a high-power microscope, which contains rich image information. How to map a small amount of local MRI information with a large amount of tissue image information is a key issue of the present invention.

[0022] The present invention realizes cross-modal generation between small-size local MRI and high-throughput tissue pathology images in the following manner: the present invention converts the local regions of MRI of multiple sequences into feature matrices, uses the self-attention mechanism of Transformer to mine the structural interaction information between sequences for image generation, and uses alternating up and down sampling to refine the input information. At the same time, the tissue pathology image is divided into several pathology sub-images, and a pathology diagnosis model that distinguishes the molecular typing and grading of glioma is trained in a weakly supervised manner, and then the pathology sub-images with a higher probability of correct diagnosis are screened out as the gold standard for the generation task, thereby reducing the amount of tissue pathology information. The pathology diagnosis model also serves as one of the discriminators of the generation model to supervise the image generation effect in terms of diagnostic ability.

[0023] The second aspect of the present invention provides a tumor diagnosis system for generating a molecular spatial distribution map based on MRI, comprising: The data acquisition module is used to collect MRI sequences, histopathology images, and molecular pathology test results of tumor patients, and record the coordinates of the center point of pathological tissue sampling in the prospective data set, while confirming the potential sampling location in the retrospective data set; Pathology diagnosis model building module, used to predict the molecular type and grade of pathology images; based on the pathology images, a pathology diagnosis model is built using a weak supervision strategy and Transformer network structure; MRI-pathology matching dataset construction module, used to crop MRI, perform spatial matching with pathology data, and construct MRI-pathology exact matching dataset and MRI-pathology fuzzy matching dataset; The pathological thumbnail generation module is used to build a TransformerGAN model to generate pathological thumbnails from local MRI regions. The TransformerGAN model includes a Transformer generator and two discriminators that are also based on the Transformer network structure. The molecular space distribution map construction module is used to generate a tissue pathology map for each MRI sub-region, and use the pathology diagnosis model to output the diagnosis result of each sub-region, and map the diagnosis result to the MRI according to the coordinates to generate a molecular space distribution map; wherein the diagnosis result is the differential diagnosis made by the pathology diagnosis model for molecular typing and grading.

[0024] The third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps of the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in the present invention.

[0025] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in the present invention are implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention provides a tumor diagnosis method based on MRI to generate a molecular spatial distribution map, which enables the diagnosis result to be advanced from the postoperative stage of the traditional method to the preoperative stage, improves the timeliness of molecular diagnosis, and facilitates the planning of surgical plans; (2) The present invention can evaluate the global situation of the tumor. The present invention proposes a method for generating a molecular feature distribution map based on MRI, which estimates the global molecular features with local matching data, effectively utilizes existing clinical resources, and indirectly realizes the three-dimensional spatial distribution evaluation of molecular features at a low cost. The model can perform pathological diagnosis in any area on the brain MRI sequence, which helps to discover the potential invasion area of ​​the tumor and provide a reference for the definition of the surgical resection range and treatment strategy planning of glioma.

[0027] (3) In terms of method, the present invention uses tissue pathology as a bridge and adopts the Transformer deep learning algorithm to construct a two-layer mapping relationship between MRI and molecular features, gradually narrowing the information difference between the two and gradually improving the accuracy of glioma diagnosis.

[0028] (4) The technical solution provided by the present invention is expected to improve the accuracy of molecular diagnosis of gliomas, and at the same time provide clinical guidance for preoperative molecular feature evaluation of gliomas and selection of pathological tissue sampling areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0030] Figure 1 It is the technical roadmap of the present invention; Figure 2 This is a schematic diagram of the network structure of the pathological diagnosis model; Figure 3 It is an accurate MRI-pathology matching method; Figure 4 It is an MRI-pathology fuzzy matching method; Figure 5 Generate pathological thumbnails for Transformer generator structure and MRI local area; Figure 6 This is the process of constructing a molecular spatial distribution map. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0032] Example 1, Tumor Diagnosis Method Based on MRI to Generate Molecular Spatial Distribution Map like Figure 1 As shown, the present invention first constructs an MRI and pathological local matching dataset, and then constructs a molecular pathological diagnosis model of glioma based on tissue pathological images, so as to select representative pathological small pictures as the gold standard for the generation task; then, the local MRI sequence containing the sampling center point is converted into a feature matrix and input into a GAN generator based on a Transformer structure to generate a tissue pathological small picture, and the generated image is identified by two aspects: the accuracy of pathological diagnosis and whether the image is real; the molecular features and WHO grades are predicted based on the pathological small pictures generated from different local MRI sub-regions, and the prediction results are combined to obtain a molecular spatial distribution map; finally, the effectiveness of the molecular spatial distribution map is verified by two aspects: a prospective dataset and a comparative experiment.

[0033] The method comprises the following steps: (1) Data collection and data set division The data sources of the present invention include public databases such as TCGA and three hospitals. The inclusion criteria are: 1) Patients diagnosed with adult diffuse glioma according to the 2021 version of the WHO standard; 2) Patients with complete preoperative MRI, histopathological sections and molecular detection data. The exclusion criteria are that the time interval between the preoperative MRI acquisition time and the tissue sampling time is more than one month. The collected MRI sequences include T1-weighted, T1-Gd, T2-weighted, FLAIR and DWI sequences. HE-stained histopathological sections were scanned into whole slide imaging (WSI) for analysis. Molecular detection results and histological grading are used as the diagnostic gold standard in the present invention. Based on the 2021 version of the WHO standard, the specific molecular typing and grading include: 1) Astrocytoma, IDH mutant, WHO grade 2~4; 2) Oligodendroglioma, IDH mutation with 1p / 19q combined deletion, WHO grade 2~3; 3) Glioblastoma, IDH wild type, WHO grade 4.

[0034] We collected 100 prospective data and 600 retrospective data, totaling 700 cases. In the prospective data set, the specific sampling points of the pathological tissue were recorded through intraoperative navigation, and multi-point sampling was performed for each patient, and tissue and molecular pathology examinations were performed separately to expand the pathological data set; at the same time, the image deformation registration algorithm was used to map the coordinates of multiple tissue sampling center points recorded by intraoperative navigation to each preoperative MRI sequence. In the retrospective data set, a possible area for tissue sampling was outlined on the preoperative MRI sequence based on clinical records and the judgment of pathologists.

[0035] All retrospective data and 50 cases of prospective data were divided into training sets, totaling 650 cases, for model training; 20 cases of prospective data were divided into test sets for parameter adjustment and optimization; the remaining 30 cases of prospective data were divided into validation sets to verify the performance of the model.

[0036] (2) Construction of pathological diagnosis model A pathological diagnosis model was constructed based on HE-stained tissue pathological whole-slice images (WSI) to perform differential diagnosis on the molecular typing and grading of glioma. Specifically, the Otsu threshold method was first used to distinguish the tissue and background on the WSI, and then the tissue part was cropped into several 256×256 pathological small images, and the vahadane method was used to perform color correction on each pathological small image. The molecular diagnosis and WHO grading results corresponding to the WSI were assigned to each pathological small image as a classification label. Based on these pathological small images, a pathological diagnosis model was constructed using a weak supervision strategy and a Transformer network structure ( Figure 2 When building the model, each pathological image is segmented and transformed into a feature matrix through a linear layer, and then input into the self-attention module of the Transformer encoder; this module calculates the relationship between image features and analyzes image features from different perspectives through multi-head attention; the superposition of the Transformer encoder gradually abstracts the image features into semantic features related to the classification task, and finally obtains the diagnosis result.

[0037] Propagating the pathological diagnosis results of the tissue sampling points to each pathological thumbnail may cause the diagnostic labels of some thumbnails to be inaccurate, i.e., noise labels. The present invention adopts a loss correction method to reduce the impact of noise labels on training results. The commonly used loss functions in deep learning are cross entropy and mean absolute error (MAE). Cross entropy is an asymmetric loss function with a faster training speed, but is more sensitive to noise labels; MAE is a symmetric loss function with strong robustness to noise labels, but slower training and prone to underfitting. The present invention uses a symmetric cross entropy function that draws on the symmetric structure and anti-noise characteristics of MAE. L sceAs the loss function in model training: (Formula 1) in, is the cross entropy loss function, which is used to speed up the convergence of the model: (Formula 2) is the reverse cross entropy loss function, which is used to improve the model's anti-noise performance: (Formula 3) Therefore, Formula 1 can be rewritten as: (Formula 4) in, k is the number of label types, y ( k ) is a one-hot data label, p ( k ) is the output of the model; for The coefficient of for The coefficients of are the hyperparameters of the model, which are adjusted during the training process. and To balance the convergence speed and noise resistance of the model.

[0038] The trained pathology diagnosis model has three main uses: 1) It is used to screen pathology sub-images. Each pathology sub-image is input into the model to obtain the diagnosis probability of each sub-image. Then, the sub-images with high probability of correct diagnosis are screened out and matched with the local areas of preoperative MRI. 2) It serves as a discriminator for the subsequent GAN model to determine whether the generated pathology image has the diagnostic ability of the real image. 3) It outputs the final molecular spatial distribution map. According to the pathology sub-images generated for each MRI sub-area, the molecular feature diagnosis results are output.

[0039] (3) Construction of MRI-pathology matching dataset The MRI is cropped through a sliding window operation and spatially matched with the pathological data. The cropped MRI local area is used as the input of the generative model, and the tissue pathological image and molecular features are used as the target of model generation. Before cropping, the MRI sequence is first preprocessed: 1) Image registration, image registration of MRIs of different sequences, and registration of the pathological sampling center point to each MRI sequence; 2) Resampling, resampling the voxel spacing of all images to the same size to eliminate the image scale differences between patients / sequences; 3) Bias field correction, reducing the brightness difference of MRI images caused by the bias field; 4) Grayscale discretization, discretizing the grayscale values ​​of the MRI image for subsequent analysis.

[0040] Constructing an MRI-pathology exact matching dataset based on a prospective dataset ( Figure 3 ). Specifically, a fixed-size cubic sliding window is used to slide around the tissue sampling center point on the MRI, and all local regions containing the sampling center point are gradually cut out, and the position of the tissue sampling center point in the sliding window is recorded to form a set of MRI local regions. The number of MRI local regions at each sampling point is equal to the number of voxels contained in the sliding window. N At the same time, the pathological diagnosis model is used to screen out the top N The representative small image with the highest probability of correct diagnosis is used as the gold standard for generating pathological images. The MRI local area and the pathological small image are randomly matched in subsequent training to increase the richness of the training set samples. After random matching, the number of MRI-pathological matches at each tissue sampling point is calculated by N To upgrade to N 2 Yes, if each patient has an average of three tissue sampling points, the number of MRI-pathology matches for each patient is 3 N 2 right.

[0041] Constructing MRI-pathology fuzzy matching dataset based on retrospective dataset ( Figure 4 Specifically, a sliding window of the same size is used to slide non-overlappingly in the possible tissue sampling area to crop out the MRI local area. The number of MRI local areas for each patient is approximately equal to the number of voxels in the outlined area. M The number of voxels with sliding window N Ratio M / N At the same time, the pathological diagnosis model was used to screen out M / N The representative small image with the highest probability of correct diagnosis is used as the gold standard for generating pathological images. The MRI local area and the pathological small image are also randomly matched in the training. The number of MRI-pathological matches for each patient in the retrospective dataset is calculated by M / N To upgrade to ( M / N ) 2 right.

[0042] Therefore, the sample size of the MRI-pathology matching dataset in the training set is: (Formula 5) In the prospective dataset, each MRI local area contains tissue sampling points, which correspond to the pathology in space, so the matching data has accurate labels in the modeling; while in the retrospective dataset, only one MRI local area contains tissue sampling points, so the number of accurate labels for each patient is 1, and the rest are noise labels. The content of noise labels in the entire training set is: (Formula 6) From Formula 5 and Formula 6, we can see that the sample size and quality of MRI and pathology matching data are determined by the size of the sliding window. N and the size of the area delineated by retrospective data M The smaller the sliding window size, the smaller the number of samples with exact matches in the total sample, the larger the number of samples with fuzzy matches, and the more noise labels. r The larger the sliding window size, the more noise labels r The lower the content, the more likely it is to affect the accuracy of the exact matching data set. If the sliding window is a 6×6×6 cube, then N =216; Based on the data collected, the tissue sampling area can be narrowed to the base or top of the tumor, depending on clinical records and physician judgment. M ≈6000. Substituting into formula 1, we get: n≈7,461,363 (Formula 7) Substituting into formula 2, we get: r ≈6% (Formula 8) Previous experience shows that introducing about 5% noise labels can increase the robustness of GAN and improve the training effect. The noise label content of the GAN model in the present invention is about 6%, and it can also be adjusted by the sliding window size according to the generation effect.

[0043] (4) Construction and verification of the noise-adjustable TransformerGAN model The TransformerGAN model is constructed to complete the task of generating pathological thumbnails from local MRI regions. The TransformerGAN model includes a Transformer generator and two discriminators based on the Transformer network structure. Its input is the multi-sequence feature matrix extracted from the local MRI region, and its output is the generated pathological thumbnails.

[0044] like Figure 5 As shown in Figure 1, the Transformer generator consists of multiple stages of processing units, each of which contains multiple Transformer encoders and up- and down-sampling modules to relatively emphasize mid- and high-level information. Figure 2The core module of .b) is the multi-head attention module, which can calculate the structural feature correlation network between different sequences and fuse sequence information from different perspectives. The MRI features of multiple sequences are combined into a feature matrix x : (Formula 9) in, F T1 , F T1-Gd , F T2 , F FLAIR , F DWI The features are extracted from the local areas of T1-weighted, T1-Gd, T2-weighted, FLAIR and DWI sequences using the Pyradiomics tool, respectively.

[0045] x After inputting the Transformer generator, the structural correlation network between sequences is constructed through the encoder's self-attention mechanism to integrate the structural information between sequences: (Formula 10) in, d k Used for QK T Perform a scale transformation to prevent QK T When the value is too large, softmax The gradient of the function disappears; Q To query the matrix, the calculation method is shown in formula 11; K is the keyword matrix, and the calculation method is shown in formula 12; V is a numerical matrix, and the calculation method is shown in formula 13.

[0046] (Formula 11) (Formula 12) (Formula 13) in, W Q , W K and W V are hyperparameters that are automatically learned during training.

[0047] The downsampling, encoding, and upsampling modules at each stage integrate and refine the structural fusion information of the previous stage and gradually transform it into a pathological image of the target resolution.

[0048] The structure of discriminator 1 is Figure 2 Similarly, it is composed of multiple Transformer encoders, which can transform the input pathological image feature matrix into a binary classification result, which is used to judge the authenticity of the generated pathological image. Discriminator 2 is the pathological diagnosis model trained in step (2). It is no longer trained here and is only used to judge the diagnostic performance of the generated image. Discriminant loss function L D is the joint loss of the two discriminators: (Formula 14) in, L sce1 is the loss function of discriminator 1, L sce2 is the loss function of discriminator 2, both of which are symmetric cross entropy losses (Formula 4), and is the coefficient of the two-part loss and is a hyperparameter of the model.

[0049] The generator and discriminator are trained alternately in a cycle to gradually improve the generation performance of the generator and the discrimination performance of the discriminator to avoid the collapse of the model due to one side being too strong. Transformer uses position encoding that can be automatically learned during training. The initial position encoding in the Transformer generator of the forward-looking dataset is the position of the tissue sampling center point in the sliding window, and the rest are randomly generated position encodings.

[0050] The present invention uses three strategies to increase the stability of TransformerGAN and reduce the difficulty of training. First, during the training process, the local MRI area and the pathological small image are randomly matched to effectively expand the amount of training data while ensuring the accuracy of the matching (see Formula 5 and Formula 7). Second, a certain amount of noise labels is introduced (see Formula 6 and Formula 8) to increase the robustness of the model, improve the generation effect, and adjust the content of the noise label by the size of the sliding window. Finally, a dual discriminator structure is adopted, and the pathological diagnosis task is added as a discriminator of GAN to evaluate the generation effect of the image from two perspectives, making the model more stable.

[0051] (5) Construction and verification of molecular spatial distribution map The construction process of the molecular spatial distribution map is as follows Figure 6As shown. Use a sliding window of the same size as in step (3) to gradually crop the region of interest to be diagnosed on the MRI, with the step size of each sliding being 1 voxel, and crop out MRI sub-regions with the same number of voxels as the region of interest, and record the coordinate information of the cropped region. Input each MRI sub-region into the Transformer generator to generate a pathological micro-image, and use the pathological diagnosis model to output the diagnosis results of each pathological micro-image, including molecular typing and grading. Finally, the diagnosis results are mapped to the MRI according to the coordinates, which is the molecular spatial distribution map generated in the present invention. The sliding window can expand the analysis range from a single voxel point to a sub-region, and then achieve voxel-level resolution on the molecular spatial distribution map by sliding the step size. Technically, this method can reduce the difficulty of modeling and improve the stability of the model. Clinically, the training effect of the model can be improved by integrating the surrounding information of the voxel.

[0052] In the validation set, the accuracy of the molecular spatial distribution map was evaluated by comparing the pathological diagnosis results of each patient's tissue sampling area in the prospective data set. Specifically, the molecular spatial distribution map was constructed on the patient's preoperative MRI and used to guide the selection of tissue sampling areas during pathological examination. The accuracy of the molecular feature prediction of the sampling area was used to verify the effectiveness of the molecular spatial distribution map constructed in this study. The preliminary experiment completed the spatial matching of preoperative images and pathological examination tissue sampling areas. The imaging omics features were extracted from the tissue sampling area and the whole tumor area on the preoperative images, respectively, and the recursive elimination algorithm was used to screen the features and the random forest algorithm was used to build a model to predict whether the molecular features were positively expressed. The results showed that the accuracy of the imaging omics features of the tissue sampling area (80%) was significantly better than the imaging omics features of the whole tumor (62%) and clinical features (64%). The experimental results show that artificial intelligence analysis based on the images of the tissue sampling site can better predict molecular features, proving that the basic assumptions of this design are correct.

[0053] Example 2: Tumor diagnosis system for generating molecular spatial distribution map based on MRI It includes: a data acquisition module, which is used to collect MRI sequences, histopathology images and molecular pathology test results of glioma patients, and construct a precise matching data set and a fuzzy matching data set; Pathology diagnosis model building module, used to predict the molecular type and grade of pathology images; based on the pathology images, a pathology diagnosis model is built using a weak supervision strategy and Transformer network structure; MRI-pathology matching dataset construction module, used to crop MRI, perform spatial matching with pathology data, and construct MRI-pathology exact matching dataset and MRI-pathology fuzzy matching dataset; The pathology thumbnail generation module is used to build a TransformerGAN model to generate pathology thumbnails based on local MRI regions. The TransformerGAN model includes a Transformer generator and two discriminators that are also based on the Transformer network structure. The molecular space distribution map construction module is used to generate a tissue pathology map for each MRI sub-region, and use the pathology diagnosis model to output the diagnosis result of each sub-region, and map the diagnosis result to the MRI according to the coordinates to generate a molecular space distribution map; wherein the diagnosis result is the differential diagnosis made by the pathology diagnosis model for molecular typing and grading.

[0054] Embodiment 3, a medium A program is stored thereon, and when the program is executed by one or more data processors described in Example 2, the steps of the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in Example 1 of the present invention are implemented.

[0055] Embodiment 3, an electronic device It includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in Example 1 of the present disclosure are implemented.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A tumor diagnosis method for generating a molecular spatial distribution map based on MRI, characterized in that: The steps include: S1, Construction of MRI-pathology local matching dataset: The data sources of the dataset are MRI sequences, histopathological images, and molecular pathology test results of tumor patients; including prospective data that records the coordinates of the center point of tissue sampling through intraoperative navigation, as well as retrospective data; local spatial matching of the MRI local area at the tissue sampling site with the histopathological images and molecular features; The tissue pathology image is divided into several pathology small images to form a pathology small image dataset for the next step of training the pathology diagnosis model; S2, Construction and verification of "MRI-histopathology" and "histopathology-molecular features" mapping relationships: The pathological sub-image obtained by cropping the tissue pathology image in S1 is used to train the pathological diagnosis model with molecular features as the prediction target, and to construct the mapping relationship between tissue pathology and molecular features; The MRI is cropped and the multi-sequence features of the local area of ​​the MRI are extracted as the input of the GAN model. The information interaction relationship between the MRI sequences is learned through the self-attention mechanism of the Transformer generator to generate pathological thumbnails and construct the mapping relationship between MRI and tissue pathology. S3, constructs a full-volume molecular spatial distribution map with the same resolution as MRI: The global MRI is cropped into sub-regions with the same input size as the GAN model, and histopathology maps are generated respectively. The pathology diagnosis model is used to predict the molecular features of the histopathology maps, and finally they are combined into a full-volume molecular spatial distribution map.

2. The tumor diagnosis method based on MRI to generate molecular spatial distribution map according to claim 1, characterized in that: In step S1, in the prospective dataset, the local MRI containing the center point of tissue sampling is cropped out to form an accurate matching dataset with the pathology; in the retrospective dataset, the possible area of ​​tissue sampling is outlined through clinical records and expert judgment to construct a fuzzy matching dataset.

3. The tumor diagnosis method based on MRI to generate molecular spatial distribution map according to claim 1, characterized in that: In step S2, the tissue pathology image is cut into several pathology sub-images, and a pathology diagnosis model is trained based on a weakly supervised method to predict the molecular typing and grading of the tumor. The model is then used to screen out pathology sub-images with a higher probability of correct diagnosis, which serve as the gold standard for MRI to generate pathology images.

4. The tumor diagnosis method based on MRI to generate molecular spatial distribution map according to claim 1, characterized in that: In step S2, the MRI is cropped by a sliding window operation.

5. The tumor diagnosis method based on MRI to generate molecular spatial distribution map according to claim 1, characterized in that: In step S2, the local MRI regions of multiple sequences are converted into feature matrices, and the self-attention mechanism of Transformer is used to mine the structural interaction information between sequences for image generation, and the input information is refined by alternating up and down sampling.

6. The tumor diagnosis method based on MRI to generate molecular spatial distribution map according to claim 1, characterized in that: In step S3, a sliding window operation is used to divide the sub-regions, and the molecular spatial distribution map reaches the same resolution as MRI by setting the step size to a single voxel.

7. The tumor diagnosis method based on MRI to generate molecular spatial distribution map according to claim 1, characterized in that: In step S3, the MRI area to be diagnosed is divided into sub-regions voxel by voxel using a sliding window, a tissue pathology micro-image is generated for each sub-region, and the molecular typing and grading of each tissue pathology micro-image is obtained through the pathology diagnosis model; the diagnosis result is mapped onto the MRI according to the coordinates of the center point of the MRI sub-region, which is the constructed molecular spatial distribution map.

8. A tumor diagnosis system for generating a molecular spatial distribution map based on MRI, characterized in that: include: The data acquisition module is used to collect MRI sequences, histopathology images, and molecular pathology test results of tumor patients, and record the coordinates of the center point of pathological tissue sampling in the prospective data set, while confirming the potential sampling location in the retrospective data set; Pathology diagnosis model building module, used to predict the molecular type and grade of pathology images; based on the pathology images, a pathology diagnosis model is built using a weak supervision strategy and Transformer network structure; MRI-pathology matching dataset construction module, used to crop MRI, perform spatial matching with pathology data, and construct MRI-pathology exact matching dataset and MRI-pathology fuzzy matching dataset; The pathology thumbnail generation module is used to build a TransformerGAN model to generate pathology thumbnails based on local MRI regions. The TransformerGAN model includes a Transformer generator and two discriminators that are also based on the Transformer network structure. The molecular space distribution map construction module is used to generate a tissue pathology map for each MRI sub-region, and use the pathology diagnosis model to output the diagnosis result of each sub-region, and map the diagnosis result to the MRI according to the coordinates to generate a molecular space distribution map; wherein the diagnosis result is the differential diagnosis made by the pathology diagnosis model for molecular typing and grading.

9. A medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in any one of claims 1 to 7 are implemented.

Images