Tumor Diagnosis Method and System for Generating Molecular Spatial Distribution Map Based on MRI

Abstract

The present invention discloses a tumor diagnosis method and system for generating a molecular spatial distribution map based on MRI, belonging to the technical field of image data processing. The method includes: constructing an MRI-pathology local matching data set; constructing a local MRI-generated pathology small map model by combining Transformer and GAN; constructing and verifying a molecular spatial distribution map. The present invention takes tissue pathology as a bridge, constructs a two-layer mapping relationship between MRI and molecular features, estimates global molecular features with local matching data, effectively utilizes existing clinical resources, and indirectly realizes the evaluation of the three-dimensional spatial distribution of molecular features at a low cost. The constructed model can perform pathological diagnosis on any region of the brain MRI sequence, helps to discover potential invasive regions of tumors, and provides a reference for defining the surgical resection range and planning treatment strategies for gliomas.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a tumor diagnosis method and system for generating a molecular spatial distribution map based on MRI. Background Art

[0002] Glioma is the most common primary brain tumor. According to the WHO standard, molecular pathological characteristics are used as the gold standard for glioma diagnosis, and glioma is graded in combination with histopathological morphological characteristics. The new standard allows similar treatment plans to be formulated for subtypes with similar biological characteristics to enhance efficacy and improve patient prognosis. However, the current pathological examination method still has defects.

[0003] First, molecular characteristics are obtained through pathological examination of surgically removed tumors, and molecular characteristics can only be obtained about one week after surgery, so the timeliness of diagnosis is poor; non-invasive diagnosis cannot be performed before surgery, which limits the planning of early precision treatment plans for gliomas. Secondly, gliomas have obvious intratumoral spatial heterogeneity, and there are large differences in the spatial distribution of molecular typing and microenvironment characteristics of individual tumors. Pathological examinations are performed on local areas of the tumor, which can only evaluate the local molecular characteristics of the tumor and cannot reflect the overall molecular spatial distribution of the tumor. Therefore, how to evaluate the distribution of its molecular characteristics from the overall space of the tumor before surgery is a key issue that needs to be solved in the precise diagnosis of gliomas.

[0004] Magnetic resonance imaging (MRI) can present the spatial distribution of gliomas and their adjacent areas non-invasively, and is an important tool for preoperative diagnosis of gliomas. Previous studies have found that the development of new imaging technologies based on MRI can map the microstructure of gliomas in the same dimension as MRI and improve the diagnostic effect of molecular classification. Artificial intelligence is currently the most effective technology in the field of medical image processing, and its performance in imaging and image post-processing is superior to traditional algorithms. Among them, generative artificial intelligence has a strong ability to create content, and can improve the information dimension through learning. It has excellent performance in MRI super-resolution, MRI-based cross-modal image generation, pathological image generation and segmentation. Through generative artificial intelligence technology, key MRI information related to glioma molecular classification is characterized, and semantic information of molecular features is generated in the MRI dimension, which is expected to provide quantitative analysis and visualization tools for preoperative evaluation of the spatial distribution of glioma molecules, and provide guidance for the selection of pathological sampling sites and accurate diagnosis of gliomas.

[0005] However, existing MRI and artificial intelligence technologies have the following shortcomings in tumor diagnosis:

[0006] 1. Existing MRI-based molecular diagnostic models only learn the mapping relationship between the macroscopic information of tumors on MRI and molecular features. There is a large information difference between the two, and the diagnostic accuracy needs to be improved.

[0007] 2. Existing MRI-based tumor global information prediction schemes for glioma local molecular typing do not spatially correspond the information in the two dimensions of MRI and pathology, lacking reliability and clinical interpretability.

[0008] 3. Artificial intelligence algorithms can identify tumor microenvironment information related to molecular typing on histopathological images, greatly improving the diagnostic efficiency and accuracy of gliomas, and playing a guiding role in the accurate diagnosis of gliomas. However, due to the limitations of tissue sampling, artificial intelligence research based on histopathological images can only predict local molecular typing and cannot evaluate the molecular spatial distribution of the entire tumor. Summary of the Invention

[0009] Aiming at the problem in the prior art that the molecular spatial distribution cannot be preoperatively evaluated in the accurate diagnosis of gliomas, the present invention provides a tumor diagnosis method and system for generating a molecular spatial distribution map based on MRI, solves the key technical problems of fine-grained cross-modal image generation through Transformer and GAN, generates a histopathological image as a bridge to construct a glioma molecular spatial distribution map, and verifies it in a prospective dataset. This technology is expected to provide clinical guidance for the preoperative molecular pathological evaluation of gliomas and the selection of pathological tissue sampling areas, and at the same time improve the accuracy of glioma molecular diagnosis.

[0010] The technical solution proposed by the present invention is a method for generating a molecular spatial distribution map based on preoperative MRI. Taking histopathology as a bridge, the mapping relationship between MRI and molecular pathology is learned through the local matching data of MRI and pathology, and the molecular spatial distribution map of the entire tumor on MRI is evaluated based on this mapping relationship.

[0011] In order to achieve the above technical objectives, the technical solution implemented by the present invention is as follows:

[0012] The first aspect of the present invention provides a tumor diagnosis method for generating a molecular spatial distribution map based on MRI, including the following steps:

[0013] S1. Construction of an MRI-pathology local matching dataset;

[0014] The data sources of the dataset are the MRI sequences, histopathological images, and molecular pathological test results of tumor patients; including prospective data and retrospective data on the coordinates of the tissue sampling center point recorded by intraoperative navigation; locally spatially matching the local MRI region at the tissue sampling site with the histopathological image and molecular features.

[0015] In the prospective dataset, the local MRI containing the tissue sampling center point is cropped and precisely matched with the pathological composition dataset; in the retrospective dataset, the possible tissue sampling areas are outlined through clinical records and expert judgment to construct a fuzzy matching dataset.

[0016] The histopathological images are cut into several small pathological images to form a small pathological image dataset for the training of the next pathological diagnosis model; at the same time, the MRI at the tissue sampling site is cropped into several local regions by sliding the sliding window voxel by voxel, exploring the possibility of generating a high-resolution molecular spatial distribution map under the condition of expanding the dataset, and evaluating the influence of the sliding window size on the generation effect in the modeling.

[0017] S2, construction and verification of the mapping relationship between "MRI-histopathology" and "histopathology-molecular features";

[0018] Taking the small pathological images cropped from the histopathological images in S1 as the prediction target, training a pathological diagnosis model to construct the mapping relationship between histopathology and molecular features; several small pathological images cut from the histopathological images are used to train the pathological diagnosis model based on the weakly supervised method to predict the molecular typing and grading of tumors, and the small pathological images with a higher probability of correct diagnosis are selected by this model as the gold standard for generating pathological images from MRI.

[0019] The MRI is cropped, and the multi-sequence features of the cropped local region of the MRI are extracted as the input of the GAN model. The self-attention mechanism of the Transformer generator is used to learn the information interaction relationship between MRI sequences for generating small pathological images, and the mapping relationship between MRI and histopathology is constructed. In the embodiments of the present invention, the MRI is cropped by sliding window operation.

[0020] The GAN model adopts a dual discriminator structure. One discriminator is a pathological diagnosis model for judging whether the generated pathological image has the diagnostic ability of real images, and the other discriminator is used to identify the authenticity of the generated pathological images. Both adopt the Transformer encoder as the core network structure. The generator and the authenticity discriminator are gradually optimized through alternating training. During the training process, the local region of the MRI and the small pathological images are randomly matched to expand the training sample size.

[0021] The multi-sequence local region of the MRI is converted into a feature matrix, and the self-attention mechanism of the Transformer is used to mine the structural interaction information between sequences for image generation, and the input information is refined by alternating upsampling and downsampling.

[0022] S3, constructing a whole-volume molecular spatial distribution map with the same resolution as the MRI:

[0023] The global MRI is cropped into sub-regions according to the same input size of the GAN model to generate small histopathological images respectively, and the pathological diagnosis model is used to predict the molecular features of the small histopathological images, and finally the whole-volume molecular spatial distribution map is combined.

[0024] The MRI region to be diagnosed is divided into sub-regions voxel by voxel using a sliding window. Small histopathological images are generated for each sub-region respectively, and the molecular typing and grading of each small histopathological image are obtained through the pathological diagnosis model. The diagnostic results are mapped onto the MRI according to the coordinates of the central points of the MRI sub-regions, which is the molecular spatial distribution map constructed by the present invention. This map can characterize the tumor tissue grading and the molecular spatial feature distribution at the same resolution as the MRI.

[0025] In one embodiment of the present invention, the tumor is a brain tumor. Further, the tumor is a glioma.

[0026] Based on the MRI-pathology local matching dataset, the present invention first constructs a model to learn the mapping relationship between the local MRI region and the pathology, then crops the global MRI into sub-regions according to the same input size, predicts the molecular features respectively, and finally combines them into the whole-volume molecular spatial distribution map. The size of the local MRI region affects the resolution and accuracy of the molecular spatial distribution map. To construct a molecular spatial distribution map with the same resolution as the MRI, the most direct method is to set the size of the MRI region input to the model as a single voxel, but this will cause difficulties in modeling and cannot analyze its relationship with the surrounding voxels, and the generated results lack reliability and are prone to outliers. Enlarging the size of the MRI region can reduce the modeling difficulty and improve the reliability of the results, but it will lead to a decrease in the resolution of the molecular spatial distribution map. Therefore, how to construct a high-precision whole-volume molecular spatial distribution map with the same resolution as the MRI is the key problem to be solved by the present invention.

[0027] To solve this problem, the present invention uses a sliding window operation to enlarge the size of the analysis region and expand the sample size, and crops the sub-regions with a sliding step of a single voxel, so as to maintain the same resolution as the MRI for generation. At the same time, taking histopathology as a bridge, the diagnostic information is integrated and refined by increasing and then decreasing the amount of information, improving the molecular prediction accuracy under a small-size sliding window and reducing the requirement of the model for the sliding window size.

[0028] Since pathological examination of the entire tumor volume is not possible for gliomas, the present invention learns the mapping relationship between local MRI and pathology based on an MRI-pathology local matching dataset. Given the limitations of local MRI information and the potential of histopathology in the molecular typing and grading of gliomas, the present invention first generates histopathological images based on local MRI to augment the local MRI information, and then uses the generated histopathological images to predict molecular typing. The input of the generation model is a small-sized local MRI region corresponding to the pathological sampling area, and it is difficult for traditional convolutional neural network models to effectively capture image features; the generated target histopathological images are high-throughput microscopic structure images under a high-power microscope, containing rich image information. How to establish a mapping between a small amount of local MRI information and a large amount of tissue image information is the key problem of the present invention.

[0029] The present invention realizes cross-modal generation between small-sized local MRI and high-throughput histopathological images in the following way: The present invention converts local MRI regions of multiple sequences into feature matrices, adopts the self-attention mechanism of Transformer to mine the structural interaction information between sequences for image generation, and uses an alternating upsampling and downsampling method to refine the input information. At the same time, the histopathological images are sliced into several small pathological images, and a pathological diagnosis model for distinguishing the molecular typing and grading of gliomas is trained in a weakly supervised manner, and then the small pathological images with a relatively high probability of correct diagnosis are selected as the gold standard for the generation task to reduce the amount of information in the histopathology. The pathological diagnosis model also serves as one of the discriminators of the generation model to supervise the image generation effect from the aspect of diagnostic ability.

[0030] The second aspect of the present invention provides a tumor diagnosis system for generating a molecular spatial distribution map based on MRI, including:

[0031] A data acquisition module for collecting the MRI sequences, histopathological images, and molecular pathology test results of tumor patients, recording the coordinates of the center points of pathological tissue sampling in the prospective dataset, and simultaneously identifying the potential sampling positions in the retrospective dataset;

[0032] A pathological diagnosis model construction module for predicting the molecular type and grading of small pathological images; constructing a pathological diagnosis model based on small pathological images using a weakly supervised strategy and a Transformer network structure;

[0033] An MRI-pathology matching dataset construction module for cropping MRI, performing spatial matching with pathological data, and constructing an MRI-pathology exact matching dataset and an MRI-pathology fuzzy matching dataset;

[0034] The pathological micrograph generation module is used to construct a TransformerGAN model to generate pathological micrographs from local regions of MRI; the TransformerGAN model includes a Transformer generator and two discriminators both based on the Transformer network structure;

[0035] The molecular spatial distribution map construction module is used to generate histopathological maps for each MRI sub-region, output the diagnostic results of each sub-region using a pathological diagnosis model, and map the diagnostic results to the MRI according to the coordinates to generate a molecular spatial distribution map; wherein, the diagnostic result is the differential diagnosis made by the pathological diagnosis model for molecular typing and grading.

[0036] In the third aspect of the present invention, a medium is provided, on which a program is stored, and when the program is executed by a processor, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in the present invention are implemented.

[0037] In the fourth aspect of the present invention, an electronic device is provided, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in the present invention are implemented.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] (1) The present invention provides a tumor diagnosis method for generating a molecular spatial distribution map based on MRI, realizing that the diagnostic result is advanced from the postoperative stage of the traditional method to the preoperative stage, improving the timeliness of molecular diagnosis, and being beneficial to the planning of the surgical plan;

[0040] (2) The present invention can evaluate the global situation of the tumor. The present invention proposes a method for generating a molecular feature distribution map based on MRI, estimating the global molecular features with local matching data, effectively utilizing the existing clinical resources, and indirectly realizing the evaluation of the three-dimensional spatial distribution of molecular features at a lower cost. This model can perform pathological diagnosis on any region of the brain MRI sequence, helping to discover potential invasive regions of the tumor and providing a reference for defining the surgical resection range and planning the treatment strategy of glioma.

[0041] (3) In terms of method, the present invention uses the histopathology as a bridge and adopts the Transformer deep learning algorithm to construct two-layer mapping relationships between MRI and molecular features, gradually narrowing the information difference between the two and gradually improving the diagnostic accuracy of glioma.

[0042] (4) The technical solution provided by the present invention is expected to improve the accuracy of molecular diagnosis of glioma, and at the same time provide clinical guidance for preoperative molecular feature evaluation and selection of pathological tissue sampling regions of glioma. Description of the Drawings

[0043] The accompanying drawings of the specification, which form a part of the present invention, are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0044] Figure 1 is the technical roadmap of the present invention;

[0045] Figure 2 is a schematic diagram of the network structure of the pathological diagnosis model;

[0046] Figure 3 is the MRI-pathology precise matching method;

[0047] Figure 4 is the MRI-pathology fuzzy matching method;

[0048] Figure 5 is the Transformer generator structure and the process of generating pathological small images for local MRI regions;

[0049] Figure 6 is the construction process of the molecular spatial distribution map. Detailed Description of the Invention

[0050] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0051] Example 1, Tumor Diagnosis Method for Generating Molecular Spatial Distribution Map Based on MRI

[0052] As Figure 1 shown, the present invention first constructs an MRI and pathology local matching dataset, then constructs a molecular pathological diagnosis model of glioma based on histopathological images, so as to select representative pathological small images as the gold standard for the generation task; then transforms the local MRI sequence containing the sampling center point into a feature matrix and inputs it into the GAN generator based on the Transformer structure to generate histopathological small images, and discriminates the generated images from two aspects: pathological diagnosis accuracy and image authenticity; predicts molecular features and WHO grades respectively based on the pathological small images generated from different local MRI sub-regions, and combines the prediction results to obtain the molecular spatial distribution map; finally, verifies the effectiveness of the molecular spatial distribution map through two aspects: prospective dataset and comparative experiment.

[0053] The method includes the following steps:

[0054] (1) Data collection and dataset division

[0055] The data sources of the present invention include public databases such as TCGA and three hospitals. The inclusion criteria are as follows: 1) Patients diagnosed with adult diffuse glioma according to the 2021 WHO standard; 2) Having complete preoperative MRI, histopathological sections, and molecular detection data. The exclusion criterion is that the interval between the preoperative MRI acquisition time and the tissue sampling time exceeds one month. The collected MRI sequences include T1-weighted, T1-Gd, T2-weighted, FLAIR, and DWI sequences. The histopathological sections stained with HE are scanned into whole slide imaging (WSI) for analysis. The molecular detection results and histological grades are used as the gold standard for diagnosis in the present invention. Based on the 2021 WHO standard, the specific molecular subtypes and grades include: 1) Astrocytoma, IDH mutant type, WHO grade 2-4; 2) Oligodendroglioma, IDH mutant with 1p / 19q co-deletion type, WHO grade 2-3; 3) Glioblastoma, IDH wild type, WHO grade 4.

[0056] One hundred cases of prospective data and six hundred cases of retrospective data are collected, totaling seven hundred cases. In the prospective dataset, the specific sampling points of the pathological tissues are recorded through intraoperative navigation, and multiple tissue samples are taken for each patient and subjected to tissue and molecular pathological examinations respectively to expand the pathological dataset; meanwhile, the coordinates of the central points of multiple tissue samples recorded by intraoperative navigation are mapped to each preoperative MRI sequence using the image deformation registration algorithm. In the retrospective dataset, a possible area for tissue sampling is outlined from the preoperative MRI sequences according to the clinical records and the judgment of pathological experts.

[0057] All the retrospective data and fifty cases of prospective data are divided into the training set, totaling six hundred and fifty cases, for the training of the model; twenty cases of prospective data are divided into the test set for parameter adjustment and optimization; the remaining thirty cases of prospective data are divided into the validation set to verify the performance of the model.

[0058] (2) Construction of the pathological diagnosis model

[0059] A pathological diagnosis model is constructed based on the HE-stained histopathological whole slide images (WSI) to differentially diagnose the molecular subtypes and grades of gliomas. Specifically, first, the Otsu threshold method is used to separate the tissue and the background on the WSI, and then the tissue part is cropped into several pathological small images of 256×256, and the vahadane method is used to correct the color of each pathological small image. The corresponding molecular diagnosis and WHO grade results of the WSI are assigned to each pathological small image as classification labels. Based on these pathological small images, a pathological diagnosis model is constructed using a weakly supervised strategy and a Transformer network structure ( Figure 2) During model construction, each pathological small image is segmented and transformed into a feature matrix through a linear layer, and then input into the self-attention module of the Transformer encoder; this module calculates the mutual relationships between image features and analyzes the image features from different perspectives through multi-head attention; the stacking of the Transformer encoder gradually abstracts the image features into semantic features related to the classification task, and finally obtains the diagnosis result.

[0060] Spreading the pathological diagnosis results of tissue sampling points to each pathological small image may result in inaccurate diagnosis labels for some small images, that is, noise labels. The present invention adopts a loss correction method to reduce the impact of noise labels on the training results. The commonly used loss functions in deep learning are cross-entropy and mean absolute error (MAE). Cross-entropy is an asymmetric loss function, with a relatively fast training speed but sensitive to noise labels; MAE is a symmetric loss function, with strong robustness to noise labels but slow training and prone to underfitting. The present invention uses a symmetric cross-entropy function that draws on the symmetric structure and anti-noise characteristics of MAE L sce as the loss function in model training:

[0061] (Formula 1)

[0062] where is the cross-entropy loss function, used to accelerate the convergence speed of the model:

[0063] (Formula 2)

[0064] is the reverse cross-entropy loss function, used to improve the anti-noise performance of the model:

[0065] (Formula 3)

[0066] Therefore, Formula 1 can be rewritten as:

[0067] (Formula 4)

[0068] where k is the number of label categories, y ( k ) is the data label in one-hot form, p ( k ) is the output of the model; is 's coefficient, is 's coefficient, and both are hyperparameters of the model. During the training process, by adjusting and to balance the convergence speed and anti-noise performance of the model.

[0069] The trained pathological diagnosis model has three main uses: 1) for screening pathological small images, inputting each pathological small image into the model to obtain the diagnosis probability of each small image, and then screening out the small images with high correct diagnosis probability to match with the local area of preoperative MRI; 2) as a discriminator for the subsequent GAN model to judge whether the generated pathological images have the diagnostic ability of real images; 3) output the final molecular spatial distribution map, and according to the pathological small images generated by each MRI sub-region, output the molecular feature diagnosis results.

[0070] (3)Construction of MRI-pathology matching dataset

[0071] The MRI is cropped through a sliding window operation and spatially matched with the pathological data. The cropped local area of MRI is used as the input of the generation model, and the tissue pathological image and molecular features are used as the generation targets of the model. Before cropping, a series of preprocessings are first performed on the MRI sequence: 1) image registration, registering the MRI of different sequences and registering the pathological sampling center point to each MRI sequence; 2) resampling, resampling the voxel spacing of all images to the same size to eliminate the image scale differences between patients / sequences; 3) bias field correction, reducing the brightness difference of MRI images caused by the bias field; 4) gray level discretization, discretizing the gray level values of MRI images for subsequent analysis.

[0072] Construct an MRI-pathology precise matching dataset based on the prospective dataset ( Figure 3 ). Specifically, a cubic sliding window of a fixed size is slid around the tissue sampling center point on the MRI, gradually cropping out all local areas containing the sampling center point, and recording the position of the tissue sampling center point in the sliding window to form a set of MRI local areas. The number of MRI local areas for each sampling point is equal to the number of voxels contained in the sliding window N . At the same time, using the pathological diagnosis model, the top N representative small images with the highest correct diagnosis probability are screened out for each tissue sampling point as the gold standard for generating pathological images. The MRI local areas and pathological small images are randomly matched in the subsequent training to increase the richness of the training set samples. After random matching, the number of MRI-pathology matches for each tissue sampling point is increased from N pairs to N 2 pairs. If each patient has an average of three tissue sampling points, the number of MRI-pathology matches for each patient is 3 N 2 pairs.

[0073] Construct an MRI-pathology fuzzy matching dataset based on a retrospective dataset ( Figure 4 ). Specifically, use a sliding window of the same size to slide step by step without overlap in the possible areas of tissue sampling, crop out the local MRI regions, and the number of local MRI regions for each patient is approximately equal to the number of voxels in the delineated region M The ratio with the number of voxels in the sliding window N Ratio M / N . At the same time, use a pathological diagnosis model to screen out the first M / N representative small images with the highest probability of correct diagnosis as the gold standard for generating pathological images. The local MRI regions and the pathological small images are also randomly matched during training, and the number of MRI-pathology matches for each patient in the retrospective dataset is increased from M / N pairs to ([[]] M / N ) 2 pairs.

[0074] Therefore, the sample size of the MRI-pathology matching dataset in the training set is:

[0075] (Formula 5)

[0076] In the prospective dataset, each local MRI region contains tissue sampling points and is spatially corresponding to the pathology, so the matching data has accurate labels in the modeling; while in the retrospective dataset, only one local MRI region contains tissue sampling points, so the number of accurate labels for each patient is 1, and the rest are noise labels. The content of noise labels in the entire training set is:

[0077] (Formula 6)

[0078] It can be seen from Formula 5 and Formula 6 that the sample size and quality of the MRI-pathology matching data are determined by the size of the sliding window N and the size of the delineated region in the retrospective data M . The smaller the size of the sliding window, the smaller the sample size of the precise matches in the total sample, the larger the sample size of the fuzzy matches, and the higher the content of noise labels r . The larger the size of the sliding window, the lower the content of noise labels r , but it may affect the accuracy of the precise matching dataset. If the sliding window is a 6×6×6 cube, then N = 216; according to the data already collected, according to clinical records and the judgment of doctors, the tissue sampling area can be reduced to the bottom or top of the tumor, M ≈ 6000. Substituting into Formula 1, we get

[0079] n ≈ 7,461,363 (Formula 7)

[0080] Substituting into Formula 2, we get

[0081] r ≈ 6% (Formula 8)

[0082] Previous experience has shown that introducing noise labels of about 5% can increase the robustness of the GAN and improve the training effect. In this invention, the content of noise labels in the GAN model is about 6%, and it can also be adjusted according to the generation effect through the sliding window size.

[0083] (4) Construction and verification of the TransformerGAN model with adjustable noise

[0084] Construct a TransformerGAN model to complete the task of generating pathological small images for local regions of MRI. The TransformerGAN model includes a Transformer generator and two discriminators also based on the Transformer network structure. Its input is the multi-sequence feature matrix extracted from the local region of MRI, and the output is the generated pathological small image.

[0085] As Figure 5 shown, the Transformer generator consists of processing units in multiple stages. Each stage contains multiple Transformer encoders and upsampling and downsampling modules to relatively emphasize the middle and high-level information. The core module of the Transformer encoder ( Figure 2 .b) is the multi-head attention module, which can calculate the structural feature correlation network between different sequences and fuse the sequence information from different perspectives. The MRI features of multiple sequences are combined into a feature matrix x :

[0086] (Formula 9)

[0087] Among them, F T1 , F T1-Gd , F T2 , F FLAIR , F DWI are the features extracted from the local regions of T1-weighted, T1-Gd, T2-weighted, FLAIR, and DWI sequences using the Pyradiomics tool, respectively.

[0088] xAfter entering the Transformer generator, a structural correlation network between sequences is constructed through the self-attention mechanism of the encoder to fuse the structural information between sequences:

[0089] (Formula 10)

[0090] Among them, d k is used to QK T perform scale transformation to prevent QK T the gradient of the softmax function from vanishing when the value is too large; Q is the query matrix, and the calculation method is as shown in Formula 11; K is the keyword matrix, and the calculation method is as shown in Formula 12; V is the value matrix, and the calculation method is as shown in Formula 13.

[0091] (Formula 11)

[0092] (Formula 12)

[0093] (Formula 13)

[0094] Among them, W Q , W K and W V are hyperparameters automatically learned during training.

[0095] The downsampling, encoding, and upsampling modules in each stage integrate and refine the structural fusion information of the previous stage, gradually transforming it into a pathological image with the target resolution.

[0096] The structure of discriminator 1 is similar to Figure 2 , and it consists of multiple Transformer encoders, which can transform the input feature matrix of the pathological small image into a binary classification result for judging the authenticity of the generated pathological small image. Discriminator 2 is the pathological diagnosis model trained in step (2), which is not trained here and is only used to judge the diagnostic performance of the generated image. The discriminative loss function L D is the combined loss of the two discriminators:

[0097] (Formula 14)

[0098] Among them, L sce1 is the loss function of discriminator 1,L sce2 They are the loss functions of discriminator 2, both being symmetric cross-entropy losses (Equation 4). and are the coefficients of the two parts of the loss, which are hyperparameters of the model.

[0099] The generation performance of the generator and the discrimination performance of the discriminator are gradually improved by alternately training the generator and discriminator 1 in a loop, avoiding the collapse of the model caused by one side being too powerful. Transformer uses position encoding that can be automatically learned during training. Among them, the initial position encoding in the prospective dataset Transformer generator is the position of the tissue sampling center point in the sliding window, and the rest are randomly generated position encodings.

[0100] The present invention uses three strategies to increase the stability of TransformerGAN and reduce the training difficulty. One is to randomly match the local regions of MRI and the pathological small images during the training process, effectively expanding the training data volume while ensuring the matching accuracy (see Equations 5 and 7). The second is to introduce a certain amount of noisy labels (see Equations 6 and 8), increasing the robustness of the model, improving the generation effect, and adjusting the content of the noisy labels through the size of the sliding window. Finally, a dual discriminator structure is adopted, adding a pathological diagnosis task as a discriminator of GAN, evaluating the generation effect of the image from two perspectives, and making the model more stable.

[0101] (5)Construction and verification of the molecular spatial distribution map

[0102] The construction process of the molecular spatial distribution map is as Figure 6 shown. The region of interest to be diagnosed on the MRI is gradually cropped using a sliding window of the same size as in step (3), with a step size of 1 voxel for each slide, cropping out MRI sub-regions with the same number of voxels as the region of interest, and at the same time recording the coordinate information of the cropped regions. Each MRI sub-region is input into the Transformer generator to generate pathological small images, and the diagnosis results of each pathological small image are output using a pathological diagnosis model, including molecular typing and grading. Finally, mapping the diagnosis results to the MRI according to the coordinates is the molecular spatial distribution map generated in the present invention. The sliding window can expand the analysis range from a single voxel point to a sub-region, and then achieve voxel-level resolution on the molecular spatial distribution map through the sliding step size. Technically, this method can reduce the difficulty of modeling and improve the stability of the model. Clinically, integrating the surrounding information of the voxels can improve the training effect of the model.

[0103] In the validation set, the accuracy of the molecular spatial distribution map was evaluated by comparing the pathological diagnosis results of the tissue sampling areas of each patient in the prospective dataset. Specifically, the molecular spatial distribution map was constructed on the preoperative MRI of the patients and used to guide the selection of tissue sampling areas during pathological examinations. The effectiveness of the molecular spatial distribution map constructed in this study was verified by the accuracy of predicting the molecular characteristics of the sampling areas. The previous experiment completed the spatial matching between the preoperative images and the tissue sampling areas of the pathological examinations. Radiomics features were extracted from the tissue sampling areas and the whole tumor areas on the preoperative images, and the recursive elimination algorithm was used to screen the features and the random forest algorithm was used to construct the model to predict whether the molecular characteristics were positively expressed. The results showed that the accuracy rate of the radiomics features of the tissue sampling areas (80%) was significantly better than that of the whole tumor radiomics features (62%) and clinical features (64%). The experimental results indicate that artificial intelligence analysis based on the images of the tissue sampling sites can better predict the molecular characteristics, proving that the basic hypothesis of this design is correct.

[0104] Example 2, a tumor diagnosis system for generating a molecular spatial distribution map based on MRI

[0105] It includes: a data acquisition module for collecting the MRI sequences, tissue pathology images and molecular pathology test results of glioma patients to construct a precise matching dataset and a fuzzy matching dataset;

[0106] A pathological diagnosis model construction module for predicting the molecular type and grade of pathological sub-images; based on the pathological sub-images, a pathological diagnosis model is constructed using a weakly supervised strategy and a Transformer network structure;

[0107] An MRI-pathology matching dataset construction module for cropping the MRI, performing spatial matching with the pathological data, and constructing an MRI-pathology precise matching dataset and an MRI-pathology fuzzy matching dataset;

[0108] A pathological sub-image generation module for constructing a TransformerGAN model to generate pathological sub-images based on local regions of the MRI; the TransformerGAN model includes a Transformer generator and two discriminators also based on the Transformer network structure;

[0109] A molecular spatial distribution map construction module for generating tissue pathology images for each MRI sub-region and using the pathological diagnosis model to output the diagnosis results of each sub-region, and mapping the diagnosis results to the MRI according to the coordinates to generate a molecular spatial distribution map; among them, the diagnosis result is the differential diagnosis made by the pathological diagnosis model for the molecular typing and grading.

[0110] Example 3, a medium

[0111] A program is stored thereon, and when the program is executed by one or more data processors described in Embodiment 2, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in Embodiment 1 of the present invention are implemented.

[0112] Embodiment 3, an electronic device

[0113] It includes a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI as described in Embodiment 1 of the present disclosure are implemented.

[0114] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A tumor diagnosis method for generating a molecular spatial distribution map based on MRI, characterized in that, It includes the following steps: S1, Construction of the MRI-Pathology Local Matching Dataset: The data sources of the dataset are the MRI sequences, tissue pathology images, and molecular pathology test results of tumor patients; it includes prospective data and retrospective data on the coordinates of the tissue sampling center points recorded through intraoperative navigation; the local MRI regions at the tissue sampling sites are locally spatially matched with the tissue pathology images and molecular features; The tissue pathology images are cut into several small pathology images to form a small pathology image dataset for the training of the next pathology diagnosis model; S2, Construction and Verification of the "MRI-Tissue Pathology" and "Tissue Pathology-Molecular Feature" Mapping Relationships: Taking the small pathology images obtained by cutting the tissue pathology images in S1 as the input and using the molecular features as the prediction target, training a pathology diagnosis model to construct the mapping relationship between tissue pathology and molecular features; Cropping the MRI, extracting the multi-sequence features of the local MRI regions as the input of the GAN model, and learning the information interaction relationship between MRI sequences through the self-attention mechanism of the Transformer generator for generating small pathology images to construct the mapping relationship between MRI and tissue pathology; S3, Construction of the Whole-Volume Molecular Spatial Distribution Map with the Same Resolution as the MRI: Cropping the global MRI into sub-regions according to the same input size of the GAN model, respectively generating small pathology images of tissue pathology, and using the pathology diagnosis model to predict the molecular features of the small pathology images of tissue pathology, and finally combining them into the whole-volume molecular spatial distribution map.

2. The tumor diagnosis method for generating a molecular spatial distribution map based on MRI according to claim 1, wherein In step S1, in the prospective dataset, the local MRI containing the tissue sampling center point is cropped out to form a precise matching dataset with the pathology; in the retrospective dataset, the possible tissue sampling regions are outlined through clinical records and expert judgment to construct a fuzzy matching dataset.

3. The tumor diagnosis method for generating a molecular spatial distribution map based on MRI according to claim 1, characterized in that, In step S2, based on the several small pathology images obtained by cutting the tissue pathology images, training the pathology diagnosis model in a weakly supervised manner to predict the molecular typing and grading of tumors, and using this model to screen out the small pathology images with a relatively high probability of correct diagnosis as the gold standard for MRI to generate pathology images.

4. The tumor diagnosis method for generating a molecular spatial distribution map based on MRI according to claim 1, wherein In step S2, the MRI is cropped through a sliding window operation.

5. The tumor diagnosis method for generating a molecular spatial distribution map based on MRI according to claim 1, characterized in that In step S2, the local MRI regions of multiple sequences are converted into feature matrices, and the self-attention mechanism of the Transformer is used to mine the structural interaction information between sequences for image generation, and the input information is refined by alternating upsampling and downsampling.

6. The tumor diagnosis method for generating a molecular spatial distribution map based on MRI according to claim 1, wherein In step S3, a sliding window operation is used to divide the sub-regions, and by setting the step size to a single voxel, the molecular spatial distribution map reaches the same resolution as the MRI.

7. The tumor diagnosis method for generating a molecular spatial distribution map based on MRI according to claim 1, characterized in that In step S3, the MRI region to be diagnosed is divided into sub-regions voxel by voxel through a sliding window, small pathology images of tissue pathology are generated for each sub-region respectively, and the molecular typing and grading of each small pathology image of tissue pathology are obtained through the pathology diagnosis model; mapping the diagnosis results to the MRI according to the coordinates of the center points of the MRI sub-regions is the constructed molecular spatial distribution map.

8. A tumor diagnosis system for generating a molecular spatial distribution map based on MRI, characterized in that, It includes: A data acquisition module, which is used to acquire the MRI sequences, histopathological images and molecular pathological test results of tumor patients, record the coordinates of the central point of pathological tissue sampling in the prospective dataset, and confirm the potential sampling positions in the retrospective dataset; A pathological diagnosis model construction module, which is used to predict the molecular type and grade of pathological sub-images; based on the pathological sub-images, a pathological diagnosis model is constructed by using a weakly supervised strategy and a Transformer network structure; An MRI-pathology matching dataset construction module, which is used to crop the MRI, perform spatial matching with the pathological data, and construct an MRI-pathology precise matching dataset and an MRI-pathology fuzzy matching dataset; A pathological sub-image generation module, which is used to construct a TransformerGAN model to generate pathological sub-images based on the local region of the MRI; the TransformerGAN model includes a Transformer generator and two discriminators also based on the Transformer network structure; A molecular spatial distribution map construction module, which is used to generate histopathological images for each MRI sub-region, use the pathological diagnosis model to output the diagnosis results of each sub-region, and map the diagnosis results to the MRI according to the coordinates to generate a molecular spatial distribution map; among them, the diagnosis result is the differential diagnosis made by the pathological diagnosis model for the molecular typing and grading.

9. A medium, characterized in that, A program is stored thereon, and when the program is executed by a processor, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that, It includes a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in the tumor diagnosis method for generating a molecular spatial distribution map based on MRI according to any one of claims 1 to 7 are implemented.

Images