Method for fully automatically generating enhanced image from plain-scan image of brain image meningioma patient

Abstract

The invention provides a method for fully automatically generating an enhanced image from a plain-scan image of a brain image meningioma patient. The method comprises the following steps: performing format conversion on an original image file of the meningioma patient, sorting and dividing the original image file into a training set and a verification set; performing image preprocessing and image data enhancement and data volume amplification on the images of the training set and the verification set; carrying out training and verification, and storing the trained model; and performing post-processing reasoning reconstruction and storage on the meningioma CT slice images of the test set. The data acquired by each device can be converted into a data format suitable for the meningioma patient to enhance the image generation model, so that the method is more suitable for clinical needs of doctors, is convenient to look up, supports data batch processing operation, and has high suitability; the method has the advantages of being repeatable, noninvasive and batched, and improving the diagnosis efficiency of clinicians, and can provide an auxiliary means for simple screening of meningioma images.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and specifically, relates to a method for fully automatically generating enhanced images from plain scan images of brain images of meningioma patients, and more particularly to a method for fully automatically generating cross-modal synthesis of medical images from CT plain scans to MRI enhanced sequences of brain images of meningioma patients based on deep learning. Background Art

[0002] Meningioma is the most common primary intracranial and extracranial tumor in adults. According to the guidelines of the European Association of Neuro-Oncology (ENAO), preoperative imaging examinations, especially T1CE (T1-weighted contrast enhancement) and CT, play a key role in the diagnosis and management of meningioma. T1CE can usually depict the clear boundaries, location, size and surrounding blood supply of meningioma; while CT is generally more sensitive to the detection of bony components, but lacks clear boundaries in the depiction of soft tissue, which is especially challenging in identifying meningioma. However, in the clinical diagnostic process, plain CT scan of the brain is often the first choice for intracranial space-occupying lesions, and subsequent identification and diagnosis of intracranial tumors often require further magnetic resonance imaging (MRI) examinations. For meningioma, T1CE is a key examination method for visualization and management of intracranial meningioma; however, obtaining MRI enhancement sequences (mainly T1CE) takes a lot of time and is expensive, which has certain limitations for patients with contraindications to contrast agents, poor economic conditions and primary hospitals.

[0003] With the development of deep learning in the field of medical image processing, it has become possible to use deep learning algorithms to achieve cross-modal medical image synthesis. Without the need for MRI scanning, artificial intelligence can be used to synthesize MRI enhanced sequence images from CT plain scan images, which greatly facilitates patients and hospitals that are not convenient for MRI enhanced sequence acquisition.

[0004] Previous studies have shown that there is redundant information between images of different modalities and sequences, and parameters can be shared. Through machine learning methods, T1WI, T2WI and PDWI are used as inputs and random forest regression is used to reconstruct fluid attenuated inversion recovery (FLAIR) images. This method is far from the final synthesis goal. Some studies have extracted lesion image information through pyradiomis feature extractors, screened features to construct feature maps, and combined with generative adversarial network (GAN) models for the task of generating MRI from CT of acute ischemic stroke. This feature selection based on information gain only reflects limited lesion features, which ultimately leads to loss of details in the generated image. At the same time, due to the intervention of human factors in the feature selection step, the generated results are far from the fully automatic meningioma image generation process.

[0005] In terms of brain tumor image generation tasks, some studies have trained conditional GANs based on the pix2pix framework to complete generation tasks between different MRI sequences, such as generating T2 FLAIR from T2WI and generating T1WI from T1CE. However, most of these studies focus on MRI image generation tasks for gliomas, and there are many uncertainties in the generalization of cross-modal image generation tasks. In addition, the GAN model consists of a generator and a discriminator, and the adversarial training process between the two requires a lot of computing power. Although the GAN model is widely used in the field of image generation, the complexity of its training structure often causes problems such as mode collapse and training instability, and it also consumes a lot of computing resources.

[0006] The patent document "A method for cross-modal synthesis of medical images from CT to MRI conventional sequences of cerebral infarction" (CN116129235A) discloses a method for cross-modal synthesis of medical images from CT to MRI conventional sequences of cerebral infarction, which completes the task of generating plain scan images from CT to MRI without MRI scanning. However, it focuses on adversarial learning between the generator and the discriminator, and optimizes the parameters of the generator through the loss function constrained by edge features and the Markov discriminator. The calculation is large and the calculation cost is high. The robustness and generation effect of the image generation task are relatively poor, and it is prone to common image generation problems such as higher false positives and checkerboard artifacts.

[0007] In practical applications, the existing technologies’ consumption of computing resources and training stability make it difficult for many algorithms to achieve stable and efficient results. Various methods are also difficult to adapt to the pan-centralized generalization of brain imaging data collected by different imaging devices, resulting in the generated images lacking details and low signal-to-noise ratio, and lacking robustness in the task of synthesizing plain scan CT into MRI enhanced images.

[0008] MRI enhancement is still a necessary means of detecting brain tumor lesions. The edema band at the edge of the tumor and the unclear space-occupying make it difficult to accurately locate the tumor, which places higher demands on the robustness and generalization ability of the deep learning model. Therefore, the clinical value and significance of training CT plain scans to brain image enhancement images with the help of more robust deep learning models are particularly prominent.

[0009] The present invention can be used not only for CT plain scan sequences, but also for one-to-one generation of any two sequences including two parts matching, image signal-to-noise ratio and structural consistency having differences. Summary of the invention

[0010] In view of the defects in the prior art, the object of the present invention is to provide a method for automatically generating enhanced images from plain scan images of brain imaging meningioma patients.

[0011] The method for automatically generating enhanced images from plain scan images of brain imaging meningioma patients provided by the present invention comprises:

[0012] Step S1: converting the format of the original image file of the meningioma patient;

[0013] Step S2: sorting and dividing the meningioma patient image files that have completed format conversion into a training set and a validation set;

[0014] Step S3: performing image preprocessing on the images of the training set and the validation set;

[0015] Step S4: performing image data enhancement on the training set data;

[0016] Step S5: input the images of the training set after image data enhancement into the model training and verify them through the verification set to obtain the trained model;

[0017] Step S6: Input the meningioma CT slice images of the test set into the trained model for training, post-process and save the generated results.

[0018] Preferably, in step S1, MRIcron software is used to convert the original image files of the meningioma patient collected into a neuroimaging informatics technology program format, recorded as NifTI format.

[0019] The original image files of the meningioma patient include computer tomography scan and magnetic resonance enhanced image files in DICOM format, which are recorded as CT and MRI enhanced image files.

[0020] In step S2, the medical image paired data set of meningioma uses image information corresponding to T1CE of CT and MRI;

[0021] CT and MRI imaging devices were used to obtain DICOM format CT data and thin-layer MRI data, respectively, to obtain image data, and then the data was anonymized and desensitized;

[0022] Use ITK-SNAP software to segment the image data, manually outline the region of interest of the entire tumor, obtain the Label file and review and confirm it;

[0023] The delineated Label files and the corresponding CT files and T1CE files are organized into medical image paired data sets and divided into training sets and validation sets.

[0024] In step S6, the test set CT slice images are input into the trained deep learning model, and the fake-T1CE slice images are reconstructed by reasoning and saved.

[0025] The post-processing and saving are performed by reading the image function in the Python image processing standard library, converting it into an array form and storing it in a list, converting the NifTI format file through a simplified interface toolkit of the image analysis software algorithm and data structure, and performing a resampling operation at the same time, and saving the generated NifTI format file in a corresponding folder.

[0026] The list contains the number of samples, image height, and image width data.

[0027] Preferably, step S3 comprises:

[0028] Step S3.1: For the sorted training set and validation set, use the Elastix open source software package to register the CT 3D image to the MRI 3D image space;

[0029] Step S3.2: read the registered training set and validation set into MATLAB and convert them into two-dimensional slices in MAT format;

[0030] Step S3.3: resampling the two-dimensional slices to the same matrix size;

[0031] Step S3.4: Using CT as a template, skull stripping was performed on the two-dimensional slices using MATLAB software;

[0032] Step S3.5: The CT images of the training set and the validation set are normalized, with the Hounsfield unit cutoff range of 0-100, minus the image grayscale mean, and then divided by the standard deviation.

[0033] The registered fixed image is set as an MRI image, the floating image is set as a CT image, and the registration standard is rigid registration.

[0034] The image data enhancement in step S4 includes performing random predetermined angle rotation, cropping, horizontal and vertical mirroring, horizontal and vertical translation of the pre-processed image data to adjust the predetermined values ​​to the initial predetermined multiples, normalization, subtracting the pixel mean, and dividing by the pixel standard deviation;

[0035] The predetermined angle is ±15 degrees;

[0036] The predetermined value is ±32 units.

[0037] The step S3.3 includes the step S3.3.1: cropping the larger image and performing interpolation processing on the smaller image around the image;

[0038] Step S3.3.2: Use interpolation to change the image matrix size to 384×384;

[0039] Step S3.3.3: Histogram equalization is used to improve the contrast between the meningioma area and the surrounding brain tissue in the image, and all data are batch processed through a normalization algorithm.

[0040] Preferably, in step S5, the constructed deep learning model is input into the processed training set for training.

[0041] The deep learning model is a DRUNet50 model, which includes an encoder, a decoder and a loss function.

[0042] The deep learning model uses the dual residual UNet model as the basic framework, and adjusts ResNet and U-Net to form the encoder and decoder of the image generation model to generate synthetic MR images from CT images. Both the input and target images are composed of single-channel 2D image blocks.

[0043] A U-shaped structure based on an encoder-decoder structure is used for image generation, and two types of residual networks are constructed;

[0044] ResNet is used as the encoder and encoder of U-Net to extract complex features layer by layer from the input CT image. The decoding process is converted into a splicing method, and the convolutional features of the decoder are fused with the corresponding features of the encoder;

[0045] The decoder uses a series of residual networks, inverting the entire convolutional component of the encoder and replacing the encoder's downsampling convolutions with nearest neighbor interpolation upsampling, adjusting the stride and kernel size during the deconvolution process.

[0046] During the encoding process, the deep learning model applies a convolutional layer with a 5x5 filter and a stride of 2 and configures a lower bottleneck block;

[0047] Down sampling includes multiple ResNet residual blocks, and a lower bottleneck block is configured with 2, 3, 5, and 2 bottleneck blocks in the DOWN2, DOWN3, DOWN4, and D-CENTER layers respectively.

[0048] In the decoding stage, the deep learning model configures 2, 3, 5, and 2 bottleneck blocks with an upper bottleneck block in the U-CENTER, UP1, UP2, and UP3 layers, respectively, and ends with an upper convolutional layer;

[0049] The upper bottleneck block fuses the feature map of the corresponding level in the encoding path with the feature map in the decoding path through a skip connection.

[0050] The deep learning model is a DRUNet50 architecture with 50 convolutional layers, using a contextual loss function and feature similarity between real MRI enhanced images and synthetic MRI enhanced images.

[0051] The calculation is to input the real MRI enhanced image and the synthetic MRI enhanced image into the pre-trained VGG-19 network respectively, extract the feature output of the fourth convolution layer of the fifth convolution module before the fully connected layer in the VGG-19 network, calculate the cosine distance between the two feature outputs, and perform normalization and power operations.

[0052] Preferably, the training in step S5 is a deep learning fully supervised method, using the processed training set image data for training, and using the trained meningioma patient enhanced image generation model to generate and save the meningioma patient slice images.

[0053] The step S5 includes a network architecture construction step: constructing a DRUNet50 model as a meningioma patient enhanced image generation model, and forming a U-Net encoder and decoder by modifying the ResNet structure, including 50 convolutional layers;

[0054] Model training and optimization steps: Use the training set after image preprocessing and image enhancement as input and output, and import the enhanced training set meningioma CT slice images into the meningioma patient enhanced image generation model for screening;

[0055] Model evaluation step: The peak signal-to-noise ratio and structural similarity between real MRI enhanced images and synthetic MRI enhanced images are calculated to evaluate the performance of the DRUNet50 model.

[0056] The DRUNet50 model has a four-layer encoder and decoder structure, each layer containing specific modules and feature scales.

[0057] The encoder and the decoder have the same hierarchical structure, and the scales of feature maps between adjacent levels are twice as large.

[0058] The context loss function in the DRUNet50 model captures the context information in the image and calculates the feature similarity between the real MRI enhanced image and the synthetic MRI enhanced image.

[0059] The meningioma patient image files that have completed format conversion are organized into image pair data sets and randomly divided into a training set, a validation set, and a test set in a ratio of 80:10:10.

[0060] The enhanced image generation model for meningioma patients adopts a transfer learning approach, uses model parameters trained with a data set, uses TensorFlow as a training framework, and optimizes the parameters in the DRUNet model through Adam until the preset conditions are met and the training is stopped;

[0061] The real image data is predicted, and enhanced MRI slice images of meningioma patients are generated and saved.

[0062] The enhanced image generation model for meningioma patients adds an average pooling layer, a fully connected layer, and a technical layer to prevent network overfitting after the output of the pre-trained model. The activation function uses an exponential linear unit function, presets an initial learning rate, takes minimizing context loss as the goal, and sets the batch size and total number of training rounds of the training process.

[0063] During the training process, the learning rate adopts an adaptive attenuation method. The initial learning rate is set to 0.001, the batch size is set to 16, the total number of training rounds is 80, and when the evaluation index stops rising for 5 times, the learning rate is reduced to 0.25 times of the original value. The minimum supported learning rate is 0.00000001.

[0064] The optimizer uses a stochastic gradient descent method based on first-order adaptive estimation, uses a contextual loss function, and trains 16 samples at a time. The visualization toolkit of the end-to-end open source machine learning platform is used to visualize the model training.

[0065] The model parameters are the optimal model parameters for the validation set data.

[0066] A system for automatically generating enhanced images from plain scan images of brain imaging meningioma patients provided by the present invention comprises: a data processing unit, a meningioma image generating unit, and a post-processing unit.

[0067] The data processing unit performs format conversion on the original image file of the meningioma patient;

[0068] Perform image preprocessing on images of the medical image paired dataset;

[0069] The meningioma patient image files that have completed format conversion are organized into a paired dataset of medical images of meningioma;

[0070] Image data enhancement is performed on the image data of the image pair dataset that has completed the preprocessing.

[0071] The meningioma image generation unit trains the processed image paired data set to generate and save slice images of meningioma patients.

[0072] The post-processing unit post-processes and saves the test set and subsequently newly acquired slice images of meningioma patients.

[0073] Preferably, the original image files of the meningioma patient include computer tomography scans and magnetic resonance enhanced image files in DICOM format, recorded as CT and MRI enhanced image files;

[0074] CT and MRI imaging devices were used to obtain CT data and thin-layer MRI data in DICOM format, respectively, to obtain image data, and then the data was anonymized and desensitized.

[0075] The format conversion uses MRIcron software to convert the original image files of meningioma patients into the Neuro-Imaging Informatics Technology Initiative format, which is recorded as NifTI format.

[0076] In the image preprocessing, the medical image paired data set of meningioma uses image information corresponding to T1CE of CT and MRI;

[0077] Use ITK-SNAP software to segment the image data, manually outline the region of interest of the entire tumor, obtain the Label file and review and confirm it;

[0078] The delineated Label files and the corresponding CT files and T1CE files are organized into medical image paired data sets.

[0079] The post-processing unit applies the trained deep learning model to perform inference reconstruction and storage of magnetic resonance enhanced image post-processing on the slice images of meningioma patients in the test set.

[0080] The slice images of meningioma patients include original slice images of meningioma and MRI enhanced slice images generated by meningioma.

[0081] The post-processing and saving are performed by reading the image function in the Python image processing standard library, converting it into an array form and storing it in a list, converting the NifTI format file through a simplified interface toolkit of the image analysis software algorithm and data structure, and performing a resampling operation at the same time, and saving the generated NifTI format file in a corresponding folder.

[0082] The list contains the number of samples, image height, and image width data.

[0083] Preferably, the data processing unit comprises:

[0084] Module M3.1: Use the Elastix open source software package to register the CT 3D images in the collated medical image pair dataset to the MRI 3D image space;

[0085] The registered fixed image is set as an MRI image, the floating image is set as a CT image, and the registration standard is rigid registration;

[0086] Module M3.2: Read the registered medical image paired data sets into MATLAB and convert them into two-dimensional slices in MAT format;

[0087] Module M3.3: Resample 2D slices to the same matrix size;

[0088] Module M3.4: Use MATLAB software to perform skull stripping on two-dimensional slices using CT as a template;

[0089] Module M3.5: Normalize the CT images of the medical image paired dataset, with the Hounsfield unit cutoff range being 0-100, subtracting the image grayscale mean, and then dividing by the standard deviation.

[0090] The image data enhancement includes performing random predetermined angle rotation, cropping, horizontal and vertical mirroring, horizontal and vertical translation of the pre-processed image data to adjust the predetermined values ​​to the initial predetermined multiples, normalization, subtracting the pixel mean, and dividing by the pixel standard deviation;

[0091] The predetermined angle is ±15 degrees;

[0092] The predetermined value is ±32 units.

[0093] The module M3.3 includes a module M3.3.1: cropping the larger image and interpolating the smaller image around the image;

[0094] Module M3.3.2: Change the image matrix size to 384×384 using interpolation;

[0095] Module M3.3.3: Histogram equalization is used to improve the contrast between the meningioma area and the surrounding brain tissue in the image, and all data are batch processed through a normalization algorithm.

[0096] Preferably, the meningioma image generation unit inputs the processed image pair data set into a constructed deep learning model for training.

[0097] The deep learning model is a DRUNet50 model, which includes an encoder, a decoder and a loss function.

[0098] The deep learning model uses the dual residual UNet model as the basic framework, and adjusts ResNet and U-Net to form the encoder and decoder of the image generation model to generate synthetic MR images from CT images. Both the input and target images are composed of single-channel 2D image blocks.

[0099] A U-shaped structure based on an encoder-decoder structure is used for image generation, and two types of residual networks are constructed;

[0100] ResNet is used as the encoder and encoder of U-Net to extract complex features layer by layer from the input CT image. The decoding process is converted into a splicing method, and the convolutional features of the decoder are fused with the corresponding features of the encoder;

[0101] The decoder uses a series of residual networks, inverting the entire convolutional component of the encoder and replacing the encoder's downsampling convolutions with nearest neighbor interpolation upsampling, adjusting the stride and kernel size during the deconvolution process.

[0102] During the encoding process, the deep learning model applies a convolutional layer with a 5x5 filter and a stride of 2 and configures a lower bottleneck block;

[0103] Down sampling includes multiple ResNet residual blocks, and a lower bottleneck block is configured with 2, 3, 5, and 2 bottleneck blocks in the DOWN2, DOWN3, DOWN4, and D-CENTER layers respectively.

[0104] In the decoding stage, the deep learning model configures 2, 3, 5, and 2 bottleneck blocks with an upper bottleneck block in the U-CENTER, UP1, UP2, and UP3 layers, respectively, and ends with an upper convolutional layer;

[0105] The upper bottleneck block fuses the feature map of the corresponding level in the encoding path with the feature map in the decoding path through a skip connection.

[0106] The deep learning model is a DRUNet50 architecture with 50 convolutional layers, using a contextual loss function and feature similarity between real MRI enhanced images and synthetic MRI enhanced images.

[0107] The calculation is to input the real MRI enhanced image and the synthetic MRI enhanced image into the pre-trained VGG-19 network respectively, extract the feature output of the fourth convolution layer of the fifth convolution module before the fully connected layer in the VGG-19 network, calculate the cosine distance between the two feature outputs, and perform normalization and power operations.

[0108] Preferably, the training is a deep learning fully supervised method, using processed image data for training, and using a trained meningioma patient enhanced image generation model to generate and save slice images of meningioma patients.

[0109] The meningioma image generation unit includes a network architecture building module: building a DRUNet50 model, forming an encoder and a decoder of a U-Net by modifying a ResNet structure, and including 50 convolutional layers.

[0110] Model training and optimization module: Use the slice images of meningioma patients after image preprocessing and image enhancement as input and output, and import the enhanced slice images of meningioma patients into the meningioma patient enhanced image generation model for screening.

[0111] Model evaluation module: The peak signal-to-noise ratio and structural similarity between real MRI enhanced images and synthetic MRI enhanced images are calculated to evaluate the performance of the DRUNet50 model.

[0112] The DRUNet50 model has a four-layer encoder and decoder structure, each layer containing specific modules and feature scales.

[0113] The encoder and the decoder have the same hierarchical structure, and the scales of feature maps between adjacent levels are twice as large.

[0114] The context loss function in the DRUNet50 model captures the context information in the image and calculates the feature similarity between the real MRI enhanced image and the synthetic MRI enhanced image.

[0115] The image pair dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 80:10:10.

[0116] The enhanced image generation model for meningioma patients adopts a transfer learning approach, uses model parameters trained with a data set, uses TensorFlow as a training framework, and optimizes the parameters in the DRUNet model through Adam until the preset conditions are met and the training is stopped;

[0117] The real image data is predicted, and enhanced MRI slice images of meningioma patients are generated and saved.

[0118] The enhanced image generation model for meningioma patients adds an average pooling layer, a fully connected layer, and a technical layer to prevent network overfitting after the output of the pre-trained model. The activation function uses an exponential linear unit function, presets an initial learning rate, takes minimizing context loss as the goal, and sets the batch size and total number of training rounds of the training process.

[0119] During the training process, the learning rate adopts an adaptive attenuation method. The initial learning rate is set to 0.001, the batch size is set to 16, the total number of training rounds is 80, and when the evaluation index stops rising for 5 times, the learning rate is reduced to 0.25 times of the original value. The minimum supported learning rate is 0.00000001.

[0120] The optimizer uses a stochastic gradient descent method based on first-order adaptive estimation, uses a contextual loss function, and trains 16 samples at a time. The visualization toolkit of the end-to-end open source machine learning platform is used to visualize the model training.

[0121] The model parameters are the optimal model parameters for the validation set data.

[0122] Compared with the prior art, the present invention has the following beneficial effects:

[0123] 1. The present invention is applicable to various brain image acquisition devices. The data collected by each device can be converted into a data format suitable for the enhanced image generation model of meningioma patients through data enhancement and data preprocessing methods, and has high adaptability.

[0124] 2. The present invention generates MRI enhanced slice images of meningioma patients more efficiently. The data post-processing can be converted into NIfTI format files, which are more suitable for doctors' clinical needs and easy to consult. At the same time, all data processing, model training, and model testing support data batch processing operations.

[0125] 3. The present invention integrates deep learning image generation technology, which has the advantages of repeatability, non-invasiveness, batching, and improving the diagnostic efficiency of clinicians, providing an auxiliary means for simple screening of meningioma images.

[0126] 4. The image generation effect and robustness of the image generation task of the model built by the convolutional network in the present invention are better than those of the traditional methods, and can better avoid common image generation problems such as false positives and checkerboard artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0128] Figure 1 Schematic diagram of the process of fully automatically generating enhanced images from plain scan images of patients with meningioma.

[0129] Figure 2 Schematic diagram of the slice processing flow to be input into the DRUNet50 deep learning model.

[0130] Figure 3 A schematic diagram of the slices to be input into the DRUNet50 deep learning model.

[0131] Figure 4 This is a structural diagram of the DRUNet50 deep learning model.

[0132] Figure 5 Schematic diagram of the evaluation indicators and loss function curve of the DRUNet50 deep learning model trained in the data.

[0133] Figure 6 A diagram showing the visualization results of the DRUNet50 deep learning model trained on the data. DETAILED DESCRIPTION

[0134] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0135] The present invention is used to generate MRI enhanced images from two-dimensional CT plain scans of brain images of clinical meningioma patients. According to the present invention, a method for fully automatically generating enhanced images from plain scans of brain images of meningioma patients is provided. Figure 1 For example, the following steps are included:

[0136] Step S1: converting the format of the original image file of the meningioma patient.

[0137] Use MRIcron software to convert the acquired DICOM format CT and MRI enhanced image files into NifTI format.

[0138] The slice processing flow before inputting the DRUNet50 deep learning model is as follows Figure 2 For example, before processing the slices, the original image files are converted into a format. The image files include computed tomography (CT) and magnetic resonance (MRI) enhanced image files. The CT and MRI enhanced image files of meningioma patients in the Digital Imaging and Communications in Medicine (DICOM) format are converted into the Neuroimaging Informatics Technology Initiative (NifTI) format through the MRIcron toolkit using a programming language.

[0139] Step S2: The meningioma patient image files that have completed format conversion are organized into a paired dataset of medical images of meningioma.

[0140] In more preferred examples, the paired medical image dataset of meningioma uses the image information of the same patient corresponding to T1CE of CT and MRI, i.e., the paired CT-MRI dataset of meningioma patients;

[0141] Two modalities (CT / MRI) imaging devices were used to obtain CT and thin-layer MRI data in DICOM format, respectively, to obtain image data, and the data was anonymized and desensitized. The obtained data was used for subsequent lesion and data pre-training steps. ITK-SNAP software was used for lesion segmentation, and the whole tumor region of interest (ROI) was manually outlined on the CE-T1WI image and reviewed and confirmed. The outlined Label file and the corresponding CT and T1CE files were organized into a data set for image preprocessing before inputting the model.

[0142] Step S3: further perform image preprocessing on the images in the sorted medical image paired data set.

[0143] In more preferred embodiments, the image preprocessing of the medical image paired data set in step S3 includes the following steps:

[0144] Step S3.1: For the sorted medical image pair data set, use the Elastix open source software package to align the CT-MRI paired three-dimensional NifTI data. The fixed image is set as the MRI image, the floating image is set as the CT image, and the rigid registration is selected as the registration standard to align the CT three-dimensional image to the MRI three-dimensional image space.

[0145] Step S3.2: Read the registered medical image paired data set into MATLAB and convert it into two-dimensional slices in MAT format.

[0146] Step S3.3: All 2D slices are resampled to the same matrix size.

[0147] Specifically, after completing the MAT format conversion, first unify the image file size: crop the large image, and interpolate the small image around the image. Use interpolation to change the image matrix size to 384×384, and then unify the image size to 384×384. Perform preprocessing on the converted image.

[0148] Histogram equalization was used to improve the contrast between the meningioma area and the surrounding brain tissue in the image, and then all the data were processed in batches through a normalization algorithm to improve data consistency.

[0149] Step S3.4: Use MATLAB software to perform skull stripping on two-dimensional slices of CT and MRI enhanced images using CT as a template.

[0150] Step S3.5: Normalize the CT image, with the Hounsfield unit (HU) cutoff range of 0-100, subtract the image grayscale mean, and divide by the standard deviation. Input the image slices that have been preprocessed before training the DRUNet50 deep learning model as follows Figure 3 shown.

[0151] Step S4: further performing image data enhancement on the image data in the preprocessed image pair data set to enlarge the data volume.

[0152] In more preferred examples, in step S4, meningioma image data enhancement is performed to increase the sample size of model training data;

[0153] Specifically, the method of enhancing the image data after image preprocessing includes randomly cropping the preprocessed image at a predetermined angle, rotating, horizontally and vertically mirroring, and adjusting the horizontal and vertical translation predetermined values ​​to the initial predetermined multiples, and enhancing all 2D data after image preprocessing.

[0154] The specific process of image data enhancement is as follows:

[0155] Step S4.1: Randomly rotate ±15 degrees;

[0156] Step S4.2: horizontal mirroring;

[0157] Step S4.3: horizontal translation ±32 units;

[0158] Step S4.4: vertical translation ±32 units;

[0159] Step S4.5: Normalize by subtracting the pixel mean and dividing by the pixel standard deviation.

[0160] By enhancing image data, we can avoid overfitting, increase image diversity, avoid sample imbalance, and improve the generalization and accuracy of the training model.

[0161] Step S5: For the processed image pair dataset, that is, the CT-MRI paired dataset that has completed image preprocessing and enhancement, the constructed deep learning model is input for training to obtain the generated meningioma patient slice images and save them.

[0162] by Figure 2 and Figure 6 Taking CT images of meningioma patients as an example, a model for generating enhanced images of meningioma patients is established. The model is trained based on CT images of meningioma patients processed by image enhancement to increase the diversity of training data for the model. The model for generating enhanced images of meningioma patients adopts a deep learning full supervision method and uses processed image data for training. In the test phase, the trained model for generating enhanced images of meningioma patients is directly used to generate CT slice images of meningioma patients, and the generated enhanced slice images of meningioma patients are saved.

[0163] by Figure 4 As an example, the deep learning model uses the dual residual UNet model as the basic framework, and adjusts ResNet and U-Net to form the encoder and decoder of the image generation model (dual ResNet-U-Net, DRUnet50 model) to generate synthetic MR images from CT images. Both the input (CT) and target (CE-T1WI) images consist of single-channel 2D image blocks.

[0164] The deep learning model mainly consists of three parts: encoder, decoder, and loss function.

[0165] Based on the dual residual UNet (DRUNet) model, the encoder-decoder is constructed by integrating ResNet and U-Net. Both the encoder and the decoder use the residual network (ResNet) structure. The encoder is used to extract features from the input image, and the decoder is used to gradually restore the spatial resolution of the image and fuse the feature maps of the encoder and decoder through jump connections to preserve image details.

[0166] A U-shaped structure based on an encoder-decoder structure (i.e., a rate structure) is used for image generation, which is compatible with both ResNet and U-Net, and two types of residual networks are constructed to ensure the optimal transmission of spatial information from the input to the output image. ResNet is designed as the encoder of U-Net to extract complex features layer by layer from the input CT image; at the same time, ResNet is used as the decoder of U-Net, and the decoding process is converted into a splicing method, where the convolutional features of the decoder are fused with the corresponding features of the encoder to gradually reconstruct the predicted MR image from low resolution. To match the hierarchical structure of the encoder, the decoder uses a series of residual networks to generate synthetic images, invert the entire convolution component of the encoder, and replace the encoder's downsampling convolution with the nearest neighbor interpolation upsampling to obtain the output. The step size and kernel size are adjusted during the deconvolution process to avoid the generation of artifacts of checkerboard patterns and uneven overlap.

[0167] The present invention mainly relies on the encoder-decoder structure for image generation. The encoder extracts features layer by layer, and the decoder reconstructs the image by gradually upsampling and fusing features. The whole process does not have a direct adversarial learning mechanism, but relies on the advantages of the context loss function and the residual network to retain the detailed information of the image, but the optimization strategy is more flexible.

[0168] Compared with traditional GAN ​​models, including cGAN and cycleGAN models, the image generation effect and robustness of the image generation task built using convolutional networks are better, and they can better avoid common image generation problems such as false positives and checkerboard artifacts. Through the skip connection mechanism in the residual network, the feature map in the encoder is fused with the feature map in the decoder, which helps to retain the detailed information of the image and reduce information loss.

[0169] Coding path: During the encoding process, a convolutional layer with a 5x5 filter and a stride of 2 is first applied and a lower bottleneck block is configured to initially extract image features. Subsequently, multiple ResNet residual blocks are downsampled, and a lower bottleneck block is configured with 2, 3, 5, and 2 bottleneck blocks in the DOWN2, DOWN3, DOWN4, and D-CENTER layers, respectively. These bottleneck blocks effectively capture the high-order features of the image by reducing the size of the feature map (i.e., spatial dimension) and increasing the depth of the feature map (i.e., number of channels).

[0170] Decoding path: Symmetrical with the encoder, in the decoding stage, in order to gradually restore the spatial resolution of the image, 2, 3, 5, and 2 bottleneck blocks are configured in the U-CENTER, UP1, UP2, and UP3 layers, respectively, with an upper bottleneck block, and finally ended by an upper convolution layer. The upper bottleneck block fuses the feature map of the corresponding level in the encoding path with the feature map in the decoding path through a skip connection. This mechanism helps to retain the detailed information of the image and reduce information loss.

[0171] Network structure and optimization: The entire network forms a DRUNet50 architecture with 50 convolutional layers. In order to learn more high-dimensional features from CT images, a contextual loss function is used to optimize the network by calculating the feature similarity between the real CE-T1WI image and the synthetic CE-T1WI image. First, the real CE-T1WI image and the synthetic CE-T1WI image are input into the pre-trained VGG-19 network respectively, and then the feature output of the fourth convolutional layer (conv5_4) of the fifth convolutional module in front of the fully connected layer in the VGG-19 network is extracted. This layer can capture the high-level semantic information of the image. Finally, the cosine distance between the two feature outputs is calculated, and normalized and exponentially operated to obtain the final loss value.

[0172] In more preferred embodiments, the process of synthesizing MRI enhanced images of meningioma patients through deep learning in step S5 includes the following steps:

[0173] Network architecture design steps: Build a dual ResNet-U-Net (DRUNet50) model, which modifies the ResNet structure to form the encoder and decoder of U-Net. The entire DRUNet50 model contains a total of 50 convolutional layers.

[0174] The training process of GAN (Generative Adversarial Network) is often unstable, prone to problems such as mode collapse or training oscillation. This may lead to unstable quality of generated images, or even failure to generate valid images in some cases. The present invention does not adopt the adversarial training mechanism of GAN, thus avoiding the instability problems that may occur in GAN training. The DRUNet model generates images through an encoder-decoder structure, the training process is relatively stable, and the quality of the generated images is more controllable.

[0175] The DRUNet50 model has a four-layer encoder and decoder structure, each layer contains specific modules and feature scales, the encoder and decoder have the same hierarchical structure, and the scales of feature maps between adjacent layers are twice that of each other to ensure effective transmission of information.

[0176] The contextual loss function in the DRUNet50 model is used to capture contextual information in the image. The function optimizes the network by calculating the feature similarity between the real MRI enhanced image and the synthetic MRI enhanced image, so as to provide richer high-frequency information when synthesizing MRI enhanced images, thereby further improving the model's detail retention ability and overall quality when synthesizing MRI enhanced images.

[0177] The dual residual UNet (DRUNet) model used is relatively simple in structure and has a small amount of computation. By integrating the advantages of ResNet and U-Net, DRUNet can reduce computational complexity and resource requirements while ensuring image quality.

[0178] Model training and optimization steps: Use CT slices and MRI enhanced slices containing meningioma after image preprocessing and enhancement as input and output. Import the enhanced slice images into the enhanced image generation model for meningioma patients for screening; the establishment of the enhanced image generation model for meningioma patients adopts transfer learning, using model parameters trained with the data set. The training framework uses TensorFlow, and Adam optimizes the parameters in the DRUNet model until the preset conditions are met to stop training.

[0179] The meningioma patient CT-MRI paired dataset was randomly divided into a training set, a validation set, and a test set in a ratio of 80:10:10.

[0180] In the prediction stage, the real image data is predicted, and the enhanced MRI slice images of meningioma patients are generated and saved. The enhanced image generation model of meningioma patients adds an average pooling layer, multiple fully connected layers, and multiple techniques (Dropout) to prevent network overfitting after the output of the pre-trained model. The activation function uses the exponential linear unit (elu) function. The initial learning rate is preset, with the goal of minimizing the context loss, and the batch size and total number of training rounds of the training process are set.

[0181] The parameters of the training process are as follows: the learning rate adopts an adaptive decay method, the initial setting of the learning rate is 0.001, the batch size is set to 16, and the total number of training rounds (epochs) is 80 to minimize the context loss function. When the evaluation index stops rising for 5 times, the learning rate will be reduced to 0.25 times the original, and the minimum learning rate supported is 0.00000001. The optimizer uses a stochastic gradient descent method (Adam) optimizer based on first-order adaptive estimation. Convergence is performed using the context loss function. The number of samples for one training is 16. The visualization toolkit (TensorBoard) of the end-to-end open source machine learning platform (TensorFlow) is used to visualize the model training. The model saves the validation set data to achieve the best training parameter model.

[0182] Model evaluation step: The performance of the DRUNet50 model in the CT plain scan synthetic MRI enhanced image task is evaluated by calculating the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) between the real MRI enhanced image and the synthetic MRI enhanced image. Figure 5 For example, the visualization results of the DRUNet 50 training model during training, including the evaluation of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) and the loss function curve.

[0183] Step S6: Perform post-processing inference reconstruction on the test set and subsequent newly collected data and save them.

[0184] In more embodiments, the trained deep learning model is applied to the test set of meningioma patient CT images, i.e., meningioma patient slice images, to perform inference reconstruction and save the magnetic resonance enhanced image post-processing. In other words, the trained DRUNet50 model is applied to actual CT scan images to automatically synthesize high-quality MRI enhanced images to assist clinical diagnosis and medical research.

[0185] The image post-processing method can realize the mapping of the MRI enhanced slice image generated by meningioma with the original slice image, and save it as a NifTI format file for doctors to view using related software. The original slice image containing meningioma and the MRI enhanced slice image generated by meningioma, that is, the post-processed image information, are read through the image (Image) function in the Python Image Processing Standard Library (Python Imaging Library-PIL). After the image is read, it is converted into an array form and stored in a list. The list contains the number of samples, image height and image width data. The post-processed image information is converted into a NifTI format file through the SimpleITK toolkit for image analysis software algorithms and data structures. In this process, the image is resampled to ensure that the original slice image containing meningioma and the generated MRI enhanced slice image have the same sampling interval and a one-to-one correspondence. Finally, the generated NifTI format file is saved in the corresponding folder. Image post-processing aims to map the original slice image of meningioma patients with the generated MRI enhanced slice image, so that doctors can view it later using common clinical software.

[0186] The present invention also provides a system for fully automatically generating enhanced images from plain scan images of brain imaging meningioma patients. The system for fully automatically generating enhanced images from plain scan images of brain imaging meningioma patients can be realized by executing the process steps of the method for fully automatically generating enhanced images from plain scan images of brain imaging meningioma patients, that is, those skilled in the art can understand the method for fully automatically generating enhanced images from plain scan images of brain imaging meningioma patients as a preferred implementation of the system for fully automatically generating enhanced images from plain scan images of brain imaging meningioma patients.

[0187] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0188] According to the present invention, a system for fully automatically generating enhanced images from plain scan images of brain imaging meningioma patients includes: a data processing unit, a meningioma image generation unit, and a post-processing unit;

[0189] A data processing unit is used to perform format conversion, preprocessing, image data enhancement, etc. on the cranial magnetic resonance CT and MRI enhanced sequence images of each meningioma patient;

[0190] Specifically, the data processing unit converts the format of the original image file of the meningioma patient;

[0191] Perform image preprocessing on images of the medical image paired dataset;

[0192] The meningioma patient image files that have completed format conversion are organized into a paired dataset of medical images of meningioma;

[0193] Image data enhancement is performed on the image data of the preprocessed image pair data set to enlarge the data volume.

[0194] Based on the paired dataset of pre-processed and enhanced images of meningioma patients, the meningioma image generation unit integrated with multiple deep learning modules can complete the image generation task of generating fake-T1CE images with enhanced effects from CT plain scan images across modalities.

[0195] Specifically, the meningioma image generation unit trains the processed image pair data set to generate and save slice images of meningioma patients.

[0196] The post-processing unit trains the processed image paired data sets, converts the generated MRI enhanced slice images of meningioma patients into a format that can be read by commonly used software for clinicians and saves them.

[0197] In more preferred examples, the original image files of the meningioma patient include computer tomography scans and magnetic resonance enhanced image files in DICOM format, which are recorded as CT and MRI enhanced image files.

[0198] Two modality (CT / MRI) imaging devices were used to obtain CT and thin-layer MRI data in DICOM format, and image data were obtained, and data anonymization and desensitization were performed;

[0199] The format conversion uses MRIcron software to convert the original image files of meningioma patients into the Neuro-Imaging Informatics Technology Initiative format, which is recorded as NifTI format.

[0200] In the image preprocessing, the medical image paired data set of meningioma uses image information corresponding to T1CE of CT and MRI;

[0201] Use ITK-SNAP software to segment the image data, manually outline the region of interest (ROI) of the entire tumor, obtain the Label file and review and confirm it;

[0202] The delineated Label files and the corresponding CT files and T1CE files are organized into medical image paired data sets.

[0203] The post-processing unit applies the trained deep learning model to perform inference reconstruction and storage of magnetic resonance enhanced image post-processing on the slice images of meningioma patients in the test set.

[0204] The slice images of meningioma patients include original slice images of meningioma and MRI enhanced slice images generated by meningioma.

[0205] The post-processing and saving are performed by reading the image function in the Python image processing standard library, converting it into an array form and storing it in a list, converting the NifTI format file through a simplified interface toolkit of the image analysis software algorithm and data structure, and performing a resampling operation at the same time, and saving the generated NifTI format file in a corresponding folder.

[0206] The list contains the number of samples, image height, and image width data.

[0207] In more preferred embodiments, the data processing unit includes:

[0208] Module M3.1: Use the Elastix open source software package to register the CT 3D images in the collated medical image pair dataset to the MRI 3D image space;

[0209] The fixed image to be registered is set as an MRI image, the floating image is set as a CT image, the registration standard is rigid registration, and the input CT / MRI image can be adjusted according to training needs;

[0210] Module M3.2: Read the registered medical image paired data sets into MATLAB and convert them into two-dimensional slices in MAT format;

[0211] Module M3.3: Resample 2D slices to the same matrix size;

[0212] Module M3.4: Use MATLAB software to perform skull stripping on two-dimensional slices using CT as a template;

[0213] Module M3.5: Normalize the CT images of the medical image paired dataset, with the Hounsfield unit cutoff range being 0-100, subtracting the image grayscale mean, and then dividing by the standard deviation.

[0214] The image data enhancement includes performing random predetermined angle rotation, cropping, horizontal and vertical mirroring, horizontal and vertical translation of the pre-processed image data to adjust the predetermined values ​​to the initial predetermined multiples, normalization, subtracting the pixel mean, and dividing by the pixel standard deviation;

[0215] The predetermined angle is ±15 degrees;

[0216] The predetermined value is ±32 units.

[0217] The module M3.3 includes a module M3.3.1: cropping the larger image and interpolating the smaller image around the image;

[0218] Module M3.3.2: Change the image matrix size to 384×384 using interpolation;

[0219] Module M3.3.3: Histogram equalization is used to improve the contrast between the meningioma area and the surrounding brain tissue in the image, and all data are batch processed through a normalization algorithm.

[0220] In more preferred examples, the meningioma image generation unit inputs the processed image pair data set into a constructed deep learning model for training.

[0221] The deep learning model is a DRUNet50 model, which includes an encoder, a decoder and a loss function.

[0222] The deep learning model uses the dual residual UNet model as the basic framework, and adjusts ResNet and U-Net to form the encoder and decoder of the image generation model to generate synthetic MR images from CT images. Both the input and target images are composed of single-channel 2D image blocks.

[0223] A U-shaped structure based on an encoder-decoder structure is used for image generation, and two types of residual networks are constructed;

[0224] ResNet is used as the encoder and encoder of U-Net to extract complex features layer by layer from the input CT image. The decoding process is converted into a splicing method, and the convolutional features of the decoder are fused with the corresponding features of the encoder;

[0225] The decoder uses a series of residual networks, inverting the entire convolutional component of the encoder and replacing the encoder's downsampling convolutions with nearest neighbor interpolation upsampling, adjusting the stride and kernel size during the deconvolution process.

[0226] During the encoding process, the deep learning model applies a convolutional layer with a 5x5 filter and a stride of 2 and configures a lower bottleneck block;

[0227] Down sampling includes multiple ResNet residual blocks, and a lower bottleneck block is configured with 2, 3, 5, and 2 bottleneck blocks in the DOWN2, DOWN3, DOWN4, and D-CENTER layers respectively.

[0228] In the decoding stage, the deep learning model configures 2, 3, 5, and 2 bottleneck blocks with an upper bottleneck block in the U-CENTER, UP1, UP2, and UP3 layers, respectively, and ends with an upper convolutional layer;

[0229] The upper bottleneck block fuses the feature map of the corresponding level in the encoding path with the feature map in the decoding path through a skip connection.

[0230] The deep learning model is a DRUNet50 architecture with 50 convolutional layers, using a contextual loss function and feature similarity between real MRI enhanced images and synthetic MRI enhanced images.

[0231] The calculation is to input the real MRI enhanced image and the synthetic MRI enhanced image into the pre-trained VGG-19 network respectively, extract the feature output of the fourth convolution layer of the fifth convolution module before the fully connected layer in the VGG-19 network, calculate the cosine distance between the two feature outputs, and perform normalization and power operations.

[0232] In more preferred examples, the training is a deep learning fully supervised method, using processed image data for training, and using a trained meningioma patient enhanced image generation model to generate and save meningioma patient slice images.

[0233] The meningioma image generation unit includes a network architecture building module: building a DRUNet50 model, forming an encoder and a decoder of a U-Net by modifying a ResNet structure, and including 50 convolutional layers.

[0234] Model training and optimization module: Use the slice images of meningioma patients after image preprocessing and image enhancement as input and output, and import the enhanced slice images of meningioma patients into the meningioma patient enhanced image generation model for screening.

[0235] Model evaluation module: The peak signal-to-noise ratio and structural similarity between real MRI enhanced images and synthetic MRI enhanced images are calculated to evaluate the performance of the DRUNet50 model.

[0236] The DRUNet50 model has a four-layer encoder and decoder structure, each layer containing specific modules and feature scales.

[0237] The encoder and the decoder have the same hierarchical structure, and the scales of feature maps between adjacent levels are twice as large.

[0238] The context loss function in the DRUNet50 model captures the context information in the image and calculates the feature similarity between the real MRI enhanced image and the synthetic MRI enhanced image.

[0239] The image pair dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 80:10:10.

[0240] The enhanced image generation model for meningioma patients adopts a transfer learning approach, uses model parameters trained with a data set, uses TensorFlow as a training framework, and optimizes the parameters in the DRUNet model through Adam until the preset conditions are met and the training is stopped;

[0241] The real image data is predicted, and enhanced MRI slice images of meningioma patients are generated and saved.

[0242] The enhanced image generation model for meningioma patients adds an average pooling layer, a fully connected layer, and a technical layer to prevent network overfitting after the output of the pre-trained model. The activation function uses an exponential linear unit function, presets an initial learning rate, takes minimizing context loss as the goal, and sets the batch size and total number of training rounds of the training process.

[0243] During the training process, the learning rate adopts an adaptive attenuation method. The initial learning rate is set to 0.001, the batch size is set to 16, the total number of training rounds is 80, and when the evaluation index stops rising for 5 times, the learning rate is reduced to 0.25 times of the original value. The minimum supported learning rate is 0.00000001.

[0244] The optimizer uses a stochastic gradient descent method based on first-order adaptive estimation, uses a contextual loss function, and trains 16 samples at a time. The visualization toolkit of the end-to-end open source machine learning platform is used to visualize the model training.

[0245] The model parameters are the optimal model parameters for the validation set data.

[0246] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for automatically generating enhanced images from plain scan images of patients with meningioma, characterized in that: include: Step S1: converting the format of the original image file of the meningioma patient; Step S2: sorting and dividing the meningioma patient image files that have completed format conversion into a training set and a validation set; Step S3: performing image preprocessing on the images of the training set and the validation set; Step S4: performing image data enhancement on the training set data; Step S5: input the images of the training set after image data enhancement into the model training and verify them through the verification set to obtain the trained model; Step S6: Input the meningioma CT slice images of the test set into the trained model for training, post-processing and saving the generated results.

2. The method for automatically generating enhanced images from plain scan images of patients with meningioma according to claim 1, characterized in that: In step S1, the original image files of the meningioma patients collected are converted into a neuroimaging informatics technology program format using MRIcron software, recorded as NifTI format; The original image files of the meningioma patient include computer tomography scans and magnetic resonance enhanced image files in DICOM format, which are recorded as CT and MRI enhanced image files; In step S2, the medical image paired data set of meningioma uses image information corresponding to T1CE of CT and MRI; CT and MRI imaging devices were used to obtain DICOM format CT data and thin-layer MRI data, respectively, to obtain image data, and then the data was anonymized and desensitized; Use ITK-SNAP software to segment the image data, manually outline the region of interest of the entire tumor, obtain the Label file and review and confirm it; The delineated Label files and the corresponding CT files and T1CE files are organized into paired medical image datasets and divided into training sets and validation sets; In step S6, the test set CT slice images are input into the trained deep learning model, and fake-T1CE slice images are reconstructed and saved by reasoning; The post-processing and saving are performed by reading the image function in the Python image processing standard library, converting it into an array form and storing it in a list, converting the NifTI format file through a simplified interface toolkit of the image analysis software algorithm and data structure, and performing a resampling operation at the same time, and saving the generated NifTI format file in a corresponding folder; The list contains the number of samples, image height, and image width data.

3. The method for automatically generating enhanced images from plain scan images of patients with meningioma according to claim 1, characterized in that: The step S3 comprises: Step S3.1: For the sorted training set and validation set, use the Elastix open source software package to register the CT 3D image to the MRI 3D image space; Step S3.2: read the registered training set and validation set into MATLAB and convert them into two-dimensional slices in MAT format; Step S3.3: resampling the two-dimensional slices to the same matrix size; Step S3.4: Using CT as a template, skull stripping was performed on the two-dimensional slices using MATLAB software; Step S3.5: The CT images of the training set and the validation set are normalized, with the Hounsfield unit cutoff range of 0-100, minus the image grayscale mean, and then divided by the standard deviation; The registered fixed image and floating image are respectively set as CT images or MRI images according to training requirements, and the registration standard is rigid registration; The image data enhancement in step S4 includes performing random predetermined angle rotation, cropping, horizontal and vertical mirroring, horizontal and vertical translation of the pre-processed image data to adjust the predetermined values ​​to the initial predetermined multiples, normalization, subtracting the pixel mean, and dividing by the pixel standard deviation; The predetermined angle is ±15 degrees; The predetermined value is ±32 units; The step S3.3 includes the step S3.3.1: cropping the larger image and performing interpolation processing on the smaller image around the image; Step S3.3.2: Use interpolation to change the image matrix size to 384×384; Step S3.3.3: Histogram equalization is used to improve the contrast between the meningioma area and the surrounding brain tissue in the image, and all data are batch processed through a normalization algorithm.

4. The method for automatically generating enhanced images from plain scan images of brain imaging meningioma patients according to claim 1, characterized in that: In step S5, the constructed deep learning model is input into the processed training set for training; The deep learning model is a DRUNet50 model, including an encoder, a decoder and a loss function; The deep learning model uses the dual residual UNet model as the basic framework, and adjusts ResNet and U-Net to form the encoder and decoder of the image generation model to generate synthetic MR images from CT images. Both the input and target images are composed of single-channel 2D image blocks. A U-shaped structure based on an encoder-decoder structure is used for image generation, and two types of residual networks are constructed; ResNet is used as the encoder and encoder of U-Net to extract complex features layer by layer from the input CT image. The decoding process is converted into a splicing method, and the convolutional features of the decoder are fused with the corresponding features of the encoder; The decoder uses a series of residual networks, inverting the entire convolutional component of the encoder and replacing the encoder's downsampling convolution with nearest neighbor interpolation upsampling, adjusting the stride and kernel size during the deconvolution process; During the encoding process, the deep learning model applies a convolutional layer with a 5x5 filter and a stride of 2 and configures a lower bottleneck block; Down sampling includes multiple ResNet residual blocks, and a lower bottleneck block is configured in the DOWN2, DOWN3, DOWN4, and D-CENTER layers with 2, 3, 5, and 2 bottleneck blocks respectively; In the decoding stage, the deep learning model configures 2, 3, 5, and 2 bottleneck blocks with an upper bottleneck block in the U-CENTER, UP1, UP2, and UP3 layers, respectively, and ends with an upper convolution layer; The upper bottleneck block fuses the feature map of the corresponding level in the encoding path with the feature map in the decoding path through a skip connection; The deep learning model is a DRUNet50 architecture with 50 convolutional layers, using a contextual loss function, and feature similarity between real MRI enhanced images and synthetic MRI enhanced images; The calculation is to input the real MRI enhanced image and the synthetic MRI enhanced image into the pre-trained VGG-19 network respectively, extract the feature output of the fourth convolution layer of the fifth convolution module before the fully connected layer in the VGG-19 network, calculate the cosine distance between the two feature outputs, and perform normalization and power operations.

5. The method for automatically generating enhanced images from plain scan images of patients with meningioma according to claim 1, characterized in that: The training in step S5 is a deep learning fully supervised method, and the training is performed using the processed training set image data; The step S5 includes a network architecture construction step: constructing a DRUNet50 model as a meningioma patient enhanced image generation model, and forming a U-Net encoder and decoder by modifying the ResNet structure, including 50 convolutional layers; Model training and optimization steps: Use the training set after image preprocessing and image enhancement as input and output, and import the enhanced training set meningioma CT slice images into the meningioma patient enhanced image generation model for screening; Model evaluation step: Calculate the peak signal-to-noise ratio and structural similarity between real MRI enhanced images and synthetic MRI enhanced images to evaluate the performance of the DRUNet50 model; The DRUNet50 model has a four-layer encoder and decoder structure, each layer contains specific modules and feature scales; The encoder and the decoder have the same hierarchical structure, and the scales of feature maps between adjacent levels are twice as large; The context loss function in the DRUNet50 model captures the context information in the image and calculates the feature similarity between the real MRI enhanced image and the synthetic MRI enhanced image; The meningioma patient image files that have completed format conversion are organized into image pair data sets and randomly divided into a training set, a validation set, and a test set in a ratio of 80:10:10; The enhanced image generation model for meningioma patients adopts a transfer learning approach, uses model parameters trained with a data set, uses TensorFlow as a training framework, and optimizes the parameters in the DRUNet model through Adam until the preset conditions are met and the training is stopped; Predict the real image data, generate and save enhanced MRI slice images of meningioma patients; The enhanced image generation model for meningioma patients adds an average pooling layer, a fully connected layer, and a technical layer to prevent network overfitting after the output of the pre-trained model. The activation function uses an exponential linear unit function, presets an initial learning rate, takes minimizing context loss as the goal, and sets the batch size and total number of training rounds of the training process. During the training process, the learning rate adopts an adaptive attenuation method. The initial learning rate is set to 0.001, the batch size is set to 16, the total number of training rounds is 80, and when the evaluation index stops rising for 5 times, the learning rate is reduced to 0.25 times of the original value. The minimum supported learning rate is 0.00000001. The optimizer uses a stochastic gradient descent method based on first-order adaptive estimation, uses a contextual loss function, and trains 16 samples at a time. The visualization toolkit of the end-to-end open source machine learning platform is used to visualize the model training. The model parameters are the optimal model parameters for the validation set data.

6. A system for automatically generating enhanced images from plain scan images of patients with meningioma, characterized in that: include: Data processing unit, meningioma image generation unit, post-processing unit; The data processing unit performs format conversion on the original image file of the meningioma patient; Perform image preprocessing on images of the medical image paired dataset; The meningioma patient image files that have completed format conversion are organized into a paired dataset of medical images of meningioma; Performing image data enhancement on the image data of the image pair data set that has completed the preprocessing; The meningioma image generation unit trains the processed image paired data set to generate and save slice images of meningioma patients; The post-processing unit post-processes and saves the test set and subsequently newly acquired slice images of meningioma patients.

7. The system for automatically generating enhanced images from plain scan images of patients with meningioma according to claim 6, characterized in that: The original image files of the meningioma patient include computer tomography scans and magnetic resonance enhanced image files in DICOM format, which are recorded as CT and MRI enhanced image files; CT and MRI imaging devices were used to obtain DICOM format CT data and thin-layer MRI data, respectively, to obtain image data, and then the data was anonymized and desensitized; The format conversion uses MRIcron software to convert the original image files of meningioma patients into the Neuro-Imaging Informatics Technology Initiative format, recorded as NifTI format; In the image preprocessing, the medical image paired data set of meningioma uses image information corresponding to T1CE of CT and MRI; Use ITK-SNAP software to segment the image data, manually outline the region of interest of the entire tumor, obtain the Label file and review and confirm it; The delineated Label files and the corresponding CT files and T1CE files are organized into medical image paired data sets; The post-processing unit applies the trained deep learning model to perform inference reconstruction and storage of magnetic resonance enhanced image post-processing on the slice images of meningioma patients in the test set; The slice images of meningioma patients include original slice images of meningioma and MRI enhanced slice images generated by meningioma; The post-processing and saving are performed by reading the image function in the Python image processing standard library, converting it into an array form and storing it in a list, converting the NifTI format file through a simplified interface toolkit of the image analysis software algorithm and data structure, and performing a resampling operation at the same time, and saving the generated NifTI format file in a corresponding folder; The list contains the number of samples, image height, and image width data.

8. The system for automatically generating enhanced images from plain scan images of patients with meningioma according to claim 6, characterized in that: The data processing unit comprises: Module M3.1: Use the Elastix open source software package to register the CT 3D images in the collated medical image pair dataset to the MRI 3D image space; The registered fixed image and floating image are respectively set as CT images or MRI images according to training requirements, and the registration standard is rigid registration; Module M3.2: Read the registered medical image paired data sets into MATLAB and convert them into two-dimensional slices in MAT format; Module M3.3: Resample 2D slices to the same matrix size; Module M3.4: Use MATLAB software to perform skull stripping on two-dimensional slices using CT as a template; Module M3.5: Normalize the CT images of the paired medical image dataset, with the Hounsfield unit cutoff range of 0-100, subtract the image grayscale mean, and then divide by the standard deviation; The image data enhancement includes performing random predetermined angle rotation, cropping, horizontal and vertical mirroring, horizontal and vertical translation of the pre-processed image data to adjust the predetermined values ​​to the initial predetermined multiples, normalization, subtracting the pixel mean, and dividing by the pixel standard deviation; The predetermined angle is ±15 degrees; The predetermined value is ±32 units; The module M3.3 includes a module M3.3.1: cropping the larger image and performing interpolation processing on the smaller image around the image; Module M3.3.2: Change the image matrix size to 384×384 using interpolation; Module M3.3.3: Use histogram equalization to improve the contrast between the meningioma area and the surrounding brain tissue in the image, and batch process all data through the normalization algorithm.

9. The system for automatically generating enhanced images from plain scan images of patients with meningioma according to claim 6, characterized in that: The meningioma image generation unit inputs the processed image paired data set into the constructed deep learning model for training; The deep learning model is a DRUNet50 model, including an encoder, a decoder and a loss function; The deep learning model uses the dual residual UNet model as the basic framework, and adjusts ResNet and U-Net to form the encoder and decoder of the image generation model to generate synthetic MR images from CT images. Both the input and target images are composed of single-channel 2D image blocks. A U-shaped structure based on an encoder-decoder structure is used for image generation, and two types of residual networks are constructed; ResNet is used as the encoder and encoder of U-Net to extract complex features layer by layer from the input CT image. The decoding process is converted into a splicing method, and the convolutional features of the decoder are fused with the corresponding features of the encoder; The decoder uses a series of residual networks, inverting the entire convolutional component of the encoder and replacing the encoder's downsampling convolution with nearest neighbor interpolation upsampling, adjusting the stride and kernel size during the deconvolution process; During the encoding process, the deep learning model applies a convolutional layer with a 5x5 filter and a stride of 2 and configures a lower bottleneck block; Down sampling includes multiple ResNet residual blocks, and a lower bottleneck block is configured in the DOWN2, DOWN3, DOWN4, and D-CENTER layers with 2, 3, 5, and 2 bottleneck blocks respectively; In the decoding stage, the deep learning model configures 2, 3, 5, and 2 bottleneck blocks with an upper bottleneck block in the U-CENTER, UP1, UP2, and UP3 layers, respectively, and ends with an upper convolution layer; The upper bottleneck block fuses the feature map of the corresponding level in the encoding path with the feature map in the decoding path through a skip connection; The deep learning model is a DRUNet50 architecture with 50 convolutional layers, using a contextual loss function, and feature similarity between real MRI enhanced images and synthetic MRI enhanced images; The calculation is to input the real MRI enhanced image and the synthetic MRI enhanced image into the pre-trained VGG-19 network respectively, extract the feature output of the fourth convolution layer of the fifth convolution module before the fully connected layer in the VGG-19 network, calculate the cosine distance between the two feature outputs, and perform normalization and power operations.

10. The system for automatically generating enhanced images from plain scan images of patients with meningioma according to claim 6, characterized in that: The training is a deep learning fully supervised method, using processed image data for training, and using a trained meningioma patient enhanced image generation model to generate and save slice images of meningioma patients; The meningioma image generation unit includes a network architecture building module: constructing a DRUNet50 model, which forms an encoder and a decoder of U-Net by modifying the ResNet structure, including 50 convolutional layers; Model training and optimization module: using the slice images of meningioma patients after image preprocessing and image enhancement as input and output, importing the enhanced slice images of meningioma patients into the meningioma patient enhanced image generation model for screening; Model evaluation module: Calculate the peak signal-to-noise ratio and structural similarity between real MRI enhanced images and synthetic MRI enhanced images to evaluate the performance of the DRUNet50 model; The DRUNet50 model has a four-layer encoder and decoder structure, each layer contains specific modules and feature scales; The encoder and the decoder have the same hierarchical structure, and the scales of feature maps between adjacent levels are twice as large; The context loss function in the DRUNet50 model captures the context information in the image and calculates the feature similarity between the real MRI enhanced image and the synthetic MRI enhanced image; The image pair dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 80:10:10; The enhanced image generation model for meningioma patients adopts a transfer learning approach, uses model parameters trained with a data set, uses TensorFlow as a training framework, and optimizes the parameters in the DRUNet model through Adam until the preset conditions are met and the training is stopped; Predict the real image data, generate and save enhanced MRI slice images of meningioma patients; The enhanced image generation model for meningioma patients adds an average pooling layer, a fully connected layer, and a technical layer to prevent network overfitting after the output of the pre-trained model. The activation function uses an exponential linear unit function, presets an initial learning rate, takes minimizing context loss as the goal, and sets the batch size and total number of training rounds of the training process. During the training process, the learning rate adopts an adaptive attenuation method. The initial learning rate is set to 0.001, the batch size is set to 16, the total number of training rounds is 80, and when the evaluation index stops rising for 5 times, the learning rate is reduced to 0.25 times of the original value. The minimum supported learning rate is 0.00000001. The optimizer uses a stochastic gradient descent method based on first-order adaptive estimation, uses a contextual loss function, and trains 16 samples at a time. The visualization toolkit of the end-to-end open source machine learning platform is used to visualize the model training. The model parameters are the optimal model parameters for the validation set data.

Images