A deep learning-based MRI image and CT image conversion method and terminal

Abstract

The present invention discloses a method and terminal for converting MRI images and CT images based on deep learning. The method comprises the following steps: obtaining a training MRI image and a training CT image, performing N4 bias correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image; training and verifying an initial fully convolutional neural network model based on the preprocessed training MRI image and the preprocessed training CT image to obtain a final fully convolutional neural network model; obtaining an image to be converted, and inputting the image to be converted into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted, thereby improving the accuracy of the conversion between MRI images and CT images.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing technology, and in particular to a method and terminal for converting MRI images and CT images based on deep learning. Background Art

[0002] Currently, commonly used medical images in clinical practice include MRI (Magnetic Resonance Imaging) and CT (Computed Tomography). The advantage of MRI is that the scanning process does not emit ionizing radiation, thus minimizing the impact on the human body. Furthermore, the images have high tissue resolution and are free of bone artifacts. However, the disadvantages are that the examination time is long, the spatial resolution is low, and MRI is not suitable for patients with metal implants. CT images have the advantage of providing the density information necessary for radiotherapy dose planning, with high spatial resolution and simple operation. However, the disadvantages are poor contrast for soft tissue imaging and the presence of ionizing radiation during the scanning process, which can affect the human body. Simultaneously performing CT and MRI can place a significant burden on the patient's health and finances. Therefore, achieving the transformation of MRI and CT images, so that they complement each other, is of great clinical significance.

[0003] In the prior art, statistical learning methods are used to establish a mapping model between MRI voxel intensity values ​​and CT grayscale values. However, this model requires manual feature extraction, and inaccurate feature extraction can easily lead to misconversion problems. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and terminal for converting MRI images and CT images based on deep learning, which can improve the accuracy of the conversion between MRI images and CT images.

[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is:

[0006] A deep learning-based method for converting MRI images to CT images, comprising:

[0007] Acquire a training MRI image and a training CT image, and perform N4 bias correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image;

[0008] Training and validating an initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a final fully convolutional neural network model;

[0009] An image to be converted is obtained, and the image to be converted is input into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted.

[0010] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0011] A deep learning-based MRI image and CT image conversion terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0012] Acquire a training MRI image and a training CT image, and perform N4 bias correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image;

[0013] Training and validating an initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a final fully convolutional neural network model;

[0014] An image to be converted is obtained, and the image to be converted is input into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted.

[0015] The beneficial effects of the present invention are that: during image preprocessing, N4 bias correction and histogram matching are performed on MRT images and CT images, which can avoid noise influence and model design redundancy, and improve the efficiency of subsequent image conversion. Different from the existing statistical learning method, the initial full convolutional neural network model is trained and verified based on the preprocessed MRT image and the preprocessed CT image to obtain the final full convolutional neural network model. Finally, the image to be converted is input into the final full convolutional neural network model to obtain a synthetic image, thereby realizing the mutual conversion between MRI images and CT images. Using the full convolutional neural network structure, since the network has no fully connected layer and only uses the convolution layer, the sampling output is combined with the high-resolution features of the entire model, and the sequential convolution layer can produce more accurate output, thereby improving the accuracy of the conversion between MRI images and CT images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of a method for converting MRI images to CT images based on deep learning according to an embodiment of the present invention;

[0017] Figure 2 Schematic diagram of the structure of a deep learning-based MRI image and CT image conversion terminal according to an embodiment of the present invention;

[0018] Figure 3-Figure 6Schematic diagram of the image conversion process from the initial generation to the optimal model in the deep learning-based MRI image and CT image conversion method according to an embodiment of the present invention;

[0019] Figure 7 Schematic diagram of the network structure of a deep learning-based MRI image and CT image conversion method in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0021] Please refer to Figure 1 , an embodiment of the present invention provides a method for converting MRI images to CT images based on deep learning, comprising:

[0022] Acquire a training MRI image and a training CT image, and perform N4 bias correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image;

[0023] Training and validating an initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a final fully convolutional neural network model;

[0024] An image to be converted is obtained, and the image to be converted is input into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted.

[0025] From the above description, it can be seen that the beneficial effect of the present invention is that: during image preprocessing, N4 bias correction and histogram matching are performed on MRT images and CT images, which can avoid noise influence and model design redundancy, and improve the efficiency of subsequent image conversion. Different from the existing statistical learning method, the initial full convolutional neural network model is trained and verified based on the preprocessed MRT image and the preprocessed CT image to obtain the final full convolutional neural network model. Finally, the image to be converted is input into the final full convolutional neural network model to obtain a synthetic image, thereby realizing the mutual conversion between MRI images and CT images. Using the full convolutional neural network structure, since the network has no fully connected layer and only uses the convolution layer, the sampling output is combined with the high-resolution features of the entire model, and the sequential convolution layer can produce a more accurate output, thereby improving the accuracy of the conversion between MRI images and CT images.

[0026] Furthermore, performing N4 offset correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image includes:

[0027] Acquire a training MRI grayscale image and a training CT grayscale image according to the training MRI image and the training CT image;

[0028] Performing floating-point image conversion on the training MRI grayscale image and the training CT grayscale image to obtain converted training MRI grayscale image and training CT grayscale image;

[0029] Correcting the converted training MRI grayscale image and training CT grayscale image using N4 bias correction to obtain corrected training MRI grayscale image and training CT grayscale image;

[0030] Histogram matching is performed on the corrected training MRI grayscale image and the training CT grayscale image to obtain a preprocessed training MRI image and a preprocessed training CT image.

[0031] From the above description, it can be seen that obtaining the grayscale image of the training image and then converting it into a pseudo-floating-point image is beneficial to subsequent image processing and improves processing quality and efficiency. The use of N4 bias correction can remove noise in the training image, and the use of histogram matching can keep the tones of the two images consistent, thereby realizing the subsequent conversion between MRI images and CT images.

[0032] Furthermore, the initial fully convolutional neural network model is trained and verified based on the preprocessed training MRI image and the preprocessed training CT image to obtain a final fully convolutional neural network model, including:

[0033] Training and validating an initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a first final fully convolutional neural network model and a second final fully convolutional neural network model;

[0034] The images to be converted include MRI images to be converted and CT images to be converted;

[0035] Inputting the image to be converted into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted includes:

[0036] Inputting the MRI image to be converted into the first final fully convolutional neural network model to obtain a pseudo CT image corresponding to the MRI image to be converted;

[0037] The CT image to be converted is input into the second final fully convolutional neural network model to obtain a pseudo MRI image corresponding to the CT image to be converted.

[0038] As can be seen from the above description, during training, the same network model is trained twice based on the preprocessed training MRI images and the preprocessed training CT images, respectively, to obtain two conversion models, one for converting MRI images into pseudo-CT images, and the other for converting CT images into pseudo-MRI images, thereby realizing the mutual conversion between CT images and MRI images.

[0039] Furthermore, the training and verification of the initial fully convolutional neural network model based on the preprocessed training MRI image and the preprocessed training CT image to obtain the first final fully convolutional neural network model and the second final fully convolutional neural network model includes:

[0040] Inputting the preprocessed training MRI image into the encoder network of the initial fully convolutional neural network model to obtain a first low-resolution feature map and a first high-resolution feature map;

[0041] Inputting the first low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the first high-resolution feature map to obtain a training pseudo CT image;

[0042] Performing loss calculation on the training pseudo CT image and the preprocessed training CT image to obtain a first error value;

[0043] Training the initial fully convolutional neural network model according to the first error value and a preset number of training rounds to obtain a first final fully convolutional neural network model;

[0044] Inputting the preprocessed training CT image into the encoder network of the initial fully convolutional neural network model to obtain a second low-resolution feature map and a second high-resolution feature map;

[0045] Inputting the low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the second high-resolution feature map to obtain a training pseudo MRI image;

[0046] Performing loss calculation on the training pseudo MRI image and the preprocessed training MRI image to obtain a second error value;

[0047] The initial fully convolutional neural network model is trained according to the second error value and the preset training rounds to obtain a second final fully convolutional neural network model.

[0048] From the above description, it can be seen that during training, the training MRI image is input into the initial fully convolutional neural network model, and after processing by the encoder network and the decoder network, the first final fully convolutional neural network model is finally obtained. This network model can be used to realize the conversion of MRI images into pseudo CT images. The training CT image is input into the initial fully convolutional neural network model, and after processing by the encoder network and the decoder network, the second final fully convolutional neural network model is finally obtained. This network model can be used to realize the conversion of CT images into pseudo MRI images. The encoder of the network is initialized by the pre-trained VGG16 weights. The idea of ​​transfer learning is adopted to adapt to new tasks by fine-tuning the entire network. Therefore, there is no need to start training from scratch during training, which saves a lot of computing power and improves the efficiency of model training.

[0049] Furthermore, the encoder network includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, a maximum pooling layer, a third activation layer and a downsampling layer arranged in sequence;

[0050] The decoder network includes an upsampling layer and a third convolutional layer arranged in sequence.

[0051] As can be seen from the above description, there are no fully connected layers in the network structure, only convolutional layers. Each standard convolutional layer is followed by an activation layer. The sampled output can be combined with the high-resolution features of the entire model to produce more accurate output, thereby improving the accuracy of the conversion between MRI images and CT images.

[0052] Please refer to Figure 2 A deep learning-based MRI image and CT image conversion terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0053] Acquire a training MRI image and a training CT image, and perform N4 bias correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image;

[0054] Training and validating an initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a final fully convolutional neural network model;

[0055] An image to be converted is obtained, and the image to be converted is input into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted.

[0056] From the above description, it can be seen that the beneficial effect of the present invention is that: during image preprocessing, N4 bias correction and histogram matching are performed on MRT images and CT images, which can avoid noise influence and model design redundancy, and improve the efficiency of subsequent image conversion. Different from the existing statistical learning method, the initial full convolutional neural network model is trained and verified based on the preprocessed MRT image and the preprocessed CT image to obtain the final full convolutional neural network model. Finally, the image to be converted is input into the final full convolutional neural network model to obtain a synthetic image, thereby realizing the mutual conversion between MRI images and CT images. Using the full convolutional neural network structure, since the network has no fully connected layer and only uses the convolution layer, the sampling output is combined with the high-resolution features of the entire model, and the sequential convolution layer can produce a more accurate output, thereby improving the accuracy of the conversion between MRI images and CT images.

[0057] Furthermore, performing N4 offset correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image includes:

[0058] Acquire a training MRI grayscale image and a training CT grayscale image according to the training MRI image and the training CT image;

[0059] Performing floating-point image conversion on the training MRI grayscale image and the training CT grayscale image to obtain converted training MRI grayscale image and training CT grayscale image;

[0060] Correcting the converted training MRI grayscale image and training CT grayscale image using N4 bias correction to obtain corrected training MRI grayscale image and training CT grayscale image;

[0061] Histogram matching is performed on the corrected training MRI grayscale image and the training CT grayscale image to obtain a preprocessed training MRI image and a preprocessed training CT image.

[0062] From the above description, it can be seen that obtaining the grayscale image of the training image and then converting it into a pseudo-floating-point image is beneficial to subsequent image processing and improves processing quality and efficiency. The use of N4 bias correction can remove noise in the training image, and the use of histogram matching can keep the tones of the two images consistent, thereby realizing the subsequent conversion between MRI images and CT images.

[0063] Furthermore, the initial fully convolutional neural network model is trained and verified based on the preprocessed training MRI image and the preprocessed training CT image to obtain a final fully convolutional neural network model, including:

[0064] Training and validating an initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a first final fully convolutional neural network model and a second final fully convolutional neural network model;

[0065] The images to be converted include MRI images to be converted and CT images to be converted;

[0066] Inputting the image to be converted into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted includes:

[0067] Inputting the MRI image to be converted into the first final fully convolutional neural network model to obtain a pseudo CT image corresponding to the MRI image to be converted;

[0068] The CT image to be converted is input into the second final fully convolutional neural network model to obtain a pseudo MRI image corresponding to the CT image to be converted.

[0069] As can be seen from the above description, during training, the same network model is trained twice based on the preprocessed training MRI images and the preprocessed training CT images, respectively, to obtain two conversion models, one for converting MRI images into pseudo-CT images, and the other for converting CT images into pseudo-MRI images, thereby realizing the mutual conversion between CT images and MRI images.

[0070] Furthermore, the training and verification of the initial fully convolutional neural network model based on the preprocessed training MRI image and the preprocessed training CT image to obtain the first final fully convolutional neural network model and the second final fully convolutional neural network model includes:

[0071] Inputting the preprocessed training MRI image into the encoder network of the initial fully convolutional neural network model to obtain a first low-resolution feature map and a first high-resolution feature map;

[0072] Inputting the first low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the first high-resolution feature map to obtain a training pseudo CT image;

[0073] Performing loss calculation on the training pseudo CT image and the preprocessed training CT image to obtain a first error value;

[0074] Training the initial fully convolutional neural network model according to the first error value and a preset number of training rounds to obtain a first final fully convolutional neural network model;

[0075] Inputting the preprocessed training CT image into the encoder network of the initial fully convolutional neural network model to obtain a second low-resolution feature map and a second high-resolution feature map;

[0076] Inputting the low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the second high-resolution feature map to obtain a training pseudo MRI image;

[0077] Performing loss calculation on the training pseudo MRI image and the preprocessed training MRI image to obtain a second error value;

[0078] The initial fully convolutional neural network model is trained according to the second error value and the preset training rounds to obtain a second final fully convolutional neural network model.

[0079] From the above description, it can be seen that during training, the training MRI image is input into the initial fully convolutional neural network model, and after processing by the encoder network and the decoder network, the first final fully convolutional neural network model is finally obtained. This network model can be used to realize the conversion of MRI images into pseudo CT images. The training CT image is input into the initial fully convolutional neural network model, and after processing by the encoder network and the decoder network, the second final fully convolutional neural network model is finally obtained. This network model can be used to realize the conversion of CT images into pseudo MRI images. The encoder of the network is initialized by the pre-trained VGG16 weights. The idea of ​​transfer learning is adopted to adapt to new tasks by fine-tuning the entire network. Therefore, there is no need to start training from scratch during training, which saves a lot of computing power and improves the efficiency of model training.

[0080] Furthermore, the encoder network includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, a maximum pooling layer, a third activation layer and a downsampling layer arranged in sequence;

[0081] The decoder network includes an upsampling layer and a third convolutional layer arranged in sequence.

[0082] As can be seen from the above description, there are no fully connected layers in the network structure, only convolutional layers. Each standard convolutional layer is followed by an activation layer. The sampled output can be combined with the high-resolution features of the entire model to produce more accurate output, thereby improving the accuracy of the conversion between MRI images and CT images.

[0083] The above-mentioned deep learning-based MRI image and CT image conversion method and terminal of the present invention can be applied to the mutual conversion of MRI images and CT images, and is described below through specific embodiments:

[0084] Example 1

[0085] Please refer to Figure 1, a method for converting MRI images to CT images based on deep learning in this embodiment includes:

[0086] S1. Acquire a training MRI image and a training CT image, and perform N4 bias correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image;

[0087] In S1, performing N4 offset correction and histogram matching on the training MRI image and the training CT image to obtain the preprocessed training MRI image and the preprocessed training CT image includes:

[0088] S11, acquiring a training MRI grayscale image and a training CT grayscale image according to the training MRI image and the training CT image;

[0089] Specifically, set corresponding file paths for the training MRI image source_A and the training CT image source_B respectively, cut the training MRI image source_A and the training CT image source_B into the same size, name them a_img and b_img, and save the training MRI image a_img and the training CT image b_img;

[0090] Use OpenCV (Open Source Computer Vision Library, a cross-platform computer vision library based on the release) to obtain the training MRI grayscale image and the training CT grayscale image according to the cropped training MRI image a_img and the cropped training CT image b_img;

[0091] S12, performing floating-point image conversion on the training MRI grayscale image and the training CT grayscale image to obtain converted training MRI grayscale image and training CT grayscale image;

[0092] Specifically, N4itk (a medical image processing tool) is used to convert the training MRI grayscale image and the training CT grayscale image into float32 floating-point images to obtain the converted training MRI grayscale image and the training CT grayscale image;

[0093] S13, correcting the converted training MRI grayscale image and training CT grayscale image using N4 offset correction to obtain corrected training MRI grayscale image and training CT grayscale image;

[0094] N4 bias correction is used, with the fitting_levels parameter set to 4 and the number of iterations to 200. Image pixels are normalized to 0-255, and the encoding format is unit8. The converted training MRI grayscale image and training CT grayscale image are corrected, and the corrected training MRI grayscale image and training CT grayscale image and the original image are returned. This process can be visualized using OpenCV;

[0095] The fitting_levels is a contraction parameter of the N4 bias correction. MRI images are affected by bias field distortion, which often causes intensity fluctuations. The fitting_levels is used to set a threshold to ensure the effect of correction and reconstruction. The value of fitting_levels can be set according to actual conditions. In this embodiment, fitting_levels = 4.

[0096] The number of iterations is the number of times the network is trained;

[0097] The reason for normalizing image pixels to 0-255 is that the pixel range of CT and MRI images is very large, such as -1000 to 3000. The darker the area, the lower the value. When processing these images, we need to normalize the pixels to 0-255, ensuring that values ​​below 0 are normalized to 0 and values ​​greater than 255 are normalized to 255. This ensures that the pixel values ​​are between 0 and 255, and the image data is still valid, so that the neural network can be better trained.

[0098] The normalization of the image pixels to the range of 0-255 specifically includes: if the corresponding image pixel value is less than 0, it is reset to 0; if the corresponding image pixel value is greater than 255, it is reset to 255, and the rest remain unchanged;

[0099] Specifically, the N4 bias correction code is as follows:

[0100] Ct_img=img.copy(), that is, use the copy image function to copy the CT image and save it to Ct_img;

[0101] Mr_img = sitk.GetImageFromArray(img), that is, use the GetImageFromArray function of the sitk tool to read the MRT image sequence and save it to Mr_img;

[0102] Mask_img = sitk.OtsuThreshold(mr_img), that is, using Otsu (a histogram-based automatic threshold filter) to create a histogram and use a heuristic method to determine the threshold to perform threshold segmentation on the MRI image;

[0103] Mr_img = sitk.Cast(mr_img), that is, using the cast function to convert the data type of the MRI image;

[0104] Among them, ct_img represents CT image, mr_img represents MRI image, Sitk uses simpleITK to process images, GetImageFromArray is a function that converts arrays into sitk images, and sitk.OtsuThreshold is a threshold segmentation function;

[0105] N4 bias correction parameters are as follows:

[0106] corrector = sitk.N4BiasFieldCorrectionImageFilter(), the N4 bias correction function. The function parameters include image dimension, input image, mask file, weight, shrinkage coefficient, convergence coefficient, B-spline fitting parameters, and histogram sharpening coefficient. Finally, the final correction image is calculated and output. The above parameters are all default parameters.

[0107] corrector.SetMaximumNumberOfIterations([iters]*num_fitting_levels), that is, set a maximum number of iterations;

[0108] Where, corrector is the corrector, N4BiasFieldCorrectionImageFilter() is the N4 bias field correction image filter, SetMaximumNumberOfIterations() is the maximum iteration parameter function, num_fitting_levels is the numerical fitting level, num_fitting_levels=4, iters is the number of iterations, iters=200;

[0109] S14, performing histogram matching on the corrected training MRI grayscale image and the training CT grayscale image to obtain a preprocessed training MRI image and a preprocessed training CT image;

[0110] Specifically, set the parameters for histogram matching, where H is the height of the feature map, W is the width of the feature map, pixels = H × W, ref_img is the image read using OpenCV, and hist_img is the image after histogram matching, that is, the preprocessed training MRI image and the preprocessed training CT image;

[0111] Calculate the cumulative histogram, i.e. cumulative histogram(hist_img, ref_img);

[0112] Calculate the probability density of the image, calculate the cumulative distribution function, calculate the output grayscale corresponding to each grayscale, and finally map it into the preprocessed training MRI image and the preprocessed training CT image;

[0113] The histogram matching parameters are as follows:

[0114] Normalization(img) = img / pixels;

[0115] Pixels = h × w, h is the height of the image, w is the width of the image;

[0116] Cumulative histogram cumulative_histogram(img);

[0117] Suppose a digital image with a grayscale range of [0, L-1], the discrete function h(rk) = nk, rk is the k-th grayscale value, nk is the number of pixels with grayscale rk in the image, M and N are the number of bits in the rows and columns of the image, MN is the total divisor of the image pixels, and after normalization, p(rk) = nk / MN, that is, p(rk) is the probability estimate of grayscale rk appearing in the image;

[0118] step:

[0119] Using continuous grayscale, x and y represent the grayscale levels of the input and output images, and set the random variable of variable s;

[0120]

[0121] The above formula is the continuous form of histogram equalization, where w is the integral pseudo variable, T(r) represents the grayscale transformation function, and r represents the grayscale level of the input image, that is, the above x, p r represents the probability density function of r, and d represents the differential symbol;

[0122] Define another random variable z so that the following relationship exists:

[0123]

[0124] Where v is a pseudo integral variable, making G(z) = T(r), that is:

[0125] z=G -1 [T(r)]=G -1 (s);

[0126] In this way, an image with gray levels having a specified probability density function is obtained from a given image;

[0127] In actual operation, the input image is obtained r(r), that is, according to the gray level r of the input image, the probability density p is calculated by the probability density function r (r), by Calculate s;

[0128] Using the formula Calculate the transformation function G(z), that is, integrate the probability density over the interval 0 to z, and assign the result of the integration to G(z);

[0129] Process the mapping from s to z and calculate z = G -1 (s);

[0130] Use the input image The output image is obtained by equalization, and the pixel value is s;

[0131] Perform inverse mapping on the s value in the equalized image to obtain the corresponding pixels of the output image. The probability density of the output image is the specified probability density function.

[0132] in, The discrete form of is:

[0133] Where r j Represents the grayscale value, n j Represents r j The number of pixels, n r represents the number of pixels of gray level r, and L is the number of gray levels in the image;

[0134] Generate the mask (mask matrix), that is:

[0135] Image=cv2.imread(file,cv2.IMREAD_GRAYSCALE)

[0136] Ct,mr=img[size],img[-size];

[0137] Among them, cv2 indicates that OpenCV is used for processing, cv2.IMREAD_GRAYSCALE indicates that the format of the parsed image is grayscale, and size is the image size;

[0138] Calculate the mask and generate the mask. The mask of the CT image is represented by Masked_ct, and the mask of the MRI image is represented by mask_mr.

[0139] Use bilateral filtering method: cv2.bilateralFilter(image);

[0140] flood filling(image);

[0141] The size needs to be 2 pixels larger than the image, i.e. h+2, w+2;

[0142] Invert the floodfilled image: bitwise_not(image);

[0143] Calculate the maximum mask contour cv2.findContours(image) to complete the mask generation (mask generation). The neural network only processes the pixels within the mask range, just like a mask covering the area to be processed. Using mask generation allows the neural network to focus only on the area covered by the mask, reducing the amount of calculation and reducing network redundancy.

[0144] S2. Training and verifying the initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a final fully convolutional neural network model;

[0145] Specifically, the initial fully convolutional neural network model is trained and verified based on the preprocessed training MRI image and the preprocessed training CT image to obtain a first final fully convolutional neural network model and a second final fully convolutional neural network model;

[0146] In an optional embodiment, the preprocessed training MRI image is used as the image to be processed during training, and the corresponding preprocessed training CT image is used as the labeled data to train and verify the initial fully convolutional neural network to obtain a first final fully convolutional neural network model; the preprocessed training CT image is used as the image to be processed during training, and the corresponding preprocessed training MRI image is used as the labeled data to train and verify the initial fully convolutional neural network to obtain a second final fully convolutional neural network model;

[0147] S3. Acquire an image to be converted, and input the image to be converted into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted;

[0148] The images to be converted include MRI images to be converted and CT images to be converted;

[0149] Inputting the MRI image to be converted into the first final fully convolutional neural network model to obtain a pseudo CT image corresponding to the MRI image to be converted;

[0150] The CT image to be converted is input into the second final fully convolutional neural network model to obtain a pseudo MRI image corresponding to the CT image to be converted.

[0151] Example 2

[0152] Please refer to Figure 1 、 3 -6. This embodiment further defines how to train the first final fully convolutional neural network model and the second final fully convolutional neural network model based on the first embodiment, specifically:

[0153] Step S2 is specifically as follows:

[0154] S21, inputting the preprocessed training MRI image into the encoder network of the initial fully convolutional neural network model to obtain a first low-resolution feature map and a first high-resolution feature map;

[0155] The encoder network includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, a maximum pooling layer, a third activation layer, and a downsampling layer arranged in sequence;

[0156] Specifically, the preprocessed training MRI image is input into the encoder network of the initial fully convolutional neural network model, and sequentially passes through the first convolutional layer, the first activation layer, the second convolutional layer, the second activation layer, the maximum pooling layer, the third activation layer and the downsampling layer of the encoder network to output a first low-resolution feature map and a first high-resolution feature map;

[0157] The encoder network has a height of H, a width of W, and a number of channels of C. The first activation layer, Relu1_1 = conv-layer-train (data, conv1_1), is immediately followed by the first convolutional layer conv1_1, and the second activation layer, Relu1_2 = conv-layer-train (data, conv2_2), is immediately followed by the second convolutional layer conv2_2; where data represents the tensor concatenation function data = tf.concat (C, H, W), the height, width, and number of channels are (256, 256, 64), and conv1_1 represents the output value of the first convolutional layer conv1_1;

[0158] The maximum pooling layer, i.e. Pool1_2 = maxpool(relu1_2), has a height, width, and number of channels of (128, 128, 128). The main purpose is to reduce computational complexity. It uses the maximum pooling method to perform dimensionality reduction in the encoding part to extract deeper features.

[0159] The third activation layer, i.e., Relu2_1=conv-layer(pool1_2,conv2_1), Relu2_1=conv-layer(relu2_1,conv2_2), introduces a nonlinear activation function (ReLU), so that the neural network model can approximate any function. This activation function is used in this part to make the output of some neurons 0, which can properly form the sparsity of the network, reduce the dependence of some parameters, and alleviate the occurrence of overfitting to a certain extent. That is, ReLU realizes that the sparse model can better mine relevant features, fit training data, and have stronger expression ability, thereby improving the quality of the conversion model obtained by training;

[0160] Multiple downsampling is performed in the downsampling layer, namely:

[0161] (64,64,256)

[0162] Pool2_2=max-pool(relu2_2)

[0163] Relu3_1=conv-layer(pool2_2,conv3_1)

[0164] Relu3_2=conv-layer(relu3_1,conv3_2)

[0165] Relu3_3=conv-layer(relu3_2,conv3_3)

[0166] (32,32,512)

[0167] Pool3_3=max-pool(relu3_3)

[0168] Relu4_1=conv-layer(pool3_3,conv4_1)

[0169] Relu4_2=conv-layer(relu4_1,conv4_2)

[0170] Relu4_3=conv-layer(relu4_2,conv4_3)

[0171] (16,16,512)

[0172] Pool4_3=max-pool(relu4_3)

[0173] Relu5_1=conv-layer(pool4_3,conv5_1)

[0174] Relu5_2=conv-layer(relu5_1,conv5_2)

[0175] Relu5_3=conv-layer(relu5_2,conv5_3)

[0176] The downsampling layer consists of two 3×3 convolutional layers (followed by an activation layer) and a 2×2 maximum pooling layer with a stride of 2. The number of channels doubles with each downsampling, i.e., 64 → 128 → 256 → 512.

[0177] The final output is the first low-resolution feature map and the first high-resolution feature map. After multiple downsampling, the first low-resolution feature map can provide the contextual semantic information of the entire image, identify the target area of ​​the conversion, and determine its category.

[0178] S22: Input the first low-resolution feature map into the decoder network of the initial fully convolutional neural network model to perform an upsampling operation and fuse the first high-resolution feature map to obtain a training pseudo CT image;

[0179] The decoder network includes an upsampling layer and a third convolutional layer arranged in sequence;

[0180] Specifically, the first low-resolution feature map is input into the initial fully convolutional neural network model, and is sequentially up-sampled through the upsampling layer and the third convolutional layer of the decoder network and fused with the first high-resolution feature map to output a training pseudo CT image;

[0181] The decoder network uses valid convolution, uses padding to prevent loss of image edge information, and has a stride of 2.

[0182] (16,16,512)

[0183] Relu5_4=conv-layer(relu5_3,output-dim=512)

[0184] Relu5_5=conv-layer(relu5_4,output-dim=512)

[0185] Relu5_6=conv-layer(relu5_5,output-dim=512);

[0186] Among them, output-dim represents the output dimension;

[0187] The upsampling part consists of a 2×2 upsampling convolution layer (followed by an activation layer ReLU) and concatenation operation, which allows the low-resolution image containing high-level abstract features to retain the high-resolution feature map, such as:

[0188] (32,32,512)

[0189] Unpool4_3=unpooling2d(relu5_6)

[0190] Concat4_3=tf.concat(relu4_3,unpool4_3)

[0191] Relu4_4=conv-layer(concat4_3,output-dim=512)

[0192] Relu4_5=conv-layer(relu4_4,output-dim=512)

[0193] Relu4_6=conv-layer(relu4_5,output-dim=512);

[0194] Among them, unpool represents upsampling pooling, and concat represents the tensor splicing function;

[0195] Similarly, a 3×3 upsampling convolution layer is used followed by an activation layer ReLU, the rows and columns are doubled by 2, and deconvolution is used. Each time the upsampling is performed, the number of channels corresponding to the feature extraction part is fused at the same scale. Cropping is required before fusion, and the fusion method is splicing, such as:

[0196] (64,64,256)

[0197] Unpool3_3=unpooling2d(relu4_6)

[0198] Concat3_3=tf.concat(relu3_3,unpool3_3)

[0199] Relu3_4=conv-layer(concat3_3,output-dim=256)

[0200] Relu3_5=conv-layer(relu3_4,output-dim=256)

[0201] Relu3_6=conv-layer(relu3_5,output-dim=256);

[0202] Then use back propagation, the convolution formula Y = Matrix_1(filter)*Matrix_2(image);

[0203] Among them, Matrix_1 and Matrix_2 convert the convolution kernel filter and image image into matrix functions respectively, Y = D*X, X is the input, D is the transposed matrix, Y is the output, and the convolution back propagation is D transposed multiplication;

[0204] (128,128,128)

[0205] Unpool2_2=unpooling2d(relu3_6)

[0206] Concat2_2=tf.concat(relu2_2,unpool2_2)

[0207] Relu2_3=conv-layer(concat2_2,output-dim=128)

[0208] Relu2_4=conv-layer(relu2_3,output-dim=128)

[0209] (256,256,64)

[0210] Unpool1_2=unpooling2d(relu2_4)

[0211] Concat1_2=tf.concat(relu1_2,unpool1_2)

[0212] Relu1_3=conv-layer(concat1_2,output-dim=64)

[0213] Relu1_4=conv-layer(relu1_3,output-dim=64);

[0214] Among them, (128,128,128) and (256,256,64) are the height H, width W, and number of channels C of the image;

[0215] The last step is to generate a feature map through two convolutions, and then use a convolution kernel of 1×1 to obtain the final result through softmax, that is, the training pseudo CT image, such as:

[0216] (256,256,1)

[0217] Pred=last-conv2d(relu1_4,output-dim=1)

[0218] Data_loss=regress_loss(pred,y)

[0219] Optimizer=Adam(learning_rate).minimize(loss);

[0220] Among them, last-conv2d represents the last convolutional layer, that is, the third convolutional layer, Adam represents the optimizer, learning_rate represents the learning rate, minimize represents minimization of loss, y represents the preprocessed training CT image, and Pred represents the predicted value;

[0221] The loss function used in this network is L2, and reg is set to weight decay according to the training iteration, so that the learning rate gradually decreases in continuous iterations;

[0222] S23, performing loss calculation on the training pseudo CT image and the preprocessed training CT image to obtain a first error value;

[0223] S24: Training the initial fully convolutional neural network model according to the first error value and a preset number of training rounds to obtain a first final fully convolutional neural network model;

[0224] Specifically, the above training process can be visualized through a visualization module, which allows direct observation of the error, accuracy, and image conversion process. The visualization module is as follows:

[0225] Tensorboard

[0226] Use the summary.scalar function to read the loss file saved during the training process, namely:

[0227] Tf.summary.scalar(loss)

[0228] Tf.summary.scalar(MAE)

[0229] Tf.summary.scalar(ME)

[0230] Tf.summary.scalar(RMSE)

[0231] Tf.summary.scalar(PCC)

[0232] Tensorboard can visualize the results of the above indicators by reading them;

[0233] Figure 7 The network structure diagram of the first final fully convolutional neural network model is shown. Figure 7 In the figure, Conv3×3 represents 3×3 convolution, BN represents batch normalization, ReLU represents activation layer, Maxpooling represents maximum pooling, Upsampling represents upsampling, Copy.crop represents copy and cropping, Dropout represents dropout rate, and Loss function represents loss function.

[0234] S25, inputting the preprocessed training CT image into the encoder network of the initial fully convolutional neural network model to obtain a second low-resolution feature map and a second high-resolution feature map;

[0235] S26, inputting the low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the second high-resolution feature map to obtain a training pseudo MRI image;

[0236] S27, performing loss calculation on the training pseudo MRI image and the preprocessed training MRI image to obtain a second error value;

[0237] S28. Training the initial fully convolutional neural network model according to the second error value and a preset training round to obtain a second final fully convolutional neural network model;

[0238] The specific details of steps S25-S28 are similar to those of steps S21-S24 and will not be repeated here;

[0239] In the above steps, S21-S24 and S25-S28 are two parallel schemes, which can be executed simultaneously, one by one, or one by one. That is, when it is necessary to obtain a neural network model for converting MRI images into pseudo-CT images, execute steps S21-S24; when it is necessary to obtain a neural network model for converting CT images into pseudo-MRI images, execute steps S25-S28.

[0240] Example 3

[0241] The difference between this embodiment and the first or second embodiment is that some specific parameters for training the initial fully convolutional neural network model are further defined:

[0242] Specify GPU, single or multiple cards are optional, and the selection is based on the image data size, that is, os.environ['CUDA_VISIBLE_DEVICES'] = GPU_id;

[0243] The image batch size Batch_size is set to 8;

[0244] The learning rate learning_rate is set to 0.0001 (1e-4);

[0245] The decay factor weight_decay is set to 1e-4;

[0246] The default training epoch is set to 600;

[0247] Save the model save_model is set to once every 50 epochs;

[0248] The error outputs for training, validation, and testing are MAE (Mean Absolute Error), ME (Mean Error), MSE (Mean Squared Error), and PCC (Pearson Correlation Coefficient). MAE, ME, MSE, and PCC are used as multiple metrics to evaluate the quality of the model from different perspectives.

[0249] The model is saved in model_dir and 6 cross validation items are set to cross_validation=6;

[0250] Some details of data loading are as follows:

[0251] Use numpy (Numerical Python, an open source numerical computing extension of Python) for processing; the training batch Train_batch uses a random method to load image data, namely MRI image batch_x and CT image batch_y; the mask is represented by batch_mask, the validation batch is val_batch,x,y,mask; the test batch is test_batch,x,y,mask;

[0252] like Figure 3-Figure 6 As shown, Figure 3-Figure 6 The image transformation process from the initial generation to the optimal model is shown. It can be seen that Figure 3-Figure 6 The first column of images in the image is the image to be transformed, the second column is the synthetic image of the non-optimal model, and the third column is the synthetic image output by the optimal model.

[0253] Example 4

[0254] Please refer to Figure 2A deep learning-based MRI image and CT image conversion terminal includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, each step of the deep learning-based MRI image and CT image conversion method in Example 1, Example 2, or Example 3 is implemented.

[0255] In summary, the present invention provides a method and terminal for converting MRI images and CT images based on deep learning, which obtains training MRI images and training CT images, and performs N4 bias correction and histogram matching on the training MRI images and training CT images to obtain preprocessed training MRI images and preprocessed training CT images, which can avoid noise influence and model design redundancy and improve the efficiency of subsequent image conversion; based on the preprocessed training MRI images and the preprocessed training CT images, the initial full convolutional neural network model is trained and verified to obtain the first final full convolutional neural network model and the second final full convolutional neural network model, using a full convolutional neural network structure, since the network has no fully connected layer, only uses a convolutional layer, and the sampling output is the same as The high-resolution features of the entire model are combined, and the sequential convolutional layer can produce more accurate outputs; the MRI image to be converted is input into the first final full convolutional neural network model to obtain a pseudo CT image corresponding to the MRI image to be converted, and the CT image to be converted is input into the second final full convolutional neural network model to obtain a pseudo MRI image corresponding to the CT image to be converted. The same network model is trained twice based on the preprocessed training MRI image and the preprocessed training CT image to obtain two conversion models, one for converting MRI images into pseudo CT images, and the other for converting CT images into pseudo MRI images, thereby realizing the mutual conversion between CT images and MRI images, thereby improving the accuracy of the conversion between MRI images and CT images.

[0256] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for converting MRI images to CT images based on deep learning, characterized in that: include: Acquire a training MRI image and a training CT image, and perform N4 bias correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image; Training and validating an initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a final fully convolutional neural network model; Acquire an image to be converted, and input the image to be converted into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted; The initial fully convolutional neural network model is trained and verified based on the preprocessed training MRI image and the preprocessed training CT image to obtain a final fully convolutional neural network model, including: Inputting the preprocessed training MRI image into the encoder network of the initial fully convolutional neural network model to obtain a first low-resolution feature map and a first high-resolution feature map; Inputting the first low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the first high-resolution feature map to obtain a training pseudo CT image; Performing loss calculation on the training pseudo CT image and the preprocessed training CT image to obtain a first error value; Training the initial fully convolutional neural network model according to the first error value and a preset number of training rounds to obtain a first final fully convolutional neural network model; Inputting the preprocessed training CT image into the encoder network of the initial fully convolutional neural network model to obtain a second low-resolution feature map and a second high-resolution feature map; Inputting the low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the second high-resolution feature map to obtain a training pseudo MRI image; Performing loss calculation on the training pseudo MRI image and the preprocessed training MRI image to obtain a second error value; The initial fully convolutional neural network model is trained according to the second error value and the preset training rounds to obtain a second final fully convolutional neural network model.

2. The method for converting MRI images to CT images based on deep learning according to claim 1, characterized in that: The performing N4 offset correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image includes: Acquire a training MRI grayscale image and a training CT grayscale image according to the training MRI image and the training CT image; Performing floating-point image conversion on the training MRI grayscale image and the training CT grayscale image to obtain converted training MRI grayscale image and training CT grayscale image; Correcting the converted training MRI grayscale image and training CT grayscale image using N4 bias correction to obtain corrected training MRI grayscale image and training CT grayscale image; Histogram matching is performed on the corrected training MRI grayscale image and the training CT grayscale image to obtain a preprocessed training MRI image and a preprocessed training CT image.

3. The method for converting MRI images to CT images based on deep learning according to claim 1, characterized in that: The initial fully convolutional neural network model is trained and verified based on the preprocessed training MRI image and the preprocessed training CT image to obtain a final fully convolutional neural network model, including: The images to be converted include MRI images to be converted and CT images to be converted; Inputting the image to be converted into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted includes: Inputting the MRI image to be converted into the first final fully convolutional neural network model to obtain a pseudo CT image corresponding to the MRI image to be converted; The CT image to be converted is input into the second final fully convolutional neural network model to obtain a pseudo MRI image corresponding to the CT image to be converted.

4. The method for converting MRI images to CT images based on deep learning according to claim 1, characterized in that: The encoder network includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, a maximum pooling layer, a third activation layer and a downsampling layer arranged in sequence; The decoder network includes an upsampling layer and a third convolutional layer arranged in sequence.

5. A deep learning-based MRI image and CT image conversion terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Acquire a training MRI image and a training CT image, and perform N4 bias correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image; Training and validating an initial fully convolutional neural network model based on the preprocessed training MRI images and the preprocessed training CT images to obtain a final fully convolutional neural network model; Acquire an image to be converted, and input the image to be converted into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted; The initial fully convolutional neural network model is trained and verified based on the preprocessed training MRI image and the preprocessed training CT image to obtain a final fully convolutional neural network model, including: Inputting the preprocessed training MRI image into the encoder network of the initial fully convolutional neural network model to obtain a first low-resolution feature map and a first high-resolution feature map; Inputting the first low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the first high-resolution feature map to obtain a training pseudo CT image; Performing loss calculation on the training pseudo CT image and the preprocessed training CT image to obtain a first error value; Training the initial fully convolutional neural network model according to the first error value and a preset number of training rounds to obtain a first final fully convolutional neural network model; Inputting the preprocessed training CT image into the encoder network of the initial fully convolutional neural network model to obtain a second low-resolution feature map and a second high-resolution feature map; Inputting the low-resolution feature map into the decoder network of the initial fully convolutional neural network model for upsampling and fusing the second high-resolution feature map to obtain a training pseudo MRI image; Performing loss calculation on the training pseudo MRI image and the preprocessed training MRI image to obtain a second error value; The initial fully convolutional neural network model is trained according to the second error value and the preset training rounds to obtain a second final fully convolutional neural network model.

6. The deep learning-based MRI image and CT image conversion terminal according to claim 5, characterized in that: The performing N4 offset correction and histogram matching on the training MRI image and the training CT image to obtain a preprocessed training MRI image and a preprocessed training CT image includes: Acquire a training MRI grayscale image and a training CT grayscale image according to the training MRI image and the training CT image; Performing floating-point image conversion on the training MRI grayscale image and the training CT grayscale image to obtain converted training MRI grayscale image and training CT grayscale image; Correcting the converted training MRI grayscale image and training CT grayscale image using N4 bias correction to obtain corrected training MRI grayscale image and training CT grayscale image; Histogram matching is performed on the corrected training MRI grayscale image and the training CT grayscale image to obtain a preprocessed training MRI image and a preprocessed training CT image.

7. The deep learning-based MRI image and CT image conversion terminal according to claim 5, characterized in that: The initial fully convolutional neural network model is trained and verified based on the preprocessed training MRI image and the preprocessed training CT image to obtain a final fully convolutional neural network model, including: The images to be converted include MRI images to be converted and CT images to be converted; Inputting the image to be converted into the final fully convolutional neural network model to obtain a synthetic image corresponding to the image to be converted includes: Inputting the MRI image to be converted into the first final fully convolutional neural network model to obtain a pseudo CT image corresponding to the MRI image to be converted; The CT image to be converted is input into the second final fully convolutional neural network model to obtain a pseudo MRI image corresponding to the CT image to be converted.

8. The deep learning-based MRI image and CT image conversion terminal according to claim 5, characterized in that: The encoder network includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, a maximum pooling layer, a third activation layer and a downsampling layer arranged in sequence; The decoder network includes an upsampling layer and a third convolutional layer arranged in sequence.

Images