Brain CT-to-MRI medical image cross-modal conversion method based on deep learning

Abstract

The invention relates to a brain CT (Computed Tomography) to MRI (Magnetic Resonance Imaging) medical image cross-modal conversion method based on deep learning, and belongs to the field of medical image processing. According to the medical image cross-modal conversion method, the CT modal image can be converted into the MRI modal image so as to make up for the lack of information and help doctors to carry out clinical diagnosis. An existing conversion method has the common problems that the precision of a generated image is insufficient, the image is too smooth, and the image is not sensitive enough to details in a medical image, so that the generated image cannot meet the requirements of clinical diagnosis. A deep learning generation network framework composed of a frequency division global generation module based on wavelet transform and a local detail enhancement module based on a space and channel attention mechanism is established by adopting a global-to-local stage-by-stage refining strategy. Experimental results show that the method provided by the invention is similar to an existing advanced method in MAE and MSE indexes, but obviously exceeds the existing method in PSNR and SSIM indexes, the MAE is 0.03058, the MSE is 0.00503, the PSNR is 27.283, and the SSIM is 0.806.

Description

Technical Field

[0001] The present invention relates to a method for converting different modality images in brain medical images, and specifically to a method for cross-modality conversion of brain CT to MRI medical images based on deep learning. Background Art

[0002] In modern clinical medical diagnosis, medical images play an indispensable role in presenting the internal physiological tissue structure and pathological information of the human body. Different imaging techniques, such as computed tomography (CT) and magnetic resonance imaging (MRI), provide different types of physiological and anatomical information, each with its own advantages and disadvantages. [1] . CT has a short imaging time and low economic cost, and has advantages in the examination of bony structures. However, due to the similar density of soft tissues, CT imaging technology has poor differentiation of soft tissues. In contrast, MRI has a stronger resolution for soft tissues such as nerves, blood vessels, and muscles, which is crucial for the diagnosis and treatment planning of brain diseases. However, acquiring MRI images takes a lot of time, resulting in high time costs and may delay treatment. In addition, the high cost of MRI equipment has also limited its popularity to a certain extent.

[0003] In clinical practice, patients are not always able to undergo MRI imaging examinations due to scanning contraindications, economic, time and other reasons. For example, when a patient has a pacemaker or a magnetic prosthesis in his body, an MRI scan cannot be performed. In addition, it may be difficult to obtain MRI images in a short period of time due to rapid disease progression, limited clinical resources, and limited scanning and treatment time. In contrast, CT imaging examinations are widely used, with low imaging costs and easy acquisition. This has prompted the need to convert images from CT image modality to MRI image modality in order to make full use of easily accessible medical image data. With the development of artificial intelligence technologies represented by deep convolutional neural networks in the field of medical image processing, the use of deep learning technology to complete cross-modal conversion of medical images has received widespread attention. [2] . Most of the existing deep learning modality conversion methods are based on generative adversarial networks (GANs) or encoder-decoder networks. These networks usually use deep convolutional neural networks to extract target image features and complete image cross-modal conversion. However, existing research results have common problems such as insufficient data accuracy of generated images, overly smooth images, and insufficient sensitivity to details in medical images, resulting in the generated images failing to meet the requirements of doctors' clinical diagnosis. Therefore, it is particularly important to design a new conversion method to solve the above problems.

[0004] In order to solve the above problems, the present invention proposes a modal conversion network based on a GAN network framework of wavelet transform. Wavelet transform can decompose image signals into sub-image signals with different spatial resolutions, different frequency characteristics and directional characteristics, which allows the model to simultaneously observe the low-frequency and high-frequency parts of the image, helping to avoid ignoring the high-frequency texture details in brain medical images. At the same time, the wavelet function has localization characteristics in the time domain and frequency domain, which means that the wavelet transform can better capture the local features of brain medical images. The GAN network uses the mechanism of generation and confrontation to enable the model to continuously self-optimize during the training process, thereby further improving the training quality. In addition, this framework adopts a stage-by-stage refinement strategy from global to local, which significantly enhances the stability of the training process. Through the combination of these methods, the modal conversion network can effectively improve the accuracy and meet the needs of doctors in clinical diagnosis.

[0005] [1] Dayarathna, Sanuwani, et al. "Deep learning based synthesis of MRI, CT and PET: Review and analysis." Medical Image Analysis, pp. 103046, Dec. 2023.

[0006] [2]Li,Wen,et al."Magnetic image(MRI)synthesis from braincomputed tomography(CT)images based on deep learning methods for magneticresonance(MR)-guided radiotherapy."Quantitative imaging in medicine and surgery,vol.10,no.6,pp.1223,Apr.2020. Summary of the invention

[0007] The present invention proposes a cross-modal conversion method from brain CT to MRI medical images based on deep learning technology. The method is based on the GAN framework and mainly includes a generator and a discriminator. In the generator, a frequency-divided global generation module based on wavelet transform and a local detail enhancement module based on spatial and channel attention mechanism are used; in the discriminator, a subband discriminator module based on frequency-divided wavelet image and global image is used. At the same time, the method improves the objective function, uses multi-scale pixel loss and multi-scale structural similarity loss, and enhances the model's attention to the image structure feature information. By training the paired matching CT and MRI data sets that have been preprocessed, and using data enhancement technology to improve the generalization ability and robustness of the model, the method can efficiently and accurately convert brain CT images into MRI images, especially when MRI images cannot be obtained. This method effectively reconstructs the texture feature details of medical images, alleviates the problem that the model is not sensitive to details, not only generates high-quality brain MRI medical images, but also shortens the diagnosis time, showing great potential in the field of auxiliary clinical diagnosis.

[0008] In order to achieve the purpose of the present invention, the following technical solution will be adopted: a method for cross-modal conversion of brain CT to MRI medical images based on deep learning. It includes the following steps:

[0009] S1. Data preprocessing. The original medical image data format is processed into an image format that is convenient for computer processing. Paired matching data are selected to form a brain CT-MRI image dataset, and the collected CT data and MRI data are preprocessed to eliminate the impact of different scanner models and acquisition settings on data distribution.

[0010] S1-1. File format conversion. The CT and MRI image data collected by hospital equipment are converted from the original Dicom file format to a more conveniently processed compressed Nifti format file, i.e., nii.gz file, through MRIcron software;

[0011] S1-2. Resampling and image scaling. The voxel grids of all images were unified, and the SimpleITK software package was used to perform voxel resampling on the CT image and MRI image files so that all voxels were resampled to (1,1,1). After resampling, each slice of the horizontal plane (cross section) of the CT image and MRI image was scaled to 256×256 to facilitate subsequent deep learning network model processing;

[0012] S1-3, paired data set selection. Professional imaging physicians with many years of experience screened CT images and MRI images one by one using 3D slicer software, and selected paired CT-MRI data based on the data information of the files themselves;

[0013] S1-4, Image registration. The rigid registration method in the Elastix toolkit is used to perform image registration for paired CT data and MRI data, and the CT image is registered to the MRI image;

[0014] S1-5, bias field correction and histogram matching. The N4 bias field correction algorithm in the SimpleITK software package is used to process MRI images to reduce the problem of uneven signal intensity in MRI scans. After the bias field correction is completed, histogram matching is performed on MRI images from different scanner equipment models to make their grayscale distribution characteristics more consistent;

[0015] S1-6. Data normalization and slicing. For CT images, the HU values ​​were truncated in the range of -1000 to 2000 to remove possible noise or artifacts in the CT images, and then the data was normalized to between [-1, 1]. For MRI data, the data was normalized to between [-1, 1]. After data normalization, the CT and MRI images in Nifti format were sliced ​​from the horizontal plane (cross-section) and converted into two-dimensional grayscale images in PNG format for storage, and a two-dimensional grayscale brain CT-MRI paired matching image dataset in PNG format was prepared.

[0016] S2. Establish a deep learning model. For the preprocessed brain CT-MRI paired matching data set, establish a deep learning model framework. The proposed deep learning framework is based on the GAN network, and the framework consists of a generator G and a discriminator D. The proposed generator G is used to learn the mapping from the input CT modality data X to the MRI modality data Y, where the generator G is mainly composed of two modules. The first module is a frequency division global generation module based on wavelet transform, which is used to decompose the original CT image X into four sub-band wave images in different frequency bands. Each sub-band image is sent to a U-net structure network generator for learning, and the four sub-band images generate images corresponding to the MRI domain. Where LQ represents low quality, which is then synthesized into a low-quality MRI domain image Y through a learnable discrete inverse wavelet transform LQ The second module is a local detail enhancement module based on the spatial and channel attention mechanism, which transforms the low-quality MRI domain image Y LQ Perform detail texture refinement to generate high-quality MRI domain image Y HQ The proposed discriminator D consists of sub-band discriminator modules based on frequency-divided wavelet images and global images, and is trained on images Y from the real MRI domain and synthetic high-quality MRI images Y HQ to identify the authenticity.

[0017] S2-1, frequency division global generation module based on wavelet transform. This module consists of a two-dimensional discrete wavelet transform module, four parallel U-net generation networks, and a learnable two-dimensional discrete inverse wavelet transform module. After the original CT image X is input into this module, it first passes through the two-dimensional discrete wavelet transform module and is decomposed into four sub-band images. That is, approximate texture, vertical texture, horizontal texture and diagonal texture images (A, H, V, D), the four sub-band images They are sent to four parallel U-net generative networks to learn and generate four sub-band images corresponding to the MRI domain. This learning method based on wavelet transform sub-band images of different frequency domains helps the model better capture texture details of different scales of the image, and at the same time simplifies the conversion task between different modalities and stabilizes the training process of the GAN model. In addition to the most basic deep encoder-decoder network and skip link part of the U-net structure, the U-net generation network also adds residual connections in each layer to alleviate the gradient vanishing problem caused by the network model being too deep, making it easier for the network to learn the identity mapping, thereby effectively avoiding model degradation. Finally, the four sub-band images of the MRI domain generated by learning are The low-quality MRI domain image Y is fed into the learnable 2D discrete inverse wavelet transform module LQ , the two-dimensional discrete inverse wavelet transform module can be learned and optimized, so it can adaptively alleviate the error of the inverse wavelet transform generation result caused by the generation error in the sub-band image, thereby improving the quality of the generated image;

[0018] S2-2, local detail enhancement module based on spatial and channel attention mechanism. This module is mainly composed of convolutional layer front end, spatial and channel attention module, multi-layer residual block, and convolutional layer back end. The local detail enhancement module focuses on providing low-quality MRI domain image Y based on the global generation module. LQ Add more local details to improve the texture details and quality of the image. The feature map obtained by the original CT image X through the front end of the convolution layer is compared with the low-quality MRI domain image Y LQ The feature maps of are concatenated to obtain the feature map F, where F∈R (C×H×W) , where C represents the number of channels, H represents the image height, and W represents the image width. The feature map F is sent to the spatial and channel attention module for adaptive weighting operations to highlight the important spatial regions and channel features in the feature map, thereby improving the model's ability to perceive and utilize key information. It is then sent to a multi-layer residual block for processing and transmission of deep features, and finally a high-quality MRI domain image Y is generated through the convolutional layer backend. HQ ;

[0019] S2-3, spatial and channel attention module. Specifically, the concatenated feature map F first passes through the channel attention module, applies maximum pooling and average pooling to the spatial position of each feature map, obtains two vectors of size C×1×1, and then sends them to a shared multi-layer perceptron MLP network containing two fully connected layers, and finally performs pixel addition fusion, and passes through a sigmoid activation function to obtain the channel attention feature map W c ∈R (C×1×1) , where the subscript c represents the channel attention module. c After performing a tensor product with the feature map F, the feature map F' is obtained. Secondly, the feature map F' is passed through the spatial attention module, and the maximum pooling and average pooling are applied on each channel to obtain two vectors of size 1×H×W, which are then concatenated by channel and sent to a standard convolutional layer. After a sigmoid activation function, the spatial attention feature map W is obtained. s ∈R (1×H×W) , where the subscript s represents the spatial attention module. The spatial attention feature map W s After tensor product with feature map F', adaptive weighted feature map F' is obtained, which is then sent to the subsequent multi-layer residual block for deep transmission feature learning;

[0020] S2-4, sub-band discriminator module. In the discriminator D, exclusive sub-band discriminators are set for the wavelet domain sub-band image and the final image. Specifically, for the frequency division global generation module based on wavelet transform, four discriminators (D A ,D H ,D V ,D D ), to identify the sub-band image after wavelet transformation from the real MRI image And the four sub-band images generated by learning in the MRI domain This allows the frequency division global generation module based on wavelet transform to focus more on the learning of sub-band images. For the final image, a global discriminator D is set global , to distinguish the real MRI image Y and the generated high-quality MRI domain image Y HQ Each discriminator is a sub-band discriminator, which is used to focus on the details of different levels of the image;

[0021] S3. Model training. After the model is set up, the brain CT-MRI paired matching image dataset is input into the deep learning model framework in a certain training order, with the original CT image X as input and the MRI image Y as the target image for training, which mainly includes the following steps:

[0022] S3-1. Pre-training of the frequency division global generator module based on wavelet transform. Before starting the training of the overall model, the frequency division global generator module is pre-trained to obtain a set of better global generator module features to accelerate the learning and convergence of the overall framework. The training process uses the original CT image X to obtain the sub-band image through wavelet transform. As input, the subband image after wavelet transformation from the real MRI image For the target image, four parallel U-net network structures are used for learning representation, and finally the trained four parallel U-net network models are saved;

[0023] S3-2, train the generator. First, load the pre-trained four parallel U-net network model weights, use the frequency division global generation module based on wavelet transform to extract the global information of the original CT image X, and then enhance the texture and details through the local detail enhancement module based on the spatial and channel attention mechanism. The spatial and channel attention mechanism is used to adaptively adjust the expression ability of the feature map. After a layer-by-layer progressive learning representation from coarse to fine, the overall quality of the generated image is improved;

[0024] S3-3. Train the discriminator. Compare the real MRI image Y with the generated high-quality MRI domain image Y HQ Splice and send to the global discriminator D global Similarly, the sub-band images from the real MRI images after wavelet transformation are evaluated. Four sub-band images of the MRI domain generated by learning Separately concatenated and fed into the discriminator (D A ,D H ,D V ,D D ) is evaluated, and the labels of the real image and the generated image are combined to obtain the corresponding evaluation results to update the discriminator parameters;

[0025] S4. Model validation. Save the trained deep learning model and test and evaluate the model results on the validation set to reflect the performance of the model in the cross-modal conversion task of brain CT to MRI medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of the image conversion method.

[0027] Figure 2 Schematic diagram of the correspondence between CT and MRI.

[0028] Figure 3 Schematic diagram of the overall network structure.

[0029] Figure 4 Schematic diagram of the wavelet global generation module.

[0030] Figure 5 Schematic diagram of the local detail refinement module structure.

[0031] Figure 6 Generate a comparison chart of the resulting local curves.

[0032] Figure 7 Comparison table of results generated by different methods. DETAILED DESCRIPTION

[0033] The following describes the implementation of the present invention in detail with reference to the accompanying drawings and examples.

[0034] like Figure 1 As shown in , a method for transmembrane state conversion of brain CT to MRI medical images mainly includes the following steps:

[0035] 1. Acquisition of experimental data

[0036] The experimental data for this experiment comes from the SynthRAD2023 dataset. The dataset consists of two parts: 1) MRI-CT dataset; 2) CBCT-CT dataset. Only the first part, the MRI-CT dataset, is used here to train and verify the model, and the original dataset is divided into training set, test set, and validation set in a ratio of 6:2:2.

[0037] 2. Data preprocessing

[0038] The original medical imaging data format is processed into an image format that is convenient for computer processing. Paired matching data are selected to form a brain CT-MRI image dataset. The acquired CT data and MRI data are preprocessed to eliminate the impact of different scanner models and acquisition settings on data distribution.

[0039] 2-1. File format conversion

[0040] The CT and MRI image data collected by hospital equipment are converted from the original Dicom file format to a more conveniently processed compressed Nifti format file, i.e., nii.gz file, through MRIcron software;

[0041] 2-2. Resampling and image scaling

[0042] The voxel grids of all images were unified, and the SimpleITK software package was used to perform voxel resampling on the CT image and MRI image files so that all voxels were resampled to (1,1,1). After resampling, each slice of the horizontal plane (cross section) of the CT image and MRI image was scaled to 256×256 to facilitate subsequent deep learning network model processing;

[0043] 2-3. Paired Dataset Selection

[0044] Professional imaging physicians with many years of experience use 3D slicer software to screen CT images and MRI images one by one, and select paired CT-MRI data based on the data information of the files themselves;

[0045] 2-4. Image registration.

[0046] The rigid registration method in the Elastix toolkit is used to perform image registration for paired CT data and MRI data, and the CT image is registered to the MRI image;

[0047] 2-5. Bias field correction and histogram matching

[0048] The MRI images were processed using the N4 bias field correction algorithm in the SimpleITK software package to reduce the signal intensity unevenness problem in MRI scans. After the bias field correction was completed, histogram matching was performed on MRI images from different scanner equipment models to make their grayscale distribution characteristics more consistent;

[0049] 2-6. Data normalization and slicing

[0050] For CT images, the HU values ​​are truncated in the range of -1000 to 2000 to remove possible noise or artifacts in the CT images, and then the data is normalized to between [-1, 1]; for MRI data, the data is normalized to between [-1, 1]. After data normalization, the Nifti format CT and MRI images are sliced ​​from the horizontal plane (cross-section), converted into two-dimensional grayscale images in PNG format and saved, and a two-dimensional grayscale brain CT-MRI paired matching image dataset in PNG format is prepared. The processed paired PNG format CT and MRI images are shown in Figure 2. Figure 2 shown.

[0051] 3. Build a deep learning model

[0052] For the preprocessed brain CT-MRI paired matching dataset, a deep learning model framework is established, such as Figure 3 As shown in the figure, the proposed deep learning framework is based on the GAN network, which consists of a generator G and a discriminator D. The proposed generator G is used to learn the mapping from the input CT modality data X to the MRI modality data Y. The proposed discriminator D consists of a sub-band discriminator module based on the frequency-divided wavelet image and the global image, and is used to classify the image Y from the real MRI domain and the synthesized high-quality MRI image Y. HQ to identify the authenticity.

[0053] 3-1. Constructing a frequency division global generation module based on wavelet transform

[0054] The module is specifically composed of a two-dimensional discrete wavelet transform module, four parallel U-net structure generation networks, and a learnable two-dimensional discrete inverse wavelet transform module. Figure 4 As shown. The Haar wavelet is selected as the basis function of the two-dimensional discrete wavelet forward and inverse transform. The four parallel U-net structures generate the same network, consisting of three downsampling layers and three upsampling layers, and the corresponding layers are connected together through skip-connection. In each layer, the image undergoes two convolutional layers, normalization layers, and activation layers respectively. After the original CT image X is input into this module, it first passes through the two-dimensional discrete wavelet transform module and is decomposed into four sub-band images Then the four sub-band images The images are fed into four parallel U-net generative networks to learn and generate four sub-band images corresponding to the MRI domain. Finally, the four sub-band images of the MRI domain generated by learning The low-quality MRI domain image Y is fed into the learnable 2D discrete inverse wavelet transform module HQ ;

[0055] 3-2. Constructing a local detail enhancement module based on spatial and channel attention mechanism

[0056] The module mainly consists of a convolutional layer front end, spatial and channel attention modules, multi-layer residual blocks, and a convolutional layer back end, such as Figure 5 As shown in the figure, the number of input channels at the front end of each convolution layer is 1, and the number of output channels is 64. The feature map obtained by passing the original CT image X through the front end of the convolution layer is compared with the low-quality MRI domain image Y HQ The feature map F is concatenated with the feature maps of the previous two layers to obtain the feature map F. The feature map F is sent to the spatial and channel attention module for adaptive weighting operation, and then sent to the multi-layer residual block for processing and transmission of deep features. The number of multi-layer residual blocks here is set to 10, and each residual block consists of an activation layer, a convolution layer, and a normalization layer. Finally, a high-quality MRI domain image Y is generated through the convolution layer backend HQ ;

[0057] 3-3. Constructing spatial and channel attention modules

[0058] Specifically, the concatenated feature map F first passes through the channel attention module, applies maximum pooling and average pooling to the spatial position of each feature map, and obtains two vectors of size C×1×1, which are then sent to a shared multi-layer perceptron MLP network containing two fully connected layers. Finally, pixel addition and fusion are performed, and a sigmoid activation function is used to obtain the channel attention feature map W. c ∈R (C×1×1). The channel attention feature map W c After performing a tensor product with the feature map F, the feature map F' is obtained. Secondly, the feature map F' is passed through the spatial attention module, and the maximum pooling and average pooling are applied on each channel to obtain two vectors of size 1×H×W, which are then concatenated by channel and sent to a standard convolutional layer. After a sigmoid activation function, the spatial attention feature map W is obtained. s ∈R (1×H×W) . The spatial attention feature map W s After tensor product with feature map F', adaptive weighted feature map F' is obtained, which is then sent to the subsequent multi-layer residual block for deep transmission feature learning;

[0059] 3-4. Constructing the subband discriminator module

[0060] In the discriminator D, dedicated sub-band discriminators are set for the wavelet domain sub-band image and the final image. Specifically, four discriminators (D A ,D H ,D V ,D D ). For the final image, a global discriminator D is set global . Each discriminator is a standard Markov discriminator;

[0061] 4. Perform model training

[0062] After the model is set up, the brain CT-MRI paired matching image dataset is input into the deep learning model framework in a certain training order. The original CT image X is used as input and the MRI image Y is used as the target image for training. The training mainly includes the following steps:

[0063] 4-1. Pre-training of the frequency division global generation module based on wavelet transform

[0064] Before starting the training of the overall model, the frequency-divided global generator module is pre-trained to obtain a set of better global generation module features to accelerate the learning and convergence of the overall framework;

[0065] 4-2. Training the Generator

[0066] First, the weights of four pre-trained parallel U-net network models are loaded, and the global information of the original CT image X is extracted using the frequency division global generation module based on wavelet transform. Then, the texture and details are enhanced by the local detail enhancement module based on the spatial and channel attention mechanism. The spatial and channel attention mechanism is used to adaptively adjust the expression ability of the feature map. Through the progressive learning representation from coarse to fine, the overall quality of the generated image is improved.

[0067] 4-3. Training the discriminator

[0068] The real MRI image Y is compared with the generated high-quality MRI domain image Y HQ Splice and send to the global discriminator D global Similarly, the sub-band images from the real MRI images after wavelet transformation are evaluated. Four sub-band images of the MRI domain generated by learning Separately concatenated and fed into the discriminator (D A ,D H ,D V ,D D ) is evaluated, and the labels of the real image and the generated image are combined to obtain the corresponding evaluation results to update the discriminator parameters;

[0069] 5. Model verification.

[0070] Save the trained deep learning model and test and evaluate the model results on the validation set to reflect the performance of the model in the cross-modal conversion task of brain CT to MRI medical images.

[0071] Example

[0072] This experiment completed training and verification based on the SynthRAD2023 dataset. The dataset available for download contains CT-MRI paired matching data of 180 patients. Each patient has an average of 200 horizontal plane (cross-sectional) images available for training.

[0073] Before the experiment, the voxel spacing of CT data and MRI data was resampled to (1,1,1), and then the horizontal plane (cross section) of the data was uniformly scaled to 256×256. The resampled and scaled data were rigidly registered. After the registration, bias field correction and histogram matching were required for the MRI images. Finally, the CT images and MRI images were normalized and scaled to the range of [-1,1].

[0074] During the experiment, the model was built in the Pytorch environment and the experiment was conducted on the Nvidia RTX 4060 graphics card. In the generator, the frequency-divided global generation module based on wavelet transform uses Haar wavelet transform as the basis function of the two-dimensional discrete wavelet module. The U-net generation network has a three-layer encoder and a three-layer decoder network structure. The local detail enhancement module based on the spatial and channel attention mechanism has 10 residual blocks. In the discriminator, each sub-band discriminator is a standard Markov discriminator that focuses on the details of different levels of the image. Adam is used to optimize the model.

[0075] For the generator, the model is optimized according to the following objective function:

[0076] Loss G =λ1Loss A +λ2Loss H +λ3Loss V +λ4Loss D +λ5Loss global +λ L1 Loss patch-L1 +λ ssim Loss ms-ssim

[0077] Among them, λ1, λ2, λ3, λ4, λ5, λ L1 and λ ssim Represents the hyperparameters that control the corresponding loss. A 、Loss H 、Loss V 、Loss D represents the frequency-division global generation loss based on wavelet transform. Specifically, it is composed of the following formula:

[0078]

[0079] in Represents a subband image from a real MRI image after wavelet transformation Four sub-band images of the MRI domain generated by learning The L1 norm of . Represents four discriminators (D A ,D H ,D V ,D D ) is the adversarial loss during generative training, which is defined as follows:

[0080]

[0081] Among them, Loss global Represents the global discriminator D global The adversarial loss during training is defined as follows:

[0082] Loss global =-Ε y∈Y [log(1-D global (y,y HQ ))]-Ε x∈X [log(D global (y,y HQ ))]

[0083] Loss in the total loss function patch-L1represents the multi-scale pixel loss, which is achieved by combining the real MRI image Y and the generated high-quality MRI domain image Y HQ Use a specific tensor transformation F to extract patches with pixel features, and then calculate the L1 loss on the corresponding patches, which is defined as follows:

[0084]

[0085] Loss in the total loss function ms-ssim Represents the multi-scale structural similarity loss. By setting M scales, the real MRI image Y and the generated high-quality MRI domain image Y are taken at each scale. HQ The structural similarity index SSIM is calculated and then the average is calculated. Its definition is as follows:

[0086]

[0087] Where p represents a pixel position in the two images being calculated. The definition of msssim(p) is as follows:

[0088]

[0089] In the formula, l M (p) represents the brightness contrast factor at M scales, which is defined as follows:

[0090]

[0091] cs in the formula j (p) represents the contrast factor at the jth scale, which is defined as follows:

[0092]

[0093] In the above two formulas, C1=(k1L) 2 , C2=(k2L) 2 , are two stabilizing factors, k1 and k2 are usually set to 0.01 and 0.03, and L represents the dynamic range of the pixels of the two images.

[0094] For the discriminator, the model is optimized according to the following objective function:

[0095]

[0096] in Represents four discriminators (D A ,D H ,D V ,D D ) is the adversarial loss during the discriminative training process, which is defined as follows:

[0097]

[0098] in Represents the global discriminator D global The adversarial loss during the discriminative training process is defined as follows:

[0099]

[0100] The effectiveness of the proposed method is verified through extensive comparative experiments. In the experiment, the proposed method is compared with existing modality conversion techniques, including U-net (L1+L2), pix2pix, pix2pixhd, Transunet and ResVit. All models are trained and validated on the same dataset to demonstrate their effectiveness. The quality of the images generated by the model is evaluated by evaluation indicators such as mean absolute error (MAE), mean square error (MSE), peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). The final comparison results are shown in Figure 2. Figure 7 The results show that the proposed method is similar to the existing advanced methods in terms of MAE and MSE indicators, but significantly surpasses the existing methods in terms of PSNR and SSIM indicators (see Figure 7 ). In addition, in order to compare the local generation effect, we also plotted the local pixel value comparison diagram of the real MRI image and the generated MRI image, where the white line marks the sampling position, such as Figure 6 As shown in the figure, we can see that the generated images perform well in fitting the real images.

Claims

1. A method for cross-modal conversion of brain CT to MRI medical images based on deep learning, the method framework includes the following modules: the method framework is based on GAN, and a frequency-divided global generation module based on wavelet transform and a local detail enhancement module based on spatial and channel attention mechanism are used in the generator to realize a step-by-step refinement strategy from global to local; a sub-band discriminator module based on frequency-divided wavelet image and global image is used in the discriminator.

2. The frequency division global generation module based on wavelet transform according to claim 1 is characterized in that: The module is specifically composed of a two-dimensional discrete wavelet transform module, four parallel U-net structure generation networks, and a learnable two-dimensional discrete inverse wavelet transform module. The two-dimensional discrete wavelet transform module decomposes the CT domain image data after data preprocessing into four sub-band images, namely, approximate texture, vertical texture, horizontal texture, and diagonal texture images; then the four sub-band images are respectively sent to four parallel U-net structure generation networks for learning to generate four sub-band images in the MRI domain; the generated images are sent to the learnable two-dimensional discrete inverse wavelet transform module to be synthesized into a global image in the MRI domain.

3. The local detail enhancement module based on the spatial and channel attention mechanism according to claim 1 is characterized by: This module is mainly composed of a convolutional layer front end, a spatial and channel attention module, a multi-layer residual block, and a convolutional layer back end. The convolutional layer front end is used to extract feature maps from the image generated by the previous module and the original CT image, and the feature maps of the two are adaptively weighted fused through the spatial and channel attention module, and then processed through a multi-layer residual block to finally generate a locally refined MRI domain image.

4. The sub-band discriminator module based on frequency-divided wavelet image and global image according to claim 1 is characterized in that: This module sets up exclusive multi-scale discriminators for the sub-band images in the wavelet domain and the final image, and each sub-band discriminator is a Markov discriminator. For the frequency division global generation module based on wavelet transform, four discriminators are set up to distinguish the sub-band images from the wavelet transformed sub-band images of the real MRI image and the four sub-band images in the MRI domain generated by learning; For the final image, a global discriminator is set to distinguish between the real MRI image and the generated high-quality MRI domain image.

Images