A medical image conversion method of abdominal MRI synthesized CT

Abstract

This invention proposes a medical image conversion method for abdominal MRI to synthesize CT, belonging to the technical field of medical image data processing. Multiple sets of MRI and CT samples of the human abdomen are collected and rigidly registered as a first-stage dataset. Then, image data enhancement is performed on the MRI and CT images, extracting skeletal feature maps from the CT and fusing them into the paired MRI, resulting in a second-stage dataset. A medical image conversion model including the first-stage and second-stage models is constructed, and the first-stage and second-stage models are trained using the first-stage and second-stage datasets respectively. Finally, the MRI image data to be converted is input into the trained medical image conversion model, outputting the corresponding synthesized CT. By enhancing missing skeletal information, the quality of the abdominal synthesized image is improved, generating a better skeletal framework, making the synthesized CT more structurally compatible with the MRI, and providing clearer pathological information for depicting tumor-targeting regions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image data processing, and more particularly to a medical image conversion method for abdominal MRI synthesized CT. BACKGROUND

[0002] Medical imaging technologies such as Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) have become an indispensable part of today's medical diagnosis and treatment planning, among which MRI is widely used for the identification, staging and treatment planning of soft tissue and bone tumors due to its excellent soft tissue contrast, and CT is mainly used for imaging high electron density tissues such as bones or tumor exploration. Different medical imaging modalities provide complementary information, but also have their own limitations, such as MRI has a long imaging time and high cost, and patients with claustrophobia and cardiac pacemakers may not be able to take MRI, while CT scanning produces radiation dose and causes harm to the human body, so its use frequency is limited. In clinical practice, although a more comprehensive disease assessment can be made by combining multi-modal medical images, due to various factors such as time, cost, radiation dose, etc., it is not practical to collect data of the same patient using different imaging technologies. In this context, using modern technology to realize image conversion from one modality to another shows great value and potential.

[0003] The prior art discloses a method and system for synthesizing CT images from MRI images based on deep learning, based on the fitting ability of deep learning, a deep learning neural network model including different sub-models is constructed to learn the distribution of bones and tissues respectively, and then integrated to make the distribution information clearer; adaptive segmentation is performed using the K-means clustering method, and in the original MR-CT data set, the required bone mask data can be easily obtained in batches; at the same time, different sub-models can be selected for synthesis according to the characteristics of different parts, reducing the complexity of the network and optimizing the synthesis of the border between bones and tissues. However, the synthesis method can realize image generation of unpaired data sets and optimize the synthesis of the border between bones and tissues, but cannot well capture the content features with large differences in corresponding modal information between MRI and CT. SUMMARY

[0004] To solve the problem that the current MR-CT medical image conversion process cannot well capture the content features with large differences in corresponding modal information, the present application provides a medical image conversion method for abdominal MRI synthesized CT, constructs a medical image conversion model including a two-stage deep learning architecture and a training strategy, enhances the missing bone information to improve the quality of the abdominal synthesized image, and generates a better bone framework to solve the problem of detail loss in content and style caused by the information asymmetry between the MRI and CT domains.

[0005] To achieve the above technical effects, the technical scheme of the present application is as follows:

[0006] A medical image conversion method for abdominal MRI synthesized CT, comprising:

[0007] S1. Collecting MRI and CT image data samples of multiple human abdomens;

[0008] S2. Rigidly registering the MRI image data and the CT image data, and taking the registered paired data as a one-stage data set;

[0009] S3. Respective image data enhancement is performed on the MRI image data and the CT image data obtained in step S2, including N4 correction of the MRI image data and baffle removal processing of the CT image data;

[0010] S4. Extracting the bone feature map of the CT image data, and fusing the extracted bone feature map into the paired MRI image data to obtain a two-stage data set;

[0011] S5. Based on the CycleGAN network, a medical image conversion model is constructed, and the medical image conversion model includes a one-stage model and a two-stage model;

[0012] The one-stage model is trained using the one-stage data set, the two-stage model is trained using the two-stage data set, and the trained one-stage model and two-stage model are fused to obtain a trained medical image conversion model;

[0013] S6. Inputting the MRI image data to be converted into the trained medical image conversion model, and outputting the corresponding synthesized CT image data to complete the cross-modal conversion of the medical image data.

[0014] The present technical scheme adopts an innovative two-stage deep learning architecture and training strategy, the two-stage model allows independent improvement and optimization of each stage model, effectively supports the enhancement of the missing bone information to improve the quality of the abdominal synthesized image, and the fused medical image conversion model can more deeply learn and understand the complex mapping relationship between the MRI image and the CT image, so that the model generates an image with high appearance authenticity, and more effectively ensures the reliability of the medical image.

[0015] Preferably, in step S1, the image data samples of the abdominal MRI and CT are collected, and the process includes: screening the DICOM image data of the MRI and CT of the abdominal region, saving the MRI image data as a T1 sequence, and saving the CT image data as a venous phase sequence.

[0016] Preferably, the process of step S2 includes:

[0017] S21. Importing the image data samples of the MRI and CT of one case to be paired;

[0018] S22. Performing medical image three-dimensional reconstruction on the MRI image data and the CT image data respectively, keeping the three-dimensional coordinates of the CT image data unchanged, and performing coordinate translation on the three-dimensional reconstructed object of the MRI image data so that it completely overlaps the three-dimensional reconstructed abdominal structure of the CT image data;

[0019] S23. Performing slice registration on the MRI image data and the CT image data, outputting and saving the registered MRI image data in DICOM format, and obtaining MRI image data with the same number of slices and slice levels as the CT image data;

[0020] S24. Determining whether the rigid registration of all case data is completed, if yes, saving the paired data of all MRI image data and CT image data obtained after registration as a one-stage data set, otherwise, returning to step S21.

[0021] Here, the MRI and CT image data of each patient are rigidly registered to ensure that the number of slices of the cross-modality MRI and CT image data is consistent, and the organs of each slice are aligned at the same level.

[0022] Preferably, the process of step S4 includes:

[0023] S41. Setting a pixel value threshold, extracting all pixels with a pixel value greater than the threshold in the CT image, and saving as a feature map;

[0024] S42. Performing erosion and expansion and filtering processing on the threshold extracted feature map to remove noise points, and obtaining a bone feature map of the CT image data;

[0025] S43. Superimposing the bone feature map extracted from the CT image data onto the MRI image data to obtain an MRI data set with enhanced bones, which is used as a two-stage data set of the model.

[0026] Preferably, the first-stage dataset and the second-stage dataset are divided into a training set, a validation set, and a test set, respectively. The first-stage model and the second-stage model are trained using the training set, evaluated using the validation set, and tested using the test set to obtain the trained first-stage model and the second-stage model.

[0027] Preferably, the medical image conversion model includes a one-stage model and a two-stage model, wherein:

[0028] The first-stage model is used to extract the skeletal framework of the target CT scan, including the generator G. AB1 G BA1 Discriminator D A1 D B1 and cross-modal content structure G feature The G AB1 The input MRI image data is converted into CT image data, the G BA1 Convert the input CT to MRI; the D A1 The D is used to determine the authenticity of the input MRI image data. B1 Used to determine the authenticity of input CT image data; the G feature Set in G AB1 With G BA1 Between, used for G AB1 and G BA1 Constraining the intermediate feature layer to achieve cross-modal feature alignment;

[0029] The two-stage model is used to enhance the skeletal framework style, combining the skeletal framework content extracted from the one-stage model to generate higher-quality CT-style images, including the generator G. AB2 G BA2 Discriminator D A2 D B2

[0030] The G AB2 The input MRI image data of enhanced bone is converted into CT image data, the G BA2 The input CT scan is converted into MRI image data; the D A2 The D is used to determine the authenticity of the input MRI image data. B2 Used to determine the authenticity of input CT image data.

[0031] In this paper, the weakness of MRI and CT modalities imaging on the skeleton frame is cut in, and the synthesis of MRI into CT is divided into two steps. The first step is to improve the model's ability to learn the cross-modal content features such as the skeleton frame to synthesize the MRI-aligned bone structure. The second step is to use the advantage of CycleGAN in style conversion and combine the framework content extracted in the first step to generate higher-quality CT style images.

[0032] Preferably, in step S5, the medical image conversion model is trained, the one-stage model is trained using the training set of the one-stage data set, and the two-stage model is trained using the training set of the two-stage data set, and the process includes:

[0033] S51. Freeze the weights of the discriminator first, and train the generator: input the real MRI image data and CT image data in the one-stage data set and the two-stage data set into the corresponding generator respectively, generate false synthetic CT image data and MRI image data, and high-dimensional content features extracted from the real MRI image data and CT image data;

[0034] The false synthetic CT image data and MRI image data are input into the corresponding generator for reconstruction loss, and the reconstruction image data of the synthetic CT image data and MRI image data, and the high-dimensional content features extracted from the synthetic CT image data and MRI image data are generated respectively;

[0035] The reconstruction loss between the real MRI image data and CT image data and the corresponding reconstruction image data is calculated, and the reconstruction loss is used for back propagation training of the generator parameters;

[0036] The content loss between the high-dimensional content features extracted from the real MRI image data and CT image data and the high-dimensional content features extracted from the synthetic CT image data and MRI image data is calculated, and the content loss is used to optimize the parameters of the generator;

[0037] S52. Freeze the weights of the generator, and train the discriminator: label the real MRI image data and CT image data as True and input them into the corresponding discriminator, label the synthetic CT image data and MRI image data generated by the generator as False and input them into the corresponding discriminator, and use the calculated loss to optimize the parameters of the discriminator;

[0038] S53. Repeat steps S51 and S52, alternately train the generator and the discriminator, update the corresponding weight parameters each time, and obtain the trained one-stage model and two-stage model;

[0039] S54. Fuse the one-stage model and the two-stage model to obtain the trained medical image conversion model.

[0040] Preferably, in the iteration process of step S5, the parameters of each stage model are adjusted by constructing an overall minimization hybrid loss function of each stage model to back-propagate the parameters; the overall minimization hybrid loss function comprises: an adversarial loss, a cycle consistency loss, a content constraint loss, a skeletal structure constraint loss and a VGG network loss;

[0041] Let the MRI image data set in the paired data be denoted as domain A, and the CT image data set be denoted as domain B; let any image data in domain A be denoted as realA, i.e. real MRI image data, and any image data in domain B be denoted as realB, i.e. real CT image data; let the synthetic CT image data generated by the generator from realA be denoted as fakeB, and the synthetic MRI image data generated by the generator from realB be denoted as fakeA; let the CT reconstructed image data generated by the generator from fakeA be denoted as recB, and the MRI reconstructed image data generated by the generator from fakeB be denoted as recA; BA1 and G BA2 and G

[0042] Let the high-dimensional content features of realA in domain A output by the intermediate layer of the generator be denoted as Let the high-dimensional content features of realB in domain B be denoted as Let the high-dimensional content features of fakeA in domain A be denoted as Let the high-dimensional content features of fakeB in domain B be denoted as A content constraint loss is constructed, and the content constraint loss The formula is as follows:

[0043]

[0044] wherein, represents and constitute a new content constraint in domain A, and the represents and constitute a new content constraint in domain B;

[0045] The skeletal structure constraint comprises an L1 norm distance constraint between realA and fakeA, and an L2 norm distance constraint between realB and fakeB, and the formula is as follows:

[0046]

[0047] wherein, ∥1 represents using L1 norm, and ∥2 represents using L2 norm;

[0048] The VGG network constraint performs semantic constraint on the extracted feature map, and the formula is as follows:

[0049]

[0050] Wherein, φ represents the output of a certain layer or multiple layers of the VGG network;

[0051] The overall minimum hybrid loss function of each stage model The formula is as follows:

[0052]

[0053] Wherein, n is in {1, 2}; And Consistent with the adversarial loss in CycleGAN; Consistent with the cycle consistency loss in CycleGAN; L1 norm distance constraint is L1 norm distance constraint, L2 norm distance constraint is L2 norm distance constraint, respectively adjusted by λ L1 And λ L2 Two hyperparameters; VGG network distance constraint loss is VGG network distance constraint loss, λ VGG Is a hyperparameter for regulating the VGG network loss; Content loss is content loss, λ feature Is a hyperparameter for adjusting the content loss.

[0054] Here, by setting the content constraint and L1 loss and L2 loss constraint based on the VGG network, the accuracy of the universal feature representation between different modalities is improved, so that the model can generate a better skeleton framework in one stage, so as to effectively solve the problem of detail loss in content and style caused by the information asymmetry between MRI and CT domains.

[0055] Preferably, in step S5, the one-stage model and the two-stage model are evaluated by using the validation set, and the evaluation index of the one-stage model includes the structural similarity index, and the evaluation index of the two-stage model includes the structural similarity index, the peak signal-to-noise ratio, the mean square error and the perception loss.

[0056] Preferably, the specific process of step S6 includes: inputting the MRI image data to be converted into the trained medical image conversion model, extracting the skeleton framework of the false CT generated by the one-stage model, inputting the skeleton framework and the input MRI into the generator of the two-stage model together, and performing fusion of the MRI and the skeleton framework and CT style conversion through the generator of the two-stage model, to output corresponding synthetic CT image data

[0057] Compared with the prior art, the technical scheme of the present application has the following advantages:

[0058] The application provides a medical image conversion method for abdominal MRI synthesized CT. MRI and CT samples of multiple human abdomens are collected and rigidly registered to obtain paired data as a first-stage data set; image data of the MRI and the CT are enhanced, a bone feature map of the CT is extracted and fused into the paired MRI to obtain a second-stage data set; a medical image conversion model including a first-stage model and a second-stage model is constructed based on a CycleGAN network, the first-stage model and the second-stage model are trained by using the first-stage data set and the second-stage data set respectively, and the trained first-stage model and the second-stage model are fused; finally, MRI image data to be converted is input into the trained medical image conversion model, and corresponding synthesized CT is output, so that medical image cross-modal conversion is completed, which not only helps to reduce medical costs and reduce the radiation dose received by patients, but also enhances the missing bone information to improve the quality of the abdominal synthesized image, generates a better bone framework, so that the synthesized CT can be more matched with the MRI in structural content, and provides clearer pathological information for the delineation of a tumor target region. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 FIG. 1 shows a flowchart of the medical image conversion method for abdominal MRI synthesized CT proposed in Embodiment 1 of the application;

[0060] Figure 2 FIG. 2 shows a schematic diagram of the 3D-Slicer rigid registration operation in Embodiment 1 of the application;

[0061] Figure 3 FIG. 3 shows a schematic diagram of the CT baffle removal step process in Embodiment 1 of the application;

[0062] Figure 4 FIG. 4 shows a comparison diagram of the MR image data superimposed with the CT bone feature map in Embodiment 1 of the application;

[0063] Figure 5 FIG. 5 shows a schematic diagram of the medical image conversion model proposed in Embodiment 2 of the application. DETAILED DESCRIPTION

[0064] The drawings are only used for illustrative description and cannot be understood as a limitation of the patent;

[0065] In order to better illustrate the embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0066] For those skilled in the art, it can be understood that some well-known content in the drawings may be omitted.

[0067] The technical solutions of the application will be further described below in combination with the drawings and embodiments.

[0068] The positional relationship described in the drawings is only for illustrative purposes and should not be construed as limiting the patent;

[0069] Embodiment 1

[0070] As Figure 1 shown, the embodiment proposes a medical image conversion method for abdominal MRI synthesized CT, comprising:

[0071] S1. Collecting MRI and CT image data samples of the human abdomen of multiple patients taken within a short time interval, screening the MRI and CT DICOM image data of the abdominal region, saving the MRI as a T1 sequence, and saving the CT as a venous phase sequence.

[0072] S2. Rigidly registering the MRI image data and the CT image data, taking the paired data obtained by registration as a one-stage data set, the process comprising:

[0073] S21. Importing the MRI and CT image data samples of one case to be paired in 3D-Slicer;

[0074] S22. Using the Volume Rendering function of 3D-Slicer to perform medical image three-dimensional reconstruction on the MRI image data and the CT image data respectively, keeping the three-dimensional coordinates of the CT image data unchanged, using the Translation function to perform coordinate translation on the three-dimensional reconstruction object of the MRI image data by up and down, left and right, and front and back, so that it completely overlaps with the three-dimensional reconstruction of the abdominal structure of the CT image data, as Figure 2 shown;

[0075] S23. Selecting the plug-in General Registration (ANTs) in the Modules bar of 3D-Slicer to perform slice registration on the MRI and CT image data, then checking the registered MRI and CT image data, outputting and saving the registered MRI image data in DICOM format, obtaining MRI image data with the same number of slices and slice levels as the CT image data;

[0076] S24. Determine whether rigid registration of all case data is completed, if yes, save all paired MRI and CT data obtained after registration as a one-stage data set, otherwise, return to step S21.

[0077] S3. Since the medical image data set obtained clinically contains noise, it needs to be preprocessed to improve the quality of the data set; image data enhancement is performed on the MRI image data and the CT image data obtained in step S2, including N4 correction of the MRI image data and baffle removal processing of the CT image data;

[0078] The MRI image data is image data imaged by a magnetic field, and is prone to uneven brightness caused by uneven magnetic field, so it is necessary to correct the MRI image data by N4. The process includes:

[0079] DICOM data sets are saved as JPG format files using Python or RadiAnt DICOM Viewer (64-bit) software;

[0080] The MRI is N4-corrected using the sitk.N4BiasFieldCorrectionImageFilter() function of the SimpleITK package in Python;

[0081] The MRI image data of all case data is sequentially N4-corrected to obtain more uniform MRI image data.

[0082] The CT image obtained clinically has a baffle area, which may introduce shadows or artifacts, affecting the quality and accuracy of the image. Therefore, in this embodiment, the OpenCV library is used in Python to perform baffle removal processing on the CT image data, reducing the need for subsequent image processing and improving the quality of the image, as shown in Figure 3 The process includes:

[0083] The CT image data sample is converted into a gray-scale image by converting the brightness (or color information) of each pixel of the color image into a single brightness value, reducing the complexity of the data to simplify the subsequent processing steps,

[0084] The binaryzation technique is used to convert the gray-scale image into a binary image, i.e. the pixel points on the image have only two possible values of 0 and 255, which is achieved by setting a threshold value, all pixel points higher than the threshold value are set to white, and those lower than the threshold value are set to black;

[0085] The dilation and erosion steps in morphological operations are performed to remove small noise in the image and emphasize the main structure. The dilation operation helps to fill small holes in the foreground object, while the erosion operation is used to remove small foreground objects. The two operations are usually used together to improve the structural features of the image;

[0086] The contour extraction technique is used to identify the foreground object in the image. Compared with identifying the contour of the baffle area, this embodiment focuses on extracting the largest contour, i.e. assuming that the largest continuous area is the region of interest (ROI). Retaining the contour of the medical image part of interest makes the method more simplified, only the largest closed outer contour needs to be found, which not only helps to identify and segment the object in the image, but also provides necessary information for subsequent image processing steps;

[0087] Create a mask of the same size as the original image, fill it with white only in the largest contour area, and black in the rest of the area; apply this mask to the original image so that only the area within the largest contour is retained, achieving the purpose of removing the baffle.

[0088] S4. Extract the bone feature map of the CT image data, and fuse the extracted bone feature map into the paired MRI image data to obtain a two-stage data set, the process comprising:

[0089] S41. Set a pixel value threshold, extract all pixel points with a pixel value greater than the threshold in the CT image data, and save them as a feature map; in this embodiment, the pixel value threshold is set to 220;

[0090] S42. Perform erosion and expansion and filtering processing on the threshold-extracted feature map to remove noise points, and obtain the bone feature map of the CT image data;

[0091] S43. Superimpose the bone feature map extracted from the CT image data onto the MRI image data to obtain an MRI data set with enhanced bones, which is used as a two-stage data set of the model, and the visualization effect is as shown in Figure 4 .

[0092] S5. Based on the CycleGAN network, a medical image conversion model is constructed, which includes a one-stage model and a two-stage model;

[0093] The one-stage model is trained using the one-stage data set, and the two-stage model is trained using the two-stage data set. The trained one-stage model and two-stage model are fused to obtain a trained medical image conversion model. Specifically, the one-stage data set and the two-stage data set are divided into training set, validation set and test set respectively; the one-stage model and the two-stage model are trained using the training set, and then the one-stage model and the two-stage model are evaluated using the validation set, and the effectiveness of the one-stage model and the two-stage model is tested using the test set, to obtain the trained one-stage model and the two-stage model.

[0094] S6. Input the MRI image data to be converted into the trained medical image conversion model, extract the false CT bone framework generated by the one-stage model, and input the bone framework and the input MRI image data into the generator of the two-stage model, and perform fusion of the MRI image data and the bone framework and CT style conversion through the generator of the two-stage model, output the corresponding synthetic CT image data, and complete the medical image cross-modal conversion.

[0095] Embodiment 2

[0096] This embodiment specifically describes the medical image conversion model proposed in embodiment 1.

[0097] like Figure 5 As shown, the medical image conversion model includes a one-stage model and a two-stage model, wherein:

[0098] The first-stage model is used to extract the skeletal framework of the target CT image data, including the generator G. AB1 G BA1 Discriminator D A1 D B1 and cross-modal content structure G feature The G AB1 The input MRI image data is converted into CT image data, the G BA1 The input CT image data is converted into MRI image data; the D A1 This is used to determine the authenticity of the input MRI image data. The output 64*64*1 is a probability vector, which is related to the size of the input data. Similarly, D... B1 The D B1 Used to determine the authenticity of input CT image data; the G feature Set in G AB1 With G BA1 Between, used for G AB1 and G BA1 Constraining the intermediate feature layer to achieve cross-modal feature alignment;

[0099] In the first stage of training, for real image data input to the generator, the generator outputs fake synthetic image data. This fake data is then input to the discriminator for judgment and simultaneously input to another generator for cycle consistency constraints. By minimizing the content loss constraint, adversarial loss, cycle consistency loss, VGG loss, and L1-L2 loss proposed in this invention, the parameters are optimized through backpropagation and multiple iterations. Finally, the optimal round in which the structural similarity index (SSIM) is obtained is used as the output pass-through point by detecting the performance of the validation set.

[0100] The two-stage model is used to enhance the skeletal framework style, combining the skeletal framework content extracted from the one-stage model to generate higher-quality CT-style image data, including the generator G. AB2 G BA2 Discriminator D A2 D B2

[0101] The G AB2 The input MRI image data of enhanced bone is converted into CT image data, the G BA2 The input CT image data is converted into MRI image data; the D A2 The D is used to determine the authenticity of the input MRI image data. B2For judging the authenticity of input CT image data

[0102] In the second phase of training, the bone framework structure of CT rigidly aligned with the MRI image data is extracted and added to the channel of the MRI image data to obtain the enhanced bone MRI image data, and the enhanced bone MRI image data and the CT image data are used for training, so that the model learns to fuse the MRI image data and the CT bone framework; the hyperparameters of the adversarial loss and the cycle consistency loss are improved in the training process, so that the model pays more attention to the style feature conversion of the cross-modal image, and finally generates a CT style image that retains the high-definition content of the MRI.

[0103] Specifically, in the present embodiment, each generator contains three convolutional blocks and three deconvolutional blocks, with nine jump-connection residual blocks connecting between the encoder and the decoder. The three convolutional blocks-deconvolutional blocks contain symmetric convolutional neural network (CNN) structures, combining features with channel numbers of 64, 128 and 256 in a cascaded form. In the encoding part, the input image with a size of 512x512 first undergoes a reflection padding layer (ReflectionPad2d) to become 518x518, and then enters the first convolutional layer (Conv2d) for feature extraction. This layer contains 64 convolutional kernels, each with a size of 7x7, a step of 1, and a padding of 0, outputting a 64-channel 512x512 image. Next, the image is input into a normalization layer (BatchNorm2d) to normalize the features after convolution, in order to improve the training process, and then the image is input into a ReLU activation layer to increase the nonlinearity of the model. Subsequently, the image will enter two convolutional blocks with a convolution kernel size of 3x3 for further feature extraction. After three encoding blocks, the input image is reduced from 512x512 to 128x128, and the depth of the feature map is deepened from 1 to 256. These feature maps contain key information such as edges, textures and shapes extracted by the convolutional layer. After dimension reduction by the encoder, the image will be transmitted into 9 stacked residual blocks for further training. Each residual block internally undergoes reflection padding, 3x3 convolution, normalization layer, activation layer, dropout layer, etc. Corresponding to the encoder, the decoder part contains two 3x3 deconvolutional layers and one 7x7 deconvolutional layer to restore the image to a size of 512x512, and finally output through a Tanh activation function.

[0104] The discriminators all use a 4x4 convolution kernel, which allows the model to effectively capture details and distinguish real images from fake generated images in more detail. The discriminators use LeakyReLU activation layers during downsampling, which maintain gradients when the input is negative, increasing the network's sensitivity to these features and improving discrimination. The instantiation layer inserted in each convolution block performs individual normalization for each sample, allowing the model to focus on the style of individual samples to better adapt to various data distributions, rather than normalizing the entire batch and causing different image styles to influence each other. The last layer of the discriminator is a Sigmoid activation layer that outputs a feature vector to the range (0, 1). The length of this vector varies depending on the image size and the number of convolution layers, and each element in the final output vector represents the probability that the local region of the image is a real image.

[0105] Embodiment 3

[0106] In this embodiment, based on the above embodiments, the training process of the medical image conversion model is further described.

[0107] In step S5, after training the medical image conversion model, the one-stage model is trained using the training set of the one-stage data set, and the two-stage model is trained using the training set of the two-stage data set. The process includes:

[0108] S51. Freeze the weights of the discriminator first, and train the generator: input the real MRI image data and CT image data in the one-stage data set and the two-stage data set into the corresponding generator respectively, generate fake synthetic CT image data and MRI image data, and extract high-dimensional content features from the real MRI image data and CT image data;

[0109] Input the fake synthetic CT image data and MRI image data into the corresponding generator for reconstruction loss, respectively generate the reconstruction image data of the synthetic CT image data and MRI image data, and extract the high-dimensional content features from the synthetic CT image data and MRI image data;

[0110] Calculate the reconstruction loss between the real MRI image data and CT image data and the corresponding reconstruction image data, and use the reconstruction loss for backpropagation training of the generator parameters;

[0111] Calculate the content loss between the high-dimensional content features extracted from the real MRI image data and CT image data and the high-dimensional content features extracted from the synthetic CT image data and MRI image data, and use the content loss to optimize the parameters of the generator;

[0112] S52. Freeze the weights of the generator and train the discriminator: label the real MRI image data and CT image data as True and input them into the corresponding discriminator, label the synthetic CT image data and MRI image data generated by the generator as False and input them into the corresponding discriminator, and use the calculated loss to optimize the parameters of the discriminator;

[0113] S53. Repeat steps S51 and S52, alternately train the generator and the discriminator, update the corresponding weight parameters each time, and obtain a trained one-stage model and a two-stage model;

[0114] S54. Fuse the one-stage model and the two-stage model to obtain a trained medical image conversion model, wherein the CT image data output by the one-stage model is extracted and superimposed on the MRI image data after the MRI image data is input into the one-stage model, and the MRI image data with enhanced bone is input into the two-stage model, and the output of the two-stage model is the final target synthetic CT image data.

[0115] In this embodiment, three better one-stage models are determined based on SSIM and visual assessment, and three better two-stage models are determined based on the comprehensive performance of four evaluation indexes of structural similarity index (SSIM), peak signal to noise ratio (PSRN), mean squared error (MSE), and learned perceptual image patch similarity (LPIPS) and visual assessment. The better one-stage model and two-stage model selected in the above steps are fused to find the best medical image conversion model based on the comprehensive performance of four evaluation indexes of SSIM, PSNR, MSE, and LPIPS and visual assessment.

[0116] SSIM is an image generation quality evaluation index, which can measure the similarity between two images, and the range is between-1 and 1, and the closer the value is to 1, the more similar the two images are. SSIM calculates the similarity based on three factors: brightness contrast, structure contrast and spatial contrast. Among them, the brightness contrast measures the brightness difference between two images, the structure contrast measures the texture difference between two images, and the spatial contrast measures the position difference between two images. By weighted average of the three factors, the similarity value is finally obtained. Its formula is as follows:

[0117]

[0118] wherein, μ x and μy Let x and y represent the mean values ​​of the image, respectively. and Let σ represent the variances of the images x and y, respectively. xy c represents the covariance of the images x and y. i c2 and c2 are two small constants; the closer the value of SSIM is to 1, the more similar the two images are.

[0119] PSNR is primarily used to measure the impact of image compression or image processing on image quality. It represents the peak signal-to-noise ratio between an image and the original image; a higher value indicates higher image quality. The formula for calculating PSNR is shown below:

[0120]

[0121] Where MAX is the maximum value of a pixel, which is 255 in this embodiment; MSE is the average value of the sum of squared image errors. When MSE approaches zero, the PSNR value approaches infinity, which indicates that the higher the PSNR value, the better the image quality.

[0122] MSE is a commonly used metric in image generation quality assessment, used to predict the mean squared difference between the generated image and the real image. Its formula is shown below:

[0123]

[0124] Where, x i, These are the actual image coordinates, g(x) i, ) represents the predicted image, where m and n are the height and width of the image, respectively. The smaller the MSE, the smaller the difference between the predicted image and the real image, and the closer the prediction result is to the true value.

[0125] Perceptual loss learns to perceive the similarity of image patches. It uses a pre-trained convolutional neural network to represent the features of an image, and then uses the differences in the feature maps to calculate a score of similarity between the generated image and the target domain image. Perceptual loss can capture subtle image differences that have a significant impact on human visual perception, providing a quality metric that is more consistent with human visual evaluation.

[0126] Furthermore, during the iteration process of step S5, the parameters of each stage model are adjusted by backpropagating the overall minimization hybrid loss function constructed for each stage model; the overall minimization hybrid loss function includes: adversarial loss, cycle consistency loss, content constraint loss, skeletal structure constraint loss and VGG network loss;

[0127] The MRI image data set in the paired data is denoted as domain A, and the CT image data set is denoted as domain B; any image data in domain A is denoted as realA, i.e. real MRI image data, and any image data in domain B is denoted as realB, i.e. real CT image data; the synthetic CT image data generated by the generator from realA is denoted as fakeB, and the synthetic MRI image data generated by the generator from realB is denoted as fakeA; the CT reconstructed image data generated by the generator from fakeA is denoted as recB, and the MRI reconstructed image data generated by the generator from fakeB is denoted as recA; BA1 and G BA2 generated by the generator is denoted as recA;

[0128] The high-dimensional content feature containing realA in domain A output by the intermediate layer of the generator is denoted as The high-dimensional content feature containing realB in domain B is denoted as The high-dimensional content feature containing fakeA in domain A is denoted as The high-dimensional content feature containing fakeB in domain B is denoted as The content constraint loss is constructed, and the content constraint loss The formula is as follows:

[0129]

[0130] wherein, represents and constitute the new content constraint in domain A, and the represents and constitute the new content constraint in domain B;

[0131] The skeletal structure constraint includes the L1 norm distance constraint between realA and fakeA, and the L2 norm distance constraint between realB and fakeB, and the formula is as follows:

[0132]

[0133] wherein, ∥1 represents using L1 norm, and ∥2 represents using L2 norm;

[0134] The VGG network constraint performs semantic constraint on the extracted feature map, and the formula is as follows:

[0135]

[0136] wherein, φ represents the output of a layer or multiple layers of the VGG network;

[0137] The overall minimum hybrid loss function of each stage model The formula is as follows:

[0138]

[0139] Wherein, n e {1, 2}; And Consistent with the adversarial loss in CycleGAN; Consistent with the cycle consistency loss in CycleGAN; L1 norm distance constraint, L2 norm distance constraint, respectively adjusted by λ L1 And λ L2 Two hyperparameters; VGG network distance constraint loss, λ VGG Is a hyperparameter for regulating the VGG network loss; Content loss, λ feature Is a hyperparameter for adjusting the content loss, in the embodiment, the hyperparameter of the content loss of the two-stage model is adjusted to 0.

[0140] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary and impossible to exhaust all the embodiments. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A medical image conversion method of abdominal MRI-synthetic CT, characterized by, The method comprises the following steps: S1. Collecting a plurality of sets of MRI and CT image data samples of human abdominal regions; S2. Rigidly registering the MRI image data and the CT image data, and taking the registered paired data as a first-stage data set; S3. Respectively performing image data enhancement on the MRI image data and the CT image data obtained in step S2, including: performing N4 correction on the MRI image data, and performing baffle removal processing on the CT image data; S4. Extracting a bone feature map of the CT image data, and fusing the extracted bone feature map into the paired MRI image data to obtain a second-stage data set; S5. Based on the CycleGAN network, a medical image conversion model is constructed, and the medical image conversion model includes a one-stage model and a two-stage model; the one-stage model is used to extract the skeleton framework of the target CT, including a generator 、 , a discriminator 、 and a cross-modal content structure , which converts the input MRI image data into CT image data, which converts the input CT into MRI; which is used to judge the authenticity of the input MRI image data, which is used to judge the authenticity of the input CT image data; which is arranged between and , used to constrain the intermediate feature layer of and , realizing the feature alignment of cross-modal; The two-stage model is used to enhance the skeleton frame style, and a higher quality CT style image is generated in combination with the skeleton frame content extracted by the one-stage model, including a generator 、 and a discriminator 、 ; The converting input MRI image data of an enhanced bone into CT image data, the converting input CT into MRI image data; the for judging the authenticity of input MRI image data, the for judging the authenticity of input CT image data; training the first-stage model and the second-stage model by using the first-stage data set and the second-stage data set, and fusing the trained first-stage model and the second-stage model to obtain a trained medical image conversion model; S6. Inputting the MRI image data to be converted into the trained medical image conversion model, and outputting corresponding synthetic CT image data, thereby completing the cross-modal conversion of the medical image data.

2. The medical image converting method of an abdominal MRI synthetic CT according to claim 1, characterized in that, In step S1, the image data samples of the abdominal MRI and CT are collected, and the process comprises the following steps: screening the MRI and CT DICOM image data of the abdominal region, saving the MRI image data as a T1 sequence, and saving the CT image data as a venous phase sequence.

3. The medical image converting method of an abdominal MRI synthetic CT according to claim 2, characterized in that, The process of step S2 comprises the following steps: S21. Importing the MRI and CT image data samples of one case to be paired; S22. Respectively performing medical image three-dimensional reconstruction on the MRI image data and the CT image data, keeping the three-dimensional coordinates of the CT image data unchanged, and performing coordinate translation on the three-dimensional reconstructed object of the MRI image data, so that the three-dimensional reconstructed abdominal structure of the MRI image data completely overlaps the three-dimensional reconstructed abdominal structure of the CT image data; S23. Performing slice registration on the MRI image data and the CT image data, outputting and saving the registered MRI image data in DICOM format, and obtaining MRI image data with the same number of slices and slice levels as the CT image data; S24. Judging whether the rigid registration of all case data is completed, if yes, saving the paired data of all the MRI image data and the CT image data obtained after registration as a first-stage data set, otherwise, returning to step S21.

4. The medical image converting method of an abdominal MRI synthetic CT according to claim 1, characterized in that, The process of step S4 comprises the following steps: S41. Setting a pixel value threshold, extracting pixel points with a pixel value greater than the threshold in all CT image data, and saving them as a feature map; S42. Performing corrosion and expansion and filtering processing on the threshold extracted feature map to remove noise points, and obtaining a bone feature map of the CT image data; S43. Superimposing the bone feature map extracted from the CT image data onto the MRI image data to obtain an MRI data set with enhanced bones, which is taken as a second-stage data set of the model.

5. The medical image converting method of an abdominal MRI synthetic CT according to claim 1, characterized in that, Further comprising: Dividing the first-stage data set and the second-stage data set into a training set, a validation set and a test set respectively; training the first-stage model and the second-stage model by using the training set, then evaluating the first-stage model and the second-stage model by using the validation set, testing the effectiveness of the first-stage model and the second-stage model by using the test set, and obtaining the trained first-stage model and the second-stage model.

6. The medical image converting method of an abdominal MRI synthetic CT according to claim 5, characterized in that, In step S5, after training the medical image conversion model, the training set of the one-stage data set is used to train the one-stage model, and the training set of the two-stage data set is used to train the two-stage model, and the process includes: S51. Freeze the weight of the discriminator first, train the generator: input the real MRI image data and CT image data in the one-stage data set and the two-stage data set into the corresponding generator respectively, generate false synthetic CT image data and MRI image data, and high-dimensional content features extracted from the real MRI image data and CT image data; The false synthetic CT image data and MRI image data are input into the corresponding generator for reconstruction loss, respectively generating the reconstruction image data of the synthetic CT image data and MRI image data, and the high-dimensional content features extracted from the synthetic CT image data and MRI image data; The reconstruction loss between the real MRI image data and CT image data and the corresponding reconstruction image data is calculated, and the reconstruction loss is used for back propagation training of the generator parameters; The content loss between the high-dimensional content features extracted from the real MRI image data and CT image data and the high-dimensional content features extracted from the synthetic CT image data and MRI image data is calculated, and the content loss is used to optimize the parameters of the generator; S52. Freeze the weight of the generator, train the discriminator: mark the real MRI image data and CT image data as True and input them into the corresponding discriminator, mark the synthetic CT image data and MRI image data generated by the generator as False and input them into the corresponding discriminator, and use the calculated loss to optimize the parameters of the discriminator; S53. Repeat steps S51 and S52, alternately train the generator and the discriminator, update the corresponding weight parameters each time, and obtain the trained one-stage model and two-stage model; S54. Fuse the one-stage model and the two-stage model to obtain the trained medical image conversion model.

7. The medical image converting method of an abdominal MRI synthetic CT according to claim 6, characterized in that, In the iteration process of step S5, the parameter back propagation is performed by constructing the overall minimization hybrid loss function of each stage model to adjust the parameters of each stage model; the overall minimization hybrid loss function includes: adversarial loss, cycle consistency loss, content constraint loss, skeletal structure constraint loss and VGG network loss; Let the MRI image data set in the paired data be denoted as domain A, and the CT image data set be denoted as domain B; let any image data in domain A be denoted as realA, i.e., real MRI image data, and any image data in domain B be denoted as realB, i.e., real CT image data; let the synthetic CT image data generated by the generator from realA be denoted as fakeB, and the synthetic MRI image data generated by the generator from realB be denoted as fakeA; let the CT reconstructed image data generated by the generator from fakeA be denoted as recB, and the MRI reconstructed image data generated by the generator from fakeB be denoted as recA; and ​ Let the high-dimensional content feature containing realA on A domain outputted by the generator intermediate layer be denoted as Let the high-dimensional content feature containing realB on B domain be denoted as Let the high-dimensional content feature containing fakeA on A domain be denoted as Let the high-dimensional content feature containing fakeB on B domain be denoted as The content constraint loss is constructed, and the content constraint loss The formula is as follows: wherein represents with constitute content constraints on the new A domain, said represents with constitute content constraints on the new B domain; The skeletal structure constraint includes the L1 norm distance constraint between realA and fakeA, and the L2 norm distance constraint between realB and fakeB, and the formula is as follows: wherein, denotes the use of the L1 norm, denotes the use of the L2 norm; The VGG network constraint performs semantic constraint on the extracted feature map, and the formula is as follows: wherein, represents the output of a certain layer or layers of the VGG network; Overall minimized hybrid loss function for each stage model The formula is as follows: + wherein, ; and consistent with the adversarial loss in CycleGAN; consistent with the cycle consistency loss in CycleGAN; is an L1 norm distance constraint, is an L2 norm distance constraint, respectively adjusted by and two hyperparameters; is a distance constraint loss of VGG network, is a hyperparameter to regulate the VGG network loss; is a content loss, is a hyperparameter to adjust the content loss.

8. The medical image converting method of an abdominal MRI synthetic CT according to claim 5, characterized in that, In step S5, the one-stage model and the two-stage model are evaluated by using the verification set, and the evaluation indexes of the one-stage model include structural similarity index, and the evaluation indexes of the two-stage model include structural similarity index, peak signal-to-noise ratio, mean square error and perception loss. 9.The medical image converting method of an abdominal MRI synthetic CT according to claim 1, wherein, The specific process of step S6 includes: inputting the MRI image data to be converted into the trained medical image conversion model, extracting the bone framework of the false CT generated by the one-stage model, inputting the bone framework and the input MRI into the generator of the two-stage model, fusing the MRI and the bone framework and converting the CT style through the generator of the two-stage model, and outputting the corresponding synthetic CT image data.

Images