Method, device, equipment and readable storage medium for synthesizing ct image from mri

Abstract

The application relates to an MRI-CT image synthesis method, device and equipment and a readable storage medium, relates to the medical image synthesis technical field, and comprises the following steps: constructing a training data set comprising MRI image data and CT image data; constructing an improved cycle generative adversarial network model comprising a space domain information maintaining generator and a spectral normalization discriminator and a corresponding target function of the improved cycle generative adversarial network model, wherein the target function comprises an adversarial loss term, a frequency domain consistency loss term and an identity mapping loss term; training the improved cycle generative adversarial network model based on the training data set to obtain an image synthesis model; and converting a to-be-converted MRI image into a synthesized CT image based on the image synthesis model. The application effectively improves image details and overall image synthesis quality through the frequency domain consistency loss, maintains the consistency between a real image and a synthesized image from a space domain level through the space domain information maintaining generator, and improves the stability and reliability of a network training process through the spectral normalization discriminator.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image synthesis, and in particular to a method and device for synthesizing CT images from MRI images, equipment and a readable storage medium. BACKGROUND

[0002] Image-guided radiotherapy is a main means of cancer treatment. Among them, computed tomography (CT) and magnetic resonance imaging (MRI) are two medical images widely used in radiotherapy planning. Although CT images provide CT values in Hounsfield units (Hu) that can be directly used for dose calculation, they lack clear soft tissue contrast; while MRI has higher soft tissue contrast than CT and no ionizing radiation, so in clinical practice, CT and MRI are often combined using image registration technology to help the target area to be outlined more accurately in radiotherapy planning. However, since taking two scans will not only cause certain economic burden and radiation damage to the patient, and the registration error of CT and MRI will also affect the accuracy of subsequent treatment, if MRI images can be synthesized into CT images, radiotherapy planning based only on MRI will be possible.

[0003] In related technologies, traditional MRI-to-CT synthesis methods mainly rely on manual segmentation of image tissues and organs and image registration technology, which not only has a large workload, but also has poor synthesis image accuracy. With the rapid development of deep learning, CT synthesis methods based on deep convolutional neural networks and generative adversarial networks have received widespread attention. Such methods use a large amount of MRI and CT paired data to train deep models in a supervised manner, allowing the model to learn the mapping relationship from MRI to CT. However, it is not easy to obtain MRI and CT images of the same patient in clinical practice, resulting in a lack of paired MRI and CT data sets. In the absence of paired data sets, many researchers have used CycleGAN and non-paired data sets to implement unsupervised MRI-to-CT image synthesis, effectively avoiding the dependence on paired data sets.

[0004] Although MRI-to-CT synthesis methods based on CycleGAN have made great progress at this stage, there are still the following challenges: due to the different imaging methods of MRI and CT images, the density mapping from MRI to CT images is highly complex, and traditional CycleGAN lacks direct constraints between input images and synthesized images, so it cannot guarantee the spatial feature consistency before and after cross-domain image synthesis; in addition, the error of MRI-to-CT image synthesis mainly occurs in the loss of high-frequency parts of the image (i.e. outline, boundary, etc.), resulting in blurred synthesized images and low quality. SUMMARY

[0005] The application provides a method and device for synthesizing CT images from MRI images, an equipment and a readable storage medium, to solve the problem of poor image quality caused by lack of spatial domain information preservation ability and loss of high frequency information in the prior art.

[0006] In a first aspect, a method for synthesizing CT images from MRI images is provided, comprising the following steps:

[0007] Obtaining MRI image data and CT image data to construct a training data set;

[0008] Constructing an improved cycle generative adversarial network model comprising a spatial domain information preservation generator and a spectral normalization discriminator and a corresponding objective function thereof, the objective function comprising an adversarial loss term, a frequency domain consistency loss term and an identity mapping loss term;

[0009] Training the improved cycle generative adversarial network model based on the training data set to obtain an image synthesis model;

[0010] Converting a to-be-converted MRI image into a synthesized CT image based on the image synthesis model.

[0011] In some embodiments, the objective function is:

[0012] L(G, F, D MR , D CT ) = L GAN (G, D CT , X, Y) + L GAN (F, D MR , Y, X) + λ1L fc (G, F, X, Y) + λ2L identity (G, F)

[0013] In the formula, L(G, F, D MR , D CT ) represents the objective function, G represents a first spatial domain information preservation generator, F represents a second spatial domain information preservation generator, D MR represents a first spectral normalization discriminator, D CT represents a second spectral normalization discriminator, L GAN (G, D CT , X, Y) represents an adversarial loss between G and D CT , X and Y represent an MRI image data set and a CT image data set respectively, L GAN (F, D MR , Y, X) represents an adversarial loss between F and D MR , L fc (G, F, X, Y) represents a frequency domain consistency loss between G and F, and L identity(G, F) represents the identity mapping loss between G and F, and λ1 and λ2 represent hyperparameters that control the loss term weights.

[0014] In some embodiments, L GAN (G, D CT , X, Y) is calculated as follows:

[0015]

[0016] L GAN (F, D MR , Y, X) is calculated as follows:

[0017]

[0018] where G(x) represents the pseudo CT image synthesized by G, D CT (y) represents the discrimination result of D CT on the real CT image, D CT (G(x)) represents the discrimination result of D CT on the pseudo CT image, F(y) represents the pseudo MRI image synthesized by F, D MR (x) represents the discrimination result of D MR on the real MRI image, D MR (F(y)) represents the discrimination result of D MR on the pseudo MRI image.

[0019] In some embodiments, the frequency domain consistency loss L fc (G, F, X, Y) is calculated as follows:

[0020]

[0021] where L ff (x, F(G(x))) represents the focal frequency loss between the input MRI image x and the cyclically reconstructed MRI image F(G(x)) of the input MRI image x, and L ff (y, G(F(y))) represents the focal frequency loss between the input CT image y and the cyclically reconstructed CT image G(F(y)) of the input CT image y.

[0022] In some embodiments, the identity mapping loss L identity (G, F) is calculated as follows:

[0023]

[0024] where G(x) represents the pseudo CT image synthesized by G, and F(y) represents the pseudo MRI image synthesized by F.

[0025] In some embodiments, the spatial information preserving generator comprises an encoder, a transformer and a decoder, the encoder is configured to extract MRI features, the encoder comprises a dynamic region convolution configured to adaptively assign filters, the transformer is configured to perform feature fusion on the MRI features extracted by the encoder, and the decoder is configured to convert the fused features output by the transformer into a synthetic image.

[0026] In some embodiments, the dynamic region convolution comprises a first convolution layer, a serial assignment layer and a second convolution layer.

[0027] The first convolution layer is configured to generate a plurality of feature maps carrying different mask guides.

[0028] The serial assignment layer is configured to assign a corresponding filter to each feature map.

[0029] The second convolution layer is configured to perform convolution operation on the feature map corresponding thereto based on the different filters.

[0030] In a second aspect, an apparatus for synthesizing CT images from MRI images is provided, comprising:

[0031] a data construction unit configured to obtain MRI image data and CT image data to construct a training data set;

[0032] a model function construction unit configured to construct an improved cycle generative adversarial network model comprising a spatial information preserving generator and a spectral normalization discriminator and a corresponding target function of the improved cycle generative adversarial network model, the target function comprising an adversarial loss term, a frequency domain consistency loss term and an identity mapping loss term;

[0033] a model training unit configured to train the improved cycle generative adversarial network model based on the training data set to obtain an image synthesis model;

[0034] an image synthesis unit configured to convert a to-be-converted MRI image into a synthetic CT image based on the image synthesis model.

[0035] In a third aspect, an apparatus for synthesizing CT images from MRI images is provided, comprising a memory and a processor, the memory stores at least one instruction, the at least one instruction is loaded and executed by the processor to implement the method for synthesizing CT images from MRI images as described above.

[0036] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, to implement the method for synthesizing CT images from MRI images as described above.

[0037] The application provides a method, device and equipment for synthesizing CT images from MRI images, and a readable storage medium, which comprises the following steps: acquiring MRI image data and CT image data to construct a training data set; constructing an improved cycle generative adversarial network model comprising a spatial information maintaining generator and a spectral normalization discriminator and a corresponding target function of the improved cycle generative adversarial network model, wherein the target function comprises an adversarial loss term, a frequency domain consistency loss term and an identity mapping loss term; training the improved cycle generative adversarial network model based on the training data set to obtain an image synthesis model; and converting a to-be-converted MRI image into a synthesized CT image based on the image synthesis model. The application effectively constrains the consistency between a real image and a synthesized image from a frequency domain level through the frequency domain consistency loss, so as to guide the model to pay more attention to the frequency part that is difficult to synthesize, thereby effectively improving the image details and the overall synthesis quality of the image, and the spatial information maintaining generator is used to improve the extraction ability of the network model for deep features, thereby maintaining the consistency between the real image and the synthesized image from a spatial domain level, and the spectral normalization discriminator is used to improve the stability and reliability of the network training process. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0039] Figure 1 A flowchart of a method for synthesizing CT images from MRI images provided by the embodiments of the application is shown in the figure.

[0040] Figure 2 A schematic diagram of the overall flow of the method for synthesizing CT images from MRI images provided by the embodiments of the application is shown in the figure.

[0041] Figure 3 A structure diagram of the improved cycle generative adversarial network model provided by the embodiments of the application is shown in the figure.

[0042] Figure 4 A structure diagram of the spectral normalization discriminator provided by the embodiments of the application is shown in the figure.

[0043] Figure 5 A structure diagram of the spatial information maintaining generator provided by the embodiments of the application is shown in the figure.

[0044] Figure 6 A structure diagram of the device for synthesizing CT images from MRI images provided by the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0045] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0046] The embodiments of the present application provide a method, device, equipment and readable storage medium for synthesizing CT images from MRI, which can solve the problem of poor quality of synthesized images due to lack of spatial domain information preservation capability and loss of high frequency information in the related art.

[0047] Referring to Figures 1 to 3 The embodiments of the present application provide a method for synthesizing CT images from MRI, including the following steps:

[0048] Step S10: acquiring MRI image data and CT image data to construct a training data set;

[0049] Exemplarily, it should be understood that in the process of constructing the training data set, the acquired MRI image data and CT image data will be preprocessed.

[0050] Specifically, a certain proportion of image samples are selected from the MRI image data set and the CT image data set as preprocessing objects; then the preprocessing objects (i.e. the MRI image and the CT image) are sliced, such as slicing along the axial section at a predetermined ratio, and the slices containing less information than the set value are removed, to obtain a 2D slice data set of the MRI and a 2D slice data set of the CT; the density range of the MRI is preferably set to [0, 1000], and the density range of the CT is preferably set to [-700, 1300], and then the above density ranges are uniformly linearly normalized to the range of [-1, 1] respectively; then the image is padded, which can be preferably cropped to a size of 256x256, to obtain a preprocessed MRI image data set and a preprocessed CT image data set wherein N and M represent the number of images of the MRI and the CT respectively. As can be seen, the training data set in the embodiments includes the preprocessed MRI image data set X and the preprocessed CT image data set Y.

[0051] Step S20: constructing an improved cycle generative adversarial network model including a spatial domain information preservation generator and a spectral normalization discriminator and a corresponding objective function of the improved cycle generative adversarial network model, the objective function including an adversarial loss term, a frequency domain consistency loss term, and an identity mapping loss term;

[0052] As an example, in this embodiment, an improved recurrent generative adversarial network (GAN) deep learning network model for synthesized MRI CT images will be constructed. See also... Figure 3 As shown, the improved recurrent generative adversarial network model preferably includes two spatial information preservation generator networks (i.e., the first spatial information preservation generator G and the second spatial information preservation generator F, hereinafter referred to as generator G and generator F for the sake of simplicity) and two spectral normalization discriminator networks (i.e., the first spectral normalization discriminator D). MR Second spectral normalized discriminant D CT For the sake of brevity, it will be referred to as discriminator D for short. MR and discriminator D CT ).

[0053] It should be noted that in this embodiment, generator G and generator F have the same network results, and discriminator D... MR and discriminator D CT They have the same network structure. Specifically, generator G generates a CT image based on the input MRI image, generator F generates an MRI image based on the input CT image; discriminator D... MR Discriminant D distinguishes between synthetic and real MRI images. CT The system determines the authenticity of synthesized CT images from real CT images. It should be understood that the generator G and the discriminator D... MR The forward recurrent generative adversarial network (GAN) that forms the MRI-to-CT transmission chain consists of a generator F and a discriminator D. CT A reverse recurrent generative adversarial network is constructed from CT to MRI.

[0054] The network structure of the spectral normalization discriminator in this embodiment is shown in [reference needed]. Figure 4 As shown, it consists of multiple spectral normalized convolutional blocks and activation functions (such as LeakyReLU and sigmoid activation functions), with the discriminator D... MR and discriminator D CT Both are used to determine whether the generated image is a synthetic image or a real image, and output the probability that the image is a real image. This embodiment improves the stability and reliability of the network training process by introducing spectral normalization into the discriminator, and improves the quality of the synthetic image through the game process between the generator and the discriminator.

[0055] Furthermore, this embodiment will also construct an objective function for the improved recurrent generative adversarial network, which includes an adversarial loss term, a frequency domain consistency loss term, and an identity mapping loss term. The adversarial loss term includes the generator G and the discriminator D. CT The adversarial loss between the generator F and the discriminator D MRthe adversarial loss between them. It can be understood that, in order to ensure the consistency between the input image and the reconstructed image in the frequency domain and guide the model to pay more attention to the frequency part that is difficult to synthesize, the embodiment will introduce a frequency domain consistency loss to replace the cycle consistency loss in the original cycle generative adversarial network, and establish a constraint on the input image and the cycle reconstructed image from the frequency domain level to improve the image details and the overall synthesis quality of the image.

[0056] Further, the spatial domain information maintaining generator comprises an encoder, a transformer and a decoder, the encoder is configured to extract MRI features, the encoder comprises a dynamic region convolution configured to adaptively assign filters, the transformer is configured to fuse the MRI features extracted by the encoder, and the decoder is configured to convert the fused features output by the transformer into a synthetic image.

[0057] For example, in the embodiment, the network structure of the spatial domain information maintaining generator is shown in FIG. 2, which mainly comprises three parts of an encoder, a transformer and a decoder connected in series. Taking the generator G as an example, the encoder is configured to extract features related to MRI, the transformer is configured to integrate the features extracted by the encoder, and the decoder is configured to generate a CT synthetic image according to the features output by the transformer. Figure 5

[0058] Specifically, the encoder is a downsampling process, which is configured to extract features related to the source domain image (i.e. the MRI image in the embodiment), and specifically comprises three downsampling modules and a dynamic region convolution. Each downsampling module is composed of a convolution layer, an instance normalization layer and a ReLU activation function layer, wherein the convolution kernel sizes are 7x7, 3x3 and 3x3 respectively; in order to ensure the consistency of the spatial domain information of the cross-domain image synthesis, the embodiment will introduce a dynamic region convolution to adaptively assign the same filter to the same category of region.

[0059] The transformer is configured to integrate the MRI features extracted by the encoder, and the network structure thereof comprises a plurality of residual blocks connected in series, and the structure of each residual block is the same, specifically comprising a 3x3 convolution layer, an instance normalization layer, a ReLU layer, a 3x3 convolution layer and an instance normalization layer. The output result of the last instance normalization layer will be combined with the input of the residual block (i.e. the MRI features output by the encoder) using a residual connection to obtain fused features. The decoder is configured to generate the target domain image according to the fused features output by the transformer, and the network structure thereof comprises two deconvolution blocks, one instance normalization layer, a ReLU activation function, one 7x7 convolution layer and a Tanh activation function, which converts the fused features output by the transformer into a synthetic image.

[0060] Further, the dynamic region convolution comprises a first convolution layer, a serial assignment layer and a second convolution layer.

[0061] ​The first convolutional layer is configured to generate a plurality of feature maps carrying different mask guides;

[0062] The series distribution layer is configured to assign a corresponding filter to each feature map;

[0063] The second convolutional layer is configured to perform convolution operation on the feature map corresponding to the different filter.

[0064] Exemplarily, in the embodiment, the dynamic region convolution mainly includes a 3*3 convolutional layer (i.e., the first convolutional layer), a series distribution layer composed of a series of adaptive average pooling layer, 1*1 convolutional layer, sigmoid activation function and 1*1 convolutional layer, and another 3*3 convolutional layer (i.e., the second convolutional layer); wherein a 3*3 convolutional layer is used to generate a feature map f guided carrying four kinds of mask guides; that is, four feature maps carrying different mask guides are generated; at the same time, different filters are assigned to different mask regions through a series of adaptive average pooling layer, 1*1 convolutional layer, sigmoid activation function and 1*1 convolutional layer; and then another 3*3 convolutional layer is used to perform convolution operation on the feature map f guided corresponding to the corresponding filter.

[0065] Further, the objective function is:

[0066] L(G, F, D MR , D CT ) = L GAN (G, D CT , X, Y) + L GAN (F, D MR , Y, X) + λ1L fc (G, F, X, Y) + λ2L identity (G, F)

[0067] In the formula, L(G, F, D MR , D CT ) represents the objective function, G represents the first spatial domain information preserving generator, F represents the second spatial domain information preserving generator, D MR represents the first spectral normalization discriminator, D CT represents the second spectral normalization discriminator, L GAN (G, D CT , X, Y) represents the adversarial loss between G and D CT , X and Y represent the MRI image dataset and the CT image dataset respectively, L GAN (F, D MR , Y, X) represents the adversarial loss between F and D MR , and L fc(G, F, X, Y) represents the frequency domain consistency loss between G and F, L identity (G, F) represents the identity mapping loss between G and F, λ1 and λ2 represent hyperparameters for controlling the weight of the loss term.

[0068] Exemplarily, in the embodiment, the sum of the adversarial loss term, the frequency domain consistency loss term and the identity mapping loss term is taken as the final objective function L(G, F, D MR , D CT ) of the improved cycle generative adversarial network, as shown in the following formula:

[0069] L(G, F, D MR , D CT )=L GAN (G, D CT , X, Y)+L GAN (F, D MR , Y, X)+λ1L fc (G, F, X, Y)+λ2L identity (G, F)

[0070] Wherein, L GAN (G, D CT , X, Y) represents the adversarial loss between the generator G and the discriminator D CT , X and Y respectively represent the MRI image dataset and the CT image dataset, L GAN (F, D MR , Y, X) represents the adversarial loss between the generator F and the discriminator D MR , L fc (G, F, X, Y) represents the frequency domain consistency loss between the generator G and the generator F, L identity (G, F) represents the identity mapping loss between the generator G and the generator F, λ1 and λ2 are hyperparameters for controlling the weight of the loss term, which can be preferably set as λ1=5 and λ2=1.

[0071] Further, the calculation formula of L GAN (G, D CT , X, Y) in the adversarial loss term is:

[0072]

[0073] The calculation formula of L GAN (F, D MR , Y, X) in the adversarial loss term is:

[0074]

[0075] In the formula, G(x) represents the pseudo-CT image synthesized by G, D CT(y) represents D CT The discrimination result of real CT images, D CT (G(x)) represents D CT The discrimination result of the pseudo-CT image, F(y) represents the pseudo-MRI image synthesized by F, and D MR (x) represents D MR The discrimination results of real MRI images, D MR (F(y)) represents D MR Results of identifying fake MRI images.

[0076] As an example, in this embodiment, the adversarial loss term includes the generator G and the discriminator D. CT The adversarial loss between the generator F and the discriminator D MR The adversarial loss between them. Specifically, the generator G and discriminator D are calculated according to the first formula. CT The conflict between them resulted in losses L GAN (G, D) CT (X, Y), the generator F and discriminator D are obtained according to the second calculation formula. MR The losses in the confrontation between them;

[0077] The first calculation formula is:

[0078]

[0079] In the formula, Represents an MRI image dataset. Let G(x) represent the CT image dataset, G(x) represent the pseudo-CT images synthesized by the generator G, and D represent the pseudo-CT images synthesized by the generator G. CT (y) represents the discriminator D. CT The discrimination result of real CT images, D CT (G(x)) represents the discriminator D CT Results of identifying fake CT images.

[0080] The second calculation formula is:

[0081]

[0082] In the formula, F(y) represents the pseudo-MRI image synthesized by generator F, and D MR (x) represents the discriminator D. MR The discrimination results of real MRI images, D MR (F(y)) represents the discriminator D MR Results of identifying fake MRI images.

[0083] Furthermore, the frequency domain consistency loss L fc The formula for calculating (G, F, X, Y) is:

[0084]

[0085] In the formula, L ff (x, F(G(x))) represents the focal frequency loss between the input MRI image x and the cyclically reconstructed MRI image F(G(x)), L ff (y, G(F(y))) represents the focal frequency loss between the input CT image y and the cyclically reconstructed CT image G(F(y)).

[0086] Exemplarily, in the embodiment, the specific implementation process of the frequency domain consistency loss includes:

[0087] According to the third calculation formula, 2D Fourier transform is performed on the input image and the synthesized image respectively to obtain the frequency representation in the spectral coordinates;

[0088] According to the fourth calculation formula, the weight value corresponding to each spectral coordinate is calculated;

[0089] According to the fifth calculation formula, the focal frequency loss of the input image and the synthesized image is obtained;

[0090] According to the sixth calculation formula, the focal frequency loss between the real MRI image and the reconstructed synthesized MRI image and the focal frequency loss between the real CT image and the reconstructed synthesized CT image are calculated respectively, and then the final frequency domain consistency loss is obtained.

[0091] The third calculation formula is:

[0092]

[0093] In the formula, F x (u, v) represents the frequency value at the spectral coordinates (u, v) corresponding to the input image x, (p, q) represents the pixel space coordinates of the input image x with the size of MxN, f(p, q) represents the image pixel value at the position (p, q), i represents the imaginary unit, e represents Euler number, which follows Euler formula d iθ = cosθ + isinθ.

[0094] The fourth calculation formula is:

[0095]

[0096] In the formula, w(u, v) represents the weight value at the spectral coordinates (u, v), x represents the real image, represents the reconstructed image.

[0097] The fifth calculation formula is:

[0098]

[0099] wherein, represents the focal frequency loss between the real image x and the reconstructed image .

[0100] The sixth calculation formula is:

[0101]

[0102] wherein, L ff (x, F(G(x))) represents the focal frequency loss between the input MRI image x and the cyclically reconstructed MRI image F(G(x)), L ff (y, G(F(y))) represents the focal frequency loss between the input CT image y and the cyclically reconstructed CT image G(F(y)).

[0103] Further, the identity mapping loss L identity (G, F) is calculated according to the following formula:

[0104]

[0105] wherein, G(x) represents the G-synthesized pseudo CT image, and F(y) represents the F-synthesized pseudo MRI image.

[0106] Exemplarily, in the present embodiment, according to the seventh calculation formula, the sum of the L1 loss between the input MRI image and the synthesized pseudo CT image thereof and the L1 loss between the input CT image and the synthesized pseudo MIR image thereof is obtained to obtain the identity mapping loss. The seventh calculation formula is:

[0107]

[0108] wherein, G(x) represents the G-synthesized pseudo CT image, and F(y) represents the F-synthesized pseudo MRI image, ||G(x)-x||1 represents the L1 loss between the input MRI image and the synthesized pseudo CT image thereof, and ||F(y)-y||1 represents the L1 loss between the input CT image and the synthesized pseudo MIR image thereof.

[0109] Step S30: training the improved cyclic generative adversarial network model based on the training data set to obtain an image synthesis model;

[0110] Exemplarily, in the embodiment, after the training data set is obtained and the improved cycle generative adversarial network and the corresponding objective function are constructed, the improved cycle generative adversarial network is trained by using the training data set. The training process aims to minimize the value of the objective function, and a trained network model is obtained according to the optimal weight parameters of the discriminator and the generator. The trained CT generation network model is used as a final prediction model (i.e., an image synthesis model).

[0111] Step S40: converting the MRI image to be converted into a synthesized CT image based on the image synthesis model.

[0112] Exemplarily, in the embodiment, the MRI image to be converted is input into the prediction model to obtain a synthesized CT image. Since the embodiment considers the difficulty of serious information loss of the MRI synthesized CT image in the high-frequency part, a frequency domain consistency loss is proposed to effectively constrain the consistency of the real image and the synthesized image from the frequency domain level, so as to guide the model to pay more attention to the frequency part that is difficult to synthesize, thereby significantly improving the image details and the overall synthesis quality of the image. In addition, the embodiment also considers the difficult problem of lack of spatial domain information preservation ability in cross-domain image synthesis, and proposes a new generator network, that is, by introducing dynamic region convolution to effectively improve the extraction ability of the network to deep features, so as to preserve the consistency of the real image and the synthesized image from the spatial domain level, and introduce spectral normalization in the discriminator to improve the stability and reliability of the network training process. As can be seen, the MRI synthesized CT image method proposed in the embodiment has high adaptability, can effectively solve the problems of lack of spatial domain information preservation ability and high-frequency information loss in MRI synthesized CT image technology, and is easy to extend to other cross-domain image synthesis tasks.

[0113] In summary, the embodiment fully considers the difficult situation of lack of spatial domain information preservation ability and loss of high-frequency details in cross-domain medical image synthesis, preserves the feature consistency of cross-domain image synthesis from the spatial domain level by introducing dynamic region convolution, reduces the loss of details of the synthesized image from the frequency domain level by introducing the frequency domain consistency loss term, and improves the stability and reliability of the network training process by combining the spectral normalization strategy, thereby effectively improving the quality of the synthesized CT image.

[0114] The embodiment of the present application also provides an MRI synthesized CT image device, which comprises:

[0115] A data construction unit configured to obtain MRI image data and CT image data to construct a training data set;

[0116] a model function construction unit, configured to construct an improved cycle generative adversarial network model comprising a spatial information maintaining generator and a spectral normalization discriminator and a corresponding target function of the improved cycle generative adversarial network model, the target function comprising an adversarial loss term, a frequency domain consistency loss term and an identity mapping loss term;

[0117] a model training unit, configured to train the improved cycle generative adversarial network model based on the training data set to obtain an image synthesis model;

[0118] an image synthesis unit, configured to convert a to-be-converted MRI image into a synthesized CT image based on the image synthesis model.

[0119] Further, the target function is:

[0120] L(G, F, D MR , D CT ) = L GAN (G, D CT , X, Y) + L GAN (F, D MR , Y, X) + λ1L fc (G, F, X, Y) + λ2L identity (G, F)

[0121] In the formula, L(G, F, D MR , D CT ) represents the target function, G represents the first spatial information maintaining generator, F represents the second spatial information maintaining generator, D MR represents the first spectral normalization discriminator, D CT represents the second spectral normalization discriminator, L GAN (G, D CT , X, Y) represents an adversarial loss between G and D CT , X and Y represent an MRI image data set and a CT image data set respectively, L GAN (F, D MR , Y, X) represents an adversarial loss between F and D MR , L fc (G, F, X, Y) represents a frequency domain consistency loss between G and F, and L identity (G, F) represents an identity mapping loss between G and F, and λ1 and λ2 represent hyperparameters for controlling loss term weights.

[0122] Further, a calculation formula of L GAN (G, D CT , X, Y) in the adversarial loss term is:

[0123]

[0124] L GAN F, D MR , Y, X) is calculated as follows:

[0125]

[0126] wherein G(x) represents a pseudo-CT image synthesized by G, D CT (y) represents a discrimination result of a real CT image by D CT (y) represents a discrimination result of a real CT image by D CT (y) represents a discrimination result of a real CT image by D CT (y) represents a discrimination result of a real CT image by D MR (x) represents a pseudo-MRI image synthesized by F, D MR (x) represents a discrimination result of a real MRI image by D MR (x) represents a discrimination result of a real MRI image by D MR (x) represents a discrimination result of a real MRI image by D

[0127] Further, the frequency domain consistency loss L fc (G, F, X, Y) is calculated as follows:

[0128]

[0129] wherein L ff (x, F(G(x))) represents a focal frequency loss between the input MRI image x and the cyclically reconstructed MRI image F(G(x)), and L ff (y, G(F(y))) represents a focal frequency loss between the input CT image y and the cyclically reconstructed CT image G(F(y)).

[0130] Further, the identity mapping loss L identity (G, F) is calculated as follows:

[0131]

[0132] wherein G(x) represents a pseudo-CT image synthesized by G, and F(y) represents a pseudo-MRI image synthesized by F.

[0133] Further, the spatial domain information preserving generator comprises an encoder, a transformer and a decoder, the encoder is configured to extract MRI features, the encoder comprises a dynamic region convolution for adaptive allocation of filters, the transformer is configured to perform feature fusion on the MRI features extracted by the encoder, and the decoder is configured to convert the fused features output by the transformer into a synthesized image.

[0134] Further, the dynamic region convolution comprises a first convolution layer, a series allocation layer and a second convolution layer.

[0135] The first convolutional layer is configured to generate a plurality of feature maps carrying different mask guides;

[0136] The serial distribution layer is configured to assign a corresponding filter to each feature map;

[0137] The second convolutional layer is configured to perform convolutional operation on the feature map corresponding to the different filter.

[0138] It should be noted that, for the convenience and brevity of description, the specific working process of the above-described device and each unit can refer to the corresponding process in the MRI synthetic CT image method embodiments, which will not be described here.

[0139] The device provided in the above embodiments can be implemented in the form of a computer program, which can run on the MRI synthetic CT image device as shown in the Figure 6 The device provided in the above embodiments can be implemented in the form of a computer program, which can run on the MRI synthetic CT image device as shown in the

[0140] The MRI synthetic CT image device provided in the embodiments of the present application includes a memory, a processor and a network interface connected through a system bus, and the memory stores at least one instruction, which is loaded and executed by the processor to implement all or part of the steps of the MRI synthetic CT image method.

[0141] The network interface is configured to perform network communication, such as sending assigned tasks. Those skilled in the art can understand that, Figure 6 The network interface is configured to perform network communication, such as sending assigned tasks. Those skilled in the art can understand that,

[0142] The processor can be a CPU, and can also be other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays) or other programmable logic devices, discrete gate or transistor logic discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc. The processor is the control center of the computer device, and connects each part of the computer device through various interfaces and lines.

[0143] The memory can be used to store computer programs and / or modules, and the processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a video playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as video data, image data, etc.), and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, an SMC (Smart Media Card), an SD (Secure digital) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0144] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize all steps or part of steps of the MRI synthetic CT image method.

[0145] The embodiment of the present application realizes all or part of the above processes, and can also be realized by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium, and the computer program can realize the steps of the above methods when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, ROM (Read-Only memory), RAM (Random Access memory), electric carrier wave signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electric carrier wave signal and telecommunication signal.

[0146] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, a server, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage and so forth) embodying computer-readable program code.

[0147] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions of the flow diagrams and / or block diagrams as Figure 1 The flow diagram and / or block diagram in the flow diagrams and / or block diagrams Figure 1 The flow diagram and / or block diagram in the flow diagrams and / or block diagrams

[0148] It should be noted that, as used in this document, the terms "includes," "including," "has," "having," "contains," "containing," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or system. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or system that comprises the element.

[0149] The foregoing is merely illustrative of the principles of this application and various modifications can be made by those skilled in the art without departing from the spirit and scope of the application. The above description is embodied in the form of specific examples and embodiments. However, specific details of these examples and embodiments are not to be construed as limiting, but rather as providing an insight into the principles of the application.

Claims

1. A method of MRI synthetic CT imaging, characterized by, The method comprises the following steps: acquiring MRI image data and CT image data to construct a training data set; constructing an improved cycle generative adversarial network model comprising a spatial information preserving generator and a spectral normalization discriminator and a corresponding objective function thereof, the objective function comprising an adversarial loss term, a frequency domain consistency loss term and an identity mapping loss term; training the improved cycle generative adversarial network model based on the training data set to obtain an image synthesis model; translating a to-be-converted MRI image into a synthesized CT image based on the image synthesis model; wherein the objective function is: In the formula, denotes a target function, G denotes a first spatial domain information maintaining generator, F denotes a second spatial domain information maintaining generator, denotes a first spectral normalization discriminator, denotes a second spectral normalization discriminator, denotes an adversarial loss between G and , and denote an MRI image dataset and a CT image dataset respectively, denotes an adversarial loss between F and , denotes a frequency domain consistency loss between G and F, denotes an identity mapping loss between G and F, and denote a hyperparameter for controlling a loss term weight; the spatial information preserving generator comprises an encoder, a converter and a decoder, the encoder is configured to extract MRI features, the encoder comprises a dynamic region convolution for adaptive allocation of filters, the converter is configured to perform feature fusion on the MRI features extracted by the encoder, and the decoder is configured to convert the fused features output by the converter into a synthesized image.

2. The method of MRI synthetic CT imaging of claim 1, wherein, the calculation formula of the adversarial loss term is: the calculation formula of the adversarial loss term is: The calculation formula of the adversarial loss term is: The calculation formula of the adversarial loss term is: wherein a pseudo CT image synthesized by G, a pseudo CT image synthesized by G, a discrimination result for a real CT image, a pseudo CT image synthesized by G, a discrimination result for a pseudo CT image, a pseudo MRI image synthesized by F, a pseudo MRI image synthesized by F, a discrimination result for a real MRI image, a pseudo MRI image synthesized by F, a discrimination result for a pseudo MRI image.

3. The method of MRI synthetic CT imaging of claim 1, wherein, The The calculation formula is: wherein denotes the focus frequency loss between the input MRI image and its cyclically reconstructed MRI image denotes the focus frequency loss between the input CT image and its cyclically reconstructed CT image .​ 4. The method of MRI synthetic CT imaging of claim 1, wherein, The identity mapping loss The calculation formula is: wherein represents a pseudo CT image of G synthesis, represents a pseudo MRI image of F synthesis.

5. The method of MRI synthetic CT imaging of claim 1, wherein, The dynamic region convolution comprises a first convolution layer, a serial allocation layer and a second convolution layer; the first convolution layer is configured to generate multiple feature maps carrying different mask guides; the serial allocation layer is configured to allocate a corresponding filter to each feature map; the second convolution layer is configured to perform convolution operation on the feature map corresponding to the different filters.

6. An apparatus for MRI synthetic CT imaging, characterized by: The method comprises: a data construction unit configured to acquire MRI image data and CT image data to construct a training data set; a model function construction unit configured to construct an improved cycle generative adversarial network model comprising a spatial information preserving generator and a spectral normalization discriminator and a corresponding objective function thereof, the objective function comprising an adversarial loss term, a frequency domain consistency loss term and an identity mapping loss term; a model training unit configured to train the improved cycle generative adversarial network model based on the training data set to obtain an image synthesis model; an image synthesis unit configured to translate a to-be-converted MRI image into a synthesized CT image based on the image synthesis model wherein the objective function is: In the formula, denotes a target function, G denotes a first spatial domain information maintaining generator, F denotes a second spatial domain information maintaining generator, denotes a first spectral normalization discriminator, denotes a second spectral normalization discriminator, denotes an adversarial loss between G and , and denote an MRI image dataset and a CT image dataset respectively, denotes an adversarial loss between F and , denotes a frequency domain consistency loss between G and F, denotes an identity mapping loss between G and F, and denote a hyperparameter for controlling a loss term weight; the spatial information preserving generator comprises an encoder, a converter and a decoder, the encoder is configured to extract MRI features, the encoder comprises a dynamic region convolution for adaptive allocation of filters, the converter is configured to perform feature fusion on the MRI features extracted by the encoder, and the decoder is configured to convert the fused features output by the converter into a synthesized image.

7. An apparatus for MRI synthetic CT imaging, characterized by: The method comprises: a memory and a processor, the memory stores at least one instruction, the at least one instruction is loaded and executed by the processor to implement the method for synthesizing CT images from MRI images according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, when the computer program is executed by the processor, to implement the method for synthesizing CT images from MRI images according to any one of claims 1 to 5.

Images