An automatic method for converting CT volume data into MRI volume data

Abstract

The present application relates to a kind of CT body data conversion MRI body data automatic generation method, construct and generate the mapping of body CT data to body MRI data by adversarial network;With the training data input of processed paired three-dimensional CT and MRI data to the generation of adversarial network for training, after cutting training data, the CT body data in it is input generator, the output analog MRI body data of generator is compared with the cut MRI body data respectively with space discriminator and time discriminator, and the prediction matrix output by space discriminator and time discriminator is used for network training respectively;With the CT body data of input completed generation of adversarial network is converted into MRI body data.The present application is relative to the conversion between two-dimensional medical image, using imperfect pairing body data to complete more challenging medical body data conversion task between modalities;It can reduce the cost, time, work and health risk of obtaining medical intermodal body data.

Description

Technical Field

[0001] This invention relates to the technical field of general image data processing or generation, and more particularly to an automatic generation method for converting CT volume data into MRI volume data based on imperfectly paired data in the field of computer deep learning technology. Background Technology

[0002] Different modalities of medical imaging techniques, such as computed tomography (CT), cone-beam computed tomography (CBCT), magnetic resonance imaging (MRI), and positron emission tomography (PET), are crucial in medical practice and research and are widely used in various clinical settings and studies. These techniques each have their own imaging characteristics in capturing patient organs. For example, CT offers short imaging times and high spatial resolution, excelling at depicting structures like bones and lungs. However, it exposes patients to ionizing radiation during acquisition, potentially posing health risks, and its soft tissue imaging results lack contrast. Compared to CT, MRI is safer, as it does not involve high-energy radiation and provides high contrast in soft tissue imaging, but it is more expensive and has a longer imaging time.

[0003] In many clinical settings, such as standard MRI-guided therapy, stroke diagnosis, and liver tumor diagnosis, it is necessary to obtain imaging results from multiple medical modalities. Imaging information from different modalities can complement each other, providing better decision support for screening, diagnosis, or treatment. To achieve this, patients often have to undergo multiple medical imaging scans, which is not only expensive (each scan typically requires separate payment), time-consuming (requiring appointments weeks in advance), and labor-intensive (requiring a large number of specialized physicians), but also may pose health risks and delay treatment for certain diseases. Therefore, there is an urgent need for a technology that can automatically synthesize / convert medical imaging results from different modalities, playing a crucial role in these scenarios to reduce the time-consuming, labor-intensive, expensive, and potentially health-risk-prone drawbacks of traditional medical imaging.

[0004] Currently, image-to-image conversion between medical modalities has become a hot topic. However, most current research focuses on cross-modal conversion from two-dimensional medical images to images, with little attention paid to cross-modal conversion from volumetric data to volumetric data (i.e., three-dimensional medical images), which is crucial for 3D anatomical reconstruction and visualization analysis. Compared to two-dimensional medical image-to-image conversion, volumetric conversion is more challenging due to:

[0005] (1) Obtaining perfectly paired cross-modal volumes that can be used for model training / inference is very difficult. In reality, cross-modal volumes are collected from different modalities at different times, and therefore usually contain a certain number of imperfectly paired slices;

[0006] (2) Preprocessing imperfectly paired cross-modal bodies to make them perfect is not simple and may require a lot of manpower and time.

[0007] In existing technologies, image-to-image conversion between medical modalities is generally accomplished using trained neural networks. Training / inference data significantly impacts model results; good training / inference data typically yields good results, while poor data usually leads to poor results. When using imperfectly paired cross-modal volumes as training / inference data, the deep learning model used for volume-to-volume cross-modal conversion must be robust enough while maintaining a certain level of accuracy. Furthermore, models for volume-to-volume cross-modal conversion need to consider "slice consistency," but image-to-image cross-modal conversion only considers a single slice at a time, so "slice consistency" is not applicable. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides an automatic generation method for converting CT volume data into MRI volume data, which can automatically and effectively convert any three-dimensional CT image into a three-dimensional MRI image, including imperfectly matched data, for use by medical experts for screening, diagnosis, or image-guided treatment.

[0009] The technical solution adopted in this invention is an automatic generation method for converting CT volume data into MRI volume data. The method constructs a generative adversarial network to realize the mapping from volume CT data to volume MRI data.

[0010] The processed paired 3D CT and MRI data are used as training data to input the generative adversarial network for training. After the training data is cropped, the CT volume data is input into the generator. The simulated MRI volume data output by the generator is compared with the cropped MRI volume data by a spatial discriminator and a temporal discriminator, respectively. The network is trained based on the prediction matrices output by the spatial discriminator and the temporal discriminator, respectively.

[0011] The trained generative adversarial network converts the input CT volume data into MRI volume data.

[0012] Preferably, the data input to the generative adversarial network is raw paired CT volume data and MRI volume data that have been centrally pruned and globally normalized.

[0013] Preferably, the processed CT volume data is input into the trained generative adversarial network, and the MRI volume data based on the generator output is tested; the output results of the generative adversarial network are visualized.

[0014] In this invention, a set of bodies is given Among them, V i CT Represents the i-th independent volume CT;

[0015] Constructing a Generative Adversarial Network (GAN) can achieve mapping V MRI =f(V CT ), Indicates with V CT The corresponding set of body MRI, V u MRI Indicates with V i CT For the corresponding i-th independent volumetric MRI, the function f represents the mapping function.

[0016] In this invention, given standardized paired body CT and body MRI data are divided into a training set (training data) and a test set (test data), typically with a division ratio of 70% and 30%.

[0017] The training workflow is as follows:

[0018] (1) Extract a pair of body CT and body MRI scans from the training set to obtain a pair of real body CT scans. and

[0019] (2) For this pair and Perform random volume cropping (crop a pair of volume CT and volume MRI images at random locations) to obtain a pair of truly cropped volumes. and

[0020] (3) The data is input into the model's generator, which generates fake cropped volumes.

[0021] (4) and The values ​​are input into the spatial discriminator and temporal discriminator of the model, respectively, to generate two prediction matrices, each of which is in the range [0,1].

[0022] The reasoning workflow is as follows:

[0023] (1) Take a set of body CT data from the test set to obtain the real body

[0024] (2) The input is fed into the generator to synthesize a fake body.

[0025] (3) Visualization To obtain the final rendering result.

[0026] Preferably, the generator includes a first UNet and a second UNet connected in sequence, wherein the encoder and decoder structures of the first UNet and the second UNet are identical; the input of the second UNet is the sum of the input and output of the first UNet.

[0027] Preferably, the encoders of the first UNet and the second UNet include four sequentially connected encoding transform blocks. Each encoding transform block is followed by a first trilinear downsampling layer, and the output of the last encoding transform block is connected to a second trilinear downsampling layer to obtain the encoding result. Each encoding transform block is connected to the corresponding position of the decoder through a residual path and a corresponding transform block before the first trilinear downsampling layer. The output of the last encoding transform block is connected to the corresponding position of the decoder through an encoding transform block before the second trilinear downsampling layer. The encoding result is passed through a self-attention module to obtain a feature map.

[0028] Preferably, in the decoders of the first UNet and the second UNet, the feature maps of the corresponding encoders are input into the decoder, pass through a first trilinear upsampling layer and a convolutional layer, and correspond to the output of the last encoding transform block. Four decoding transform blocks are set sequentially, each of which is preceded by a second trilinear upsampling layer. The last decoding transform block is output after passing through a convolutional layer. The convolutional layer at the output of the first UNet is also equipped with an activation function. Each decoding transform block of the first UNet is connected to the corresponding position of the encoder of the second UNet through a residual path and the corresponding transform block before the corresponding second trilinear upsampling layer.

[0029] Preferably, in the first UNet and the second UNet, the input terminals of the first decoding transform block and the second decoding transform block are respectively connected to self-attention modules through a third trilinear downsampling layer, and the output of the self-attention module is respectively connected to the output terminals of the first decoding transform block and the second decoding transform block through the corresponding third trilinear upsampling layer and convolutional layer.

[0030] Preferably, the loss function L of the generator G satisfy,

[0031]

[0032] in, To combat the losses, For identity loss, For space loss, For timing loss and For voxel distance loss, λ identity , λ spatial , λ tem.0ral and λ v0xdis To assign weights to each loss term, λ identity , λ spatial , λ tem.0ra and λ v0xdis All are greater than 0.

[0033]

[0034] For the actual cropped body CT, D S and D T These are the spatial discriminator and the temporal discriminator, respectively. 1 represents an all-one matrix, ||·|| BCEL For the BCEWithLogitsLoss() function, D is encouraged. S and D T All fake cropped MRI images synthesized by the generator are considered real; G() is the output of the generator;

[0035]

[0036] For true cropped MRI, ||·|| SL1 The SmoothL1Loss() function measures... How much change occurs when passing through the generator, encouraging identity mapping;

[0037]

[0038] f n For the nth feature map in the spatial discriminator, ||·|| MSE The mean squared error (MSE) loss function is used to calculate the loss function for each feature map. And fake cropped body MRI data By examining the differences in feature maps over time, we can encourage the synthesis of pseudo-volumetric MRI from a spatial perspective to approximate real-volumetric MRI as closely as possible.

[0039]

[0040] For the second to the last slice of n feature maps in the temporal discriminator, For the first to the second to last slices of n feature maps in the temporal discriminator; Calculate the slice difference between real and fake feature maps in any two consecutive slices, calculate the mean square error between real and fake 3D slice differences, and encourage the slice consistency of synthesized fake volumetric MRI to be as close as possible to the slice consistency of real data from a time perspective.

[0041]

[0042] ||·|| L1 For L1 loss, Measurement and The differences between them encourage the generator to synthesize results that are closer to real data.

[0043] Preferably, the spatial discriminator and the temporal discriminator include sequentially arranged convolutional layers, three shrinking modules, and convolutional layers; a max pooling layer is provided at the output end of each shrinking module; the shrinking module includes two connected convolutional layers and one compressed excitation block.

[0044] Preferably, the loss functions of both the spatial discriminator and the temporal discriminator include a loss for maximizing the falsity of the generator output MRI volume data and a loss for maximizing the authenticity of the cropped MRI volume data.

[0045] Loss function of spatial discriminator

[0046]

[0047] The first item encourages D S synthesized from the generator The first term is identified as fake, with 0 representing an all-zero matrix, while the second term encourages D. S Will The result is identified as real, and 1 represents a matrix of all 1s; the average of these two terms is used as the final loss.

[0048] Loss function of timing discriminator

[0049]

[0050] This invention relates to an automatic generation method for converting CT volumetric data into MRI volumetric data. A generative adversarial network (GAN) is constructed to map volumetric CT data to volumetric MRI data. Processed paired 3D CT and MRI data are input into the GAN for training. After cropping the training data, the CT volumetric data is input into the generator. The generator's output simulated MRI volumetric data is compared with the cropped MRI volumetric data using a spatial discriminator and a temporal discriminator, respectively. The network is trained based on the prediction matrices output by the spatial and temporal discriminators. The trained GAN then converts the input CT volumetric data into MRI volumetric data.

[0051] The beneficial effects of this invention are as follows:

[0052] (1) Compared to the conversion between two-dimensional medical images, the conversion between medical volume data is more challenging using imperfectly paired volume data;

[0053] (2) It can reduce the cost, time, work and health risks of acquiring medical intermodal data. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the framework of the generative adversarial network of the present invention;

[0055] Figure 2 This is a schematic diagram of the generator of the present invention;

[0056] Figure 3 This is a schematic diagram of the encoding transformation block or decoding transformation block in the generator of the present invention;

[0057] Figure 4 This is a schematic diagram of the self-attention module in the generator of the present invention;

[0058] Figure 5 This is a schematic diagram of the timing discriminator and spatial discriminator framework of the present invention;

[0059] Figure 6 This is a schematic diagram comparing the brain visualization results of this invention with the results of other methods;

[0060] Figure 7 This is a schematic diagram comparing the visualization results of the pelvis in this invention with the results of other methods;

[0061] Figure 8 This is a schematic diagram comparing the structural similarity and multi-scale structural similarity values ​​of volume magnetic resonance imaging data synthesized by all techniques according to the present invention.

[0062] Figure 2 , Figure 3 and Figure 5In the diagram, 1 is a 3x3x3 convolutional layer (1x1x1 padding), 2 is a spectral normalization layer, 3 is a batch normalization layer (Batch Normalm layer), 4 is a ReLU layer, 5 is a transform block (encoding transform block or decoding transform block), 6 is a trilinear upsampling layer or trilinear downsampling layer, 7 is a PReLU (activation function), 8 is a 1x1x1 convolutional layer (0x0x0 padding), 9 is a self-attention module, a is a ResPath (residual path), b is a compressed activation block, and c is a max pooling layer. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0064] This invention also relates to an automatic generation method for converting CT volume data into MRI volume data, which is referred to as W-VCT2VMRIGAN in the following embodiments, where W is the structural simulation of the generator, VCT and VMRI are volume CT and volume MRI respectively, "2" means conversion (to), and GAN refers to the overall use of the GAN network.

[0065] The method constructs a generative adversarial network to achieve the mapping of body CT data to body MRI data;

[0066] The processed paired 3D CT and MRI data are used as training data to input the generative adversarial network for training. After the training data is cropped, the CT volume data is input into the generator. The simulated MRI volume data output by the generator is compared with the cropped MRI volume data by a spatial discriminator and a temporal discriminator, respectively. The network is trained based on the prediction matrices output by the spatial discriminator and the temporal discriminator, respectively.

[0067] The trained generative adversarial network converts the input CT volume data into MRI volume data.

[0068] The data input to the generative adversarial network consists of raw paired CT volume data and MRI volume data that have undergone central cropping and global normalization.

[0069] In this invention, such as Figure 1 As shown, the problem is defined as follows:

[0070] Given a set of body CT scans, Among them, V i CT Represents the i-th independent volume CT;

[0071] Constructing a Generative Adversarial Network (GAN) can achieve mapping V MRI =f(VCT ), Indicates with V CT The corresponding set of body MRI, V i MRI Indicates with V i CT For the corresponding i-th independent volumetric MRI, the function f represents the mapping function.

[0072] In this invention, given the original paired body CT and body MRI, they are first preprocessed to become normalized paired body CT and body MRI, involving two preprocessing algorithms:

[0073] (1) Center clipping: Based on the center of the original paired body CT and body MRI, all body CT and body MRI with different sizes are clipped to the same size to facilitate model training and inference.

[0074] (2) Global normalization: Calculate the global minimum and maximum values ​​of all paired body CT and body MRI, and use them to linearly normalize each pair of paired body CT and body MRI to make them within the range of [0,1].

[0075] Given a set of standardized paired body CT and body MRI scans, the data is divided into a training set (training data) and a test set (test data), typically with a ratio of 70% and 30%; that is, the first 70% of all paired body CT and body MRI scans are used as the training set, while the remaining 30% are used as the test set.

[0076] In this invention, the training workflow is as follows:

[0077] First, a pair of body CT and body MRI scans were extracted from the training set to obtain a pair of real body CT scans. and

[0078] Secondly, regarding this pair and Perform random volume cropping (crop a pair of volume CT and volume MRI images at random locations) to obtain a pair of truly cropped volumes. and The advantage of using randomized pruning is that it can increase the number of training samples available for model training, thereby improving the model's generalization ability while ensuring good training results, and can effectively reduce GPU memory usage.

[0079] again, The data is input into the model's generator, which generates fake cropped volumes.

[0080] at last, and The values ​​are fed into the spatial discriminator and temporal discriminator of the model, respectively, to generate two prediction matrices. Each value in the prediction matrix is ​​in the range of [0,1], representing the probability that the corresponding region is real in the input.

[0081] In this invention, the reasoning workflow is as follows:

[0082] First, a set of volume CT data was extracted from the test set to obtain the real volume data.

[0083] Then, The input is fed into the generator to synthesize a fake body.

[0084] Finally, visualization To obtain the final rendering results, which is very useful for screening, diagnosis and treatment.

[0085] As Figure 1 As shown, the W-VCT2VMRIGAN of the present invention includes a generator, a spatial discriminator, and a temporal discriminator. Since volume data pairs often contain some imperfectly paired slices, it is very difficult to obtain perfectly paired volume CT and volume MRI pairs. The generator's role is to synthesize volume MRI as realistically as possible, making it robust to imperfectly paired slices. The spatial discriminator's role is to encourage the generator to synthesize realistic volume MRI with good "spatial" consistency, while the temporal discriminator's role is to encourage the generator to synthesize realistic volume MRI with good "temporal" consistency.

[0086] like Figure 2 The generator shown includes a first UNet and a second UNet connected in sequence, with the encoder and decoder structures of the first UNet and the second UNet being identical; the input of the second UNet is the sum of the input and output of the first UNet.

[0087] The generator consists of two UNet-like networks, labeled UNet1 and UNet2 in the figure. These two networks are connected together by using the output of UNet1 as the input of UNet2. In this way, a deeper CNN can be created to obtain better transformation results.

[0088] Both UNet1 and UNet2 follow the same encoder-decoder structure. The encoder's task is to encode the input into smaller intermediate feature maps containing high-level and important features, while the decoder's task is to decode the smaller feature maps and expand them into the output.

[0089] The encoders of the first UNet and the second UNet include four sequentially connected encoding transform blocks. Each encoding transform block is followed by a first trilinear downsampling layer. The output of the last encoding transform block is connected to a second trilinear downsampling layer to obtain the encoding result. Each encoding transform block is connected to the corresponding position of the decoder through a residual path and a corresponding transform block before the first trilinear downsampling layer. The output of the last encoding transform block is connected to the corresponding position of the decoder through an encoding transform block before the second trilinear downsampling layer. The encoding result is passed through a self-attention module to obtain a feature map.

[0090] In the decoders of the first UNet and the second UNet, the feature maps of the corresponding encoders are input into the decoder, pass through the first trilinear upsampling layer and the convolutional layer, and correspond to the output of the last encoding transform block. Through four sequentially set decoding transform blocks, each decoding transform block is preceded by a second trilinear upsampling layer. After the last decoding transform block, it passes through a convolutional layer and is output. The convolutional layer at the output end of the first UNet also has an activation function. Each decoding transform block of the first UNet is connected to the corresponding position of the encoder of the second UNet through a residual path and the corresponding transform block before the corresponding second trilinear upsampling layer.

[0091] In the first UNet and the second UNet, the inputs of the first and second decoding transform blocks are respectively connected to self-attention modules through a third trilinear downsampling layer. The outputs of the self-attention modules are respectively connected to the outputs of the first and second decoding transform blocks through the corresponding third trilinear upsampling layer and convolutional layer.

[0092] Specifically, in this invention, the encoders of UNet1 and UNet2 include four encoding operations, each performed by a coding transform block (TB), followed by trilinear downsampling. The decoders of UNet1 and UNet2 include four decoding operations, each performed by trilinear upsampling. The corresponding positions between the encoders and decoders of UNet1 and UNet2 are then mapped to the corresponding transform blocks using residual paths (ResPath).

[0093] To enable the model of this invention to learn distant voxels rather than just nearby voxels and thus achieve better results, three self-attention related operations were added to UNet1 and UNet2 respectively. Each operation includes a trilinear downsampling to shrink the feature map, followed by a self-attention module (SAB), then a trilinear upsampling to expand the feature map, and finally a 1×1×1 convolution. Since directly applying SAB to the feature map would consume a lot of GPU memory and may lead to insufficient memory, the feature map is shrunk first and then the self-attention mechanism is performed.

[0094] Unlike traditional UNet which uses skip connections to connect the encoder and decoder, this invention upgrades the process by using ResPath. In addition, ResPath is also added between the decoder of UNet1 and the encoder of UNet2.

[0095] like Figure 2 As shown, 128×128×128×1 CT volume data is input to the first UNet of the generator. After passing through four encoding transform blocks and the corresponding first trilinear downsampling layer, an 8×8×8×256 encoded result is obtained. This is then processed by a self-attention module to obtain the encoded feature map. During this process, the result before each first trilinear downsampling layer is connected to the corresponding position in the decoder via a residual path (ResPath) and the corresponding transform block. The result before the second trilinear downsampling layer is connected to the corresponding position in the decoder via the corresponding transform block. Subsequently, the feature map begins decoding. First, it passes through the first trilinear upsampling layer and a 1×1×1 convolutional layer to obtain an 8×8×8×256 feature map. Then, it passes through four decoding transform blocks and the corresponding second trilinear upsampling layer. During this process... The first 16×16×16×512 result of the first decoding transform block is connected to the self-attention module through the third trilinear downsampling layer. The output of the self-attention module is combined with the first decoding transform block after passing through the third trilinear upsampling layer and convolutional layer to obtain a 16×16×16×128 result. The second decoding transform block is similar. After four decoding transform blocks, a 128×128×128×16 result is obtained. After passing through the convolutional layer and the corresponding activation function (PReLU in this case), a 128×128×128×1 result data is obtained. This result data is added to the 128×128×128×1 CT volume data at the input end and then input into the second UNet. The operation of the first UNet is basically the same as that of the first UNet, except that no activation function processing is required when outputting.

[0096] like Figure 3 As shown, the structure of the encoding transform block or the decoding transform block is consistent. In each encoding transform block or decoding transform block, the input data passes through two paths. One path is a combination of two consecutive 3×3×3 convolutional layers, a spectral normalization layer, a BatchNorm layer, and a ReLU function, and a combination of one 3×3×3 convolutional layer and a spectral normalization layer. The other path is a combination of one 3×3×3 convolutional layer and a spectral normalization layer. The results from the two paths are added together and then passed through a ReLU function to obtain the output result.

[0097] like Figure 4As shown, the input data from the attention module passes through three paths: the first is the Query convolutional layer, the second is the Key convolutional layer, and the third is the Value convolutional layer. After calculating the dot product of the results from the first two paths, they pass through the softmax layer and the attention map, and then calculate the dot product with the output of the third path. After passing through the attention weights twice, the result is added to the input data to obtain the result.

[0098] like Figure 5 As shown, the spatial discriminator and the temporal discriminator include sequentially arranged convolutional layers, three shrinking modules, and convolutional layers; a max pooling layer is provided at the output end of each shrinking module; the shrinking module includes two connected convolutional layers and one compressed excitation block.

[0099] In this invention, the spatial discriminator and the temporal discriminator share the same architecture. Compared with the generator's architecture, the spatial discriminator and the temporal discriminator have much simpler architectures. They only attempt to distinguish between real and fake cropped MRI images, thus avoiding their learning ability from exceeding that of the generator.

[0100] In this invention, the input is a real cropped body MRI. And fake cropped body MRI The output is a prediction matrix filled with [0,1] values, used to predict the true probability of the input. The architecture of the spatial discriminator and temporal discriminator has only one encoder structure, including three shrinkage operations, each performed by two 3×3×3 convolutions, followed by a squeeze excitation block (Squeeze & Excitation) and a max pooling layer (MaxPool). At the beginning and end of the architecture, 1×1×1 convolutions are used to transform the feature maps. Introducing the Squeeze & Excitation module into the architecture is very important because this module can increase the sensitivity to relevant features and suppress unnecessary features.

[0101] by Figure 5 For example, the input data is 128×128×128×1 MRI volume data. After passing through a 1×1×1 convolutional layer, the result is 128×128×128×32. After passing through two sequentially connected 3×3×3 convolutional layers, a Batch Norm layer, a ReLU layer, and a squeeze excitation block (Squeeze & Excitation), the result is then passed through a max pooling layer, resulting in a 64×64×64×64 result data. After repeating this process twice, a 16×16×16×256 result data is obtained. Finally, after passing through a 1×1×1 convolutional layer, a 16×16×16×1 prediction matrix is ​​obtained.

[0102] The loss function L of the generator G satisfy,

[0103]

[0104] in, To combat the losses, For identity loss, For space loss, For timing loss and For voxel distance loss, λ identity , λ spatial , λ templral and λ vlxdis To assign weights to each loss term, λ identity , λ s.atial , λ templra and λ vlxdis All are greater than 0.

[0105] The loss functions of both the spatial discriminator and the temporal discriminator include a loss for maximizing the falsity of the generator's output MRI volume data and a loss for maximizing the authenticity of the cropped MRI volume data.

[0106] In this invention, the generator, spatial discriminator, and temporal discriminator all have loss functions.

[0107] In this invention, the total loss function L of generator G is... G for,

[0108]

[0109] L G middle, To combat losses, to meet

[0110]

[0111] in, This represents the actual cropped volume CT scan, D S and D T These represent the spatial discriminator and the temporal discriminator, respectively. 1 represents a matrix of all 1s. ||·|| BCEL This refers to the BCEWithLogitsLoss() function, which encourages D S and D T The fake clipped MRI synthesized by the generator is considered real; G() is the output of the generator.

[0112] L G middle, For identity loss, satisfy

[0113]

[0114] in, Represents a true clipped MRI image, ||·|| SL1 This refers to the SmoothL1Loss() function, which measures... How much change occurs when passing through the generator, thus encouraging identity mapping.

[0115] L G middle, For space loss, satisfy

[0116]

[0117] Among them, f n Represents the nth feature map in the spatial discriminator, ||·|| MSE This represents the mean squared error (MSE) loss function, which, for each feature map, is calculated based on the actual cropped volumetric MRI data. And fake cropped body MRI data By examining the differences in feature maps over time, we can encourage the synthesis of spurious volumetric MRI from a spatial perspective to approximate real volumetric MRI as closely as possible.

[0118]

[0119] L G middle, For time loss, satisfy

[0120]

[0121] The temporal discriminator contains multiple feature maps, among which, This represents the second to the last slice of n feature maps in the temporal discriminator. This represents the first to the second-to-last slice of n feature maps in the temporal discriminator;

[0122] The true feature map (in the time discriminator) was calculated. (any feature map passed through) and fake feature maps (by...) Calculate the mean square error between the true 3D slice difference and the false 3D slice difference by taking the same feature map between any two consecutive slices; The consistency of synthesized fake MRI slices should be encouraged to be as close as possible to the consistency of real data slices from a temporal perspective.

[0123] L G middle, For voxel distance loss, satisfy

[0124]

[0125] Among them, ||·|| L1 Indicates L1 loss, Measurement and The differences between them encourage the generator to synthesize results that are closer to real data.

[0126] Corresponding to L G In each of the following items, λ adv , λ identity , λ spatial , λ temporal and λ voxdis This represents the corresponding weight of each of the above items, and λ adv , λ identity , λ spatial , λ temporal and λ voxdis All are greater than 0, and their values ​​can be adjusted by those skilled in the art based on the actual situation of the network.

[0127] Loss function L of spatial discriminator DS satisfy

[0128]

[0129] The first item encourages D S synthesized from the generator The first term is identified as fake; 0 represents an all-zero matrix, while the second term encourages D. S Will The result is identified as true, and 1 represents a matrix of all 1s; the average of these two terms is taken and used as the final loss.

[0130] Loss function of timing discriminator and Similarly, satisfy

[0131]

[0132] Because GANs often suffer from unstable training, which can lead to poor results or pattern collapse, this invention employs two techniques to improve the stability of GAN training:

[0133] One is the Two-Time-Scale Update Rule (TTUR), which uses separate learning rates to update the generator and discriminator, so that the discriminator can be updated less often with each generator update, which can produce better results in the same clock time.

[0134] The second is spectral normalization (or spectral norm), which is a weight normalization method that can stabilize the training of GANs. Unlike other normalization techniques that may require additional hyperparameters, it only requires setting a spectral norm after the convolution operation.

[0135] In this invention, processed CT volume data is input into a trained generative adversarial network (GAN), and MRI volume data based on the generator's output is tested; the output results of the GAN are visualized.

[0136] For each dataset, this invention applies the same visualization settings, including color mapping, transfer function, rendering angle, etc., to both the volumetric MRI generated by all comparison techniques and the real data. Furthermore, this invention visualizes the differences between each comparison technique and the real data, which helps us to better conduct qualitative analysis on these comparison techniques.

[0137] Specifically, this invention employs a GPU-accelerated 3D ray casting algorithm for visualization. This algorithm reconstructs 3D data and generates clear visualization results, reflecting the internal and external 3D structure of the 3D data. For the transfer function, this invention uses jet color mapping or its inverse color mapping for shading, determining the opacity of each voxel in the 3D data to satisfy...

[0138]

[0139] Where v represents any voxel in the 3D data, normInten(v) represents the normalized intensity value of the voxel, and foreground refers to the foreground of the 3D data, which is defined in [f min ,f max Within the range, f min and f max These represent the minimum and maximum thresholds for the foreground, respectively. β is a parameter used to control the degree of opacity, which is not 0 and can be adjusted based on the needs of those skilled in the art, such as taking 1.

[0140] Based on the technology of this invention, qualitative and quantitative analyses are performed on this invention.

[0141] The qualitative analysis of the experiment in this invention is as follows:

[0142] like Figure 6 As shown, three-dimensional visualizations of brain data from four randomly selected real / synthetic MRI scans (volumes 71, 78, 88, and 100, respectively, from (a) real data, (b) the method of the present invention, (c) Pix2Pix, (d) CycleGAN, and (e) MedSynthesisV1) are presented; a smaller grayscale image on the right next to each visualization shows a single slice (slice 93) extracted from the 3D being visualized; each column represents the same MRI 3D / slice generated from different techniques, and each row represents different MRI 3D / slices from the same technique.

[0143] By observing visualizations and slices, it is clear that the method of this invention can synthesize the most realistic somatic MRI, with fine-grained features such as gray and white matter. The second best method seems to be Pix2Pix, whose synthesized somatic MRI also has some feature details, such as still being able to distinguish gray and white matter. The third best method seems to be MedSynthesisV1, although it is almost impossible to distinguish gray and white matter based on its synthesized somatic MRI, but some of the results can still retain the features of the real data to a certain extent. For example, the slice next to volume 88 retains the "x" structure located in the center of the brain, which is very similar to the real data. The worst method is CycleGAN, whose synthesized somatic MRI shows no details (e.g., it is impossible to distinguish the internal structure of the brain), but has clear checkerboard artifacts.

[0144] like Figure 7 As shown, three-dimensional visualizations of pelvic data from four real / synthetic MRIs randomly selected from (a) real data, (b) the method of the present invention, (c) Pix2Pix, (d) CycleGAN, and (e) MedSynthesisV1 are presented (Volumes 75, 80, 83, and 91, respectively); a smaller grayscale image on the right next to each visualization shows a single slice (slice 48) extracted from the 3D being visualized; each column represents the same MRI 3D / slice generated from different techniques, and each row represents different MRI 3D / slices generated from the same technique.

[0145] By observing visualizations and slices, it is clear that CycleGAN cannot synthesize any volume MRI, making it the worst technique; the second worst technique appears to be Pix2Pix, as its synthesized volume MRI data is riddled with horizontal artifacts; the method of this invention and MedSynthesisV1 both appear to be able to synthesize volume MRIs similar to real data, but upon closer examination of these two results, it is not difficult to find that the volume MRI synthesized from MedSynthesisV1 tends to have fewer features than the method of this invention, for example, the bottom features indicated by the arrows; furthermore, the colors displayed in MedSynthesisV1 are less close to those of real data.

[0146] Therefore, in summary, the method of the present invention is the best technique.

[0147] The quantitative analysis of this invention is shown in Table 3, where bold numbers indicate the best results;

[0148] Table 3: MOS of the comparative techniques relative to the brain and pelvic datasets

[0149]

[0150] Figure 8(a) shows the SSIM and MS-SSIM3 values ​​of all comparison techniques relative to synthetic MRI of real brain datasets; for the SSIM and MS-SSIM3 indices, higher values ​​indicate better similarity and lower values ​​indicate worse similarity, and it can be concluded that the method of the present invention has the highest SSIM value for most synthetic MRIs, indicating that it is the best method.

[0151] The SSIM values ​​of MedSynthesisV1 are very close to those of the method of this invention, and some are even higher than those of the method of this invention, indicating that it is the second best method; the third best method is Pix2Pix, whose SSIM value is much higher than that of the worst CycleGAN method; for the MS-SSIM3 index, it can be seen that the method of this invention and MedSynthesisV1 have very similar and intertwined values ​​(sometimes the method of this invention has a higher value, and sometimes MedSynthesisV1 has a higher value), indicating that both methods are optimal. Conversely, CycleGAN has the lowest MS-SSIM3 value and is therefore the worst method. Finally, Pix2Pix has a moderate MS-SSIM3 value.

[0152] Figure 8 (b) Shows the SSIM and MS-SSIM3 values ​​of all contrast techniques relative to the real data of the pelvic dataset for synthetic MRI; the method of the present invention has the highest SSIM and MS-SSIM3 values ​​for all synthetic MRIs, and is therefore the best technique; conversely, CycleGAN has the lowest SSIM and MS-SSIM3 values ​​for all synthetic MRIs, and is therefore the worst technique. MedSynthesisV1 and Pix2Pix have interleaved SSIM and MS-SSIM3 values, falling between the method of the present invention and CycleGAN.

[0153] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0158] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An automatic method for converting CT volume data into MRI volume data, characterized in that: The method constructs a generative adversarial network to achieve the mapping of body CT data to body MRI data; The processed paired 3D CT and MRI data are used as training data to input the generative adversarial network for training. After the training data is cropped, the CT volume data is input into the generator. The simulated MRI volume data output by the generator is compared with the cropped MRI volume data by a spatial discriminator and a temporal discriminator, respectively. The network is trained based on the prediction matrices output by the spatial discriminator and the temporal discriminator, respectively. The generator includes a first UNet and a second UNet connected in sequence, with the encoder and decoder structures of the first UNet and the second UNet being identical. The input of the second UNet is the sum of the input and output of the first UNet; The encoders of the first and second UNets each include four sequentially connected encoding transform blocks, each followed by a first trilinear downsampling layer. The first three encoding transform blocks are output to the next encoding transform block via the subsequent first trilinear downsampling layer, and the fourth encoding transform block is output to a second trilinear downsampling layer via the subsequent first trilinear downsampling layer. The encoded result is obtained through the second trilinear downsampling layer. Each encoding transform block is connected to the corresponding position in the decoder via a residual path and a corresponding transform block before the first trilinear downsampling layer. The fourth encoding transform block is connected to the corresponding position in the decoder via encoding transform blocks after the first trilinear downsampling layer and before the second trilinear downsampling layer. The encoded result is then processed by a self-attention module to obtain a feature map. The generator's loss function satisfy, ， in, To combat the losses, For identity loss, For space loss, For timing loss and For voxel distance loss, , , , and To assign weights to each loss term, , , , and All are greater than 0; The spatial discriminator and temporal discriminator include sequentially arranged convolutional layers, three shrinking modules, and convolutional layers; a max pooling layer is provided at the output end of each shrinking module; the shrinking module includes two connected convolutional layers and one compressed excitation block; The trained generative adversarial network converts the input CT volume data into MRI volume data.

2. The automatic generation method for converting CT volume data into MRI volume data according to claim 1, characterized in that: In the decoders of the first UNet and the second UNet, the feature maps of the corresponding encoders are input into the decoder, pass through the first trilinear upsampling layer and the convolutional layer, and correspond to the output of the last encoding transform block. Through four sequentially set decoding transform blocks, each decoding transform block is preceded by a second trilinear upsampling layer. After the last decoding transform block, it passes through a convolutional layer and is output. The convolutional layer at the output end of the first UNet also has an activation function. Each decoding transform block of the first UNet is connected to the corresponding position of the encoder of the second UNet through a residual path and the corresponding transform block before the corresponding second trilinear upsampling layer.

3. The automatic generation method for converting CT volume data into MRI volume data according to claim 2, characterized in that: In the first UNet and the second UNet, the inputs of the first and second decoding transform blocks are respectively connected to self-attention modules through a third trilinear downsampling layer. The outputs of the self-attention modules are respectively connected to the outputs of the first and second decoding transform blocks through the corresponding third trilinear upsampling layer and convolutional layer.

4. The automatic generation method for converting CT volume data into MRI volume data according to claim 1, characterized in that: The loss functions of both the spatial discriminator and the temporal discriminator include a loss for maximizing the falsity of the generator's output MRI volume data and a loss for maximizing the authenticity of the cropped MRI volume data.

5. The automatic generation method for converting CT volume data into MRI volume data according to claim 1, characterized in that: The data input to the generative adversarial network consists of raw paired CT volume data and MRI volume data that have undergone central cropping and global normalization.

6. The automatic generation method for converting CT volume data into MRI volume data according to claim 1, characterized in that: The processed CT volume data was input into the trained generative adversarial network, and the test was conducted based on the MRI volume data output by the generator; the output results of the generative adversarial network were visualized.

Images