A medical image modality conversion method and device based on pre-trained StyleGAN2

Abstract

The application discloses a medical image modal conversion method and device based on pre-training StyleGAN2, and the method comprises the following steps: constructing an MT-PSGAN model comprising an encoder, a target modal image generator and a discriminator; the encoder is used for extracting multi-scale features of a source modal image; the target modal image generator is used for controlling the attributes of a generated image by using a latent vector mapped from the multi-scale features, so as to obtain a target modal image; the discriminator is used for judging the authenticity of the target modal image obtained by the target modal image generator; the discriminator and the target modal image generator are combined for training; the trained target modal image generator and the encoder are combined for training, so as to obtain a trained encoder; and the trained encoder and the target modal image generator are used for medical image modal conversion. The application can maximize the preservation of the structure and content details of a source image during image conversion, and can achieve the optimal medical image modal conversion effect.

Description

Technical Field

[0001] This invention relates to the field of medical image modality conversion technology, and in particular to a medical image modality conversion method and apparatus based on pre-trained StyleGAN2. Background Technology

[0002] Image modality conversion, commonly used in the medical field, aims to convert the modality of a source image while preserving its details and structure to the greatest extent possible, thereby aiding in medical treatment. Medical images commonly employ modalities such as Magnetic Resonance Imaging (MRI) and Computed Tomography (CT), each with its own strengths. For example, MRI excels at capturing contrast diversity and accurately locating tumors and organs in soft tissue structures, while CT is better at revealing the details and density information of lesions to help doctors plan drug dosages. Therefore, different modalities of medical images can provide complementary information to better assist clinical diagnosis and treatment. However, in clinical practice, acquiring medical images of different modalities simultaneously is time-consuming, costly, and may even cause harm to the patient. Therefore, achieving high-quality, automated medical image modality conversion is of significant research value. To achieve goals such as complementary analysis and image fusion, medical image modality conversion strives to convert the modality of a source image while preserving its structural and content details (such as brightness and contrast) to the greatest extent possible. With the rapid development of deep learning, especially generative adversarial networks, significant progress has been made in medical image modality transfer, but the following problems still exist:

[0003] (1) The solution required in this field is unique and must ensure the anatomical characteristics of the source image as much as possible. However, the mainstream CycleGAN-based method lacks consistency constraints between the output image and the source image, which often results in inaccurate mapping and makes it difficult to ensure that the anatomical structure of the image remains unchanged. Although the pix2pix-based method can ensure the uniqueness of the solution, it focuses on pixel-level information, making it difficult to ensure the consistency of the overall content and structure before and after the conversion, and it is prone to visual artifacts and blurry results.

[0004] (2) Current medical image modality conversion models usually only input a single image slice, which may lead to problems such as decreased accuracy and inconsistency in structure before and after conversion due to insufficient input information;

[0005] (3) The field of medical image modality conversion also faces the problem of insufficient paired training data. Summary of the Invention

[0006] The purpose of this invention is to solve the problems in the prior art.

[0007] The technical solution adopted by this invention to solve its technical problem is: to provide a medical image modality conversion method based on pre-trained StyleGAN2, comprising the following steps:

[0008] An MT-PSGAN model is constructed, comprising a residual network-based encoder, a StyleGAN2-based target modality image generator, and a discriminator. The encoder is used to extract multi-scale features from the source modality image. The target modality image generator uses the latent vectors obtained from the multi-scale feature mapping to generate the target modality image.

[0009] The discriminator is used in conjunction with the target modality image generator for pre-training to obtain a trained target modality image generator; the trained target modality image generator is then used in conjunction with the encoder for training to obtain a trained encoder.

[0010] Medical image modality conversion is achieved using a trained encoder and a trained target modality image generator.

[0011] Preferably, the source modal image is a three-channel image obtained by stitching the original image and its left and right adjacent slices along the channels; if the original image is located at the edge of the three-dimensional volume data, the three-channel image is obtained by copying the original image and stitching it together.

[0012] Preferably, the encoder extracts multi-scale feature maps through a ResNet-based backbone network; feature maps of different scales are passed through different mapping networks to obtain latent vectors of different scales; each mapping network includes a convolutional layer and a LeakyReLU activation function.

[0013] Preferably, the target modality image generator includes several synthesis modules connected in sequence. The number of synthesis modules is the same as the number of latent vectors. Each synthesis module receives a latent vector and generates a corresponding feature map under the control of the latent vector. The input of the first synthesis module is a randomly generated constant, the input of other synthesis modules is the feature map output by the previous synthesis module, and the output feature map of the last synthesis module is the finally generated target modality image.

[0014] The process of generating feature maps by the synthesis module includes the following steps:

[0015] The latent vector is received and transformed into modulation parameters using affine transformation. The modulation parameters are then used to scale the weights of the convolutional kernel of the current convolutional layer.

[0016] The weights of the scaled convolutional kernels are modulated, and then the current convolutional layer is used to perform a convolution operation on the input feature map.

[0017] The random noise is processed by the learned channel-level scaling factor through the feedforward network, broadcast and added to the feature map output by the current convolutional layer to obtain the output feature map of the current synthesis module.

[0018] Preferably, the discriminator receives the target modality image, gradually downsamples it to a lower resolution image, and uses different convolutional networks to extract features at different resolution stages. The extracted features are then fused for the final discrimination decision. The downsampling layer uses bilinear filtering to smooth the image.

[0019] Preferably, the discriminator is used in conjunction with the target modality image generator for pre-training to obtain a trained target modality image generator. The pre-training adopts a strategy of transfer learning combined with adaptive discriminator enhancement.

[0020] Transfer learning is implemented by loading model parameters of a target modality generator pre-trained on the FFHQ dataset.

[0021] The adaptive discriminator enhancement strategy performs several enhancement operations on the target modality image before it is input into the discriminator. Each enhancement operation has a probability of being executed as p, expressed as:

[0022] ;

[0023] in, This refers to the sign function. Discriminator The degree of overfitting, Indicates the preset target value. This indicates an adjustment of the step size.

[0024] Preferably, the enhancement operation includes:

[0025] Geometric transformations can be used to perform horizontal flipping, integer pixel translation, 90-degree rotation, random angle rotation, proportional scaling, and anisotropic scaling on images.

[0026] Color transformation, randomly adjusting the brightness, contrast, and saturation of an image; flipping the brightness channel of an image; rotating the hue at random angles;

[0027] Random perturbation: Add random noise to the image; randomly cut a rectangular region from the image and set it to zero;

[0028] Spatial filtering applies low-pass, high-pass, and other frequency domain filtering operations to an image, with random intensity.

[0029] Preferably, the pre-training of the discriminator in conjunction with the target modality image generator to obtain a trained target modality image generator uses a training function including generator loss. With discriminator loss , respectively represented as:

[0030] ;

[0031] ;

[0032] in, This indicates that the generator is adversarial against loss. Indicates the regularization loss. express The weights; This indicates that the discriminator is susceptible to loss. This represents the gradient penalty loss. express The weight.

[0033] Preferably, a trained target modality image generator is used in conjunction with an encoder to obtain a trained encoder, and the training function used is... Represented as:

[0034] ;

[0035] in, Indicates L2 loss; This represents the loss of perceived similarity. This represents the feature embedding similarity loss; This represents the loss due to latent code regularization. , , and represents the weights of L2 loss, perceptual similarity loss, feature embedding similarity loss, and latent code regularization loss, respectively.

[0036] The present invention also provides a medical image modality conversion device based on pre-trained StyleGAN2, comprising:

[0037] The model building module constructs the MT-PSGAN model, including a residual network-based encoder, a StyleGAN2-based target modality image generator, and a discriminator. The encoder is used to extract multi-scale features from the source modality image. The target modality image generator uses the latent vectors obtained from the multi-scale feature mapping to generate the target modality image.

[0038] The model training module uses a discriminator in conjunction with a target modality image generator for pre-training to obtain a trained target modality image generator; then, the trained target modality image generator is used in conjunction with an encoder for training to obtain a trained encoder.

[0039] The model application module utilizes a trained encoder and a trained target modality image generator to achieve modality conversion of medical images.

[0040] The present invention has the following beneficial effects:

[0041] (1) To address the problem that existing medical image modality conversion methods cannot guarantee the consistency of structure and content before and after conversion, MT-PSGAN utilizes the correlation between adjacent cross sections to promote the extraction of structural information and builds a generator based on StyleGAN2. It uses the source image content details and structural features extracted by the encoder to decouple and control the generation process of the target image, thereby better preserving the inherent semantic features of the source image.

[0042] (2) To address the lack of training data in this field, MT-PSGAN employs a two-stage training strategy that includes adversarial pre-training based on transfer learning and adaptive data augmentation, as well as supervised training of the encoder.

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Attached Figure Description

[0044] Figure 1 This is a diagram illustrating the method steps of an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the structure of the MT-PSGAN model according to an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the target modality image generator according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the generator and discriminator in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the training strategy according to an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram showing the qualitative comparison results of the method of this embodiment of the invention with other methods on the Brain test set;

[0050] Figure 7 This is a schematic diagram showing the qualitative comparison results of the method of this embodiment of the invention with other methods on the Pelvis test set;

[0051] Figure 8 This is a schematic diagram showing the comparison results of ablation experiments during the pre-training phase of this invention.

[0052] Figure 9This is a schematic diagram illustrating the qualitative comparison results of ablation experiments during the pre-training phase of this invention.

[0053] Figure 10 This is a schematic diagram comparing the indicators of whether the source modal image of the method of the present invention has adjacent cross-sectional information; wherein, (a) is a schematic diagram of SSIM comparison, (b) is a schematic diagram of MAE comparison, and (c) is a schematic diagram of LPIPS comparison.

[0054] Figure 11 This is a schematic diagram illustrating the qualitative results of an extension experiment of the method in an embodiment of the present invention.

[0055] Figure 12 This is a structural diagram of the device according to an embodiment of the present invention. Detailed Implementation

[0056] This invention proposes a medical image modality transformation model based on StyleGAN2. It extracts the details and structural features of the source image through an encoder and uses these features to decouple and control the generation process of the target image, thereby preserving the semantic information of the source image to the greatest extent.

[0057] See Figure 1 The diagram shows the method steps of an embodiment of the present invention, including the following steps:

[0058] S101, Construct the MT-PSGAN model, including a residual network-based encoder, a StyleGAN2-based target modality image generator, and a discriminator;

[0059] S102, the discriminator is used in conjunction with the target modality image generator for pre-training to obtain a trained target modality image generator; the trained target modality image generator is used in conjunction with the encoder for training to obtain a trained encoder.

[0060] S103 utilizes a trained encoder and a trained target modality image generator to achieve medical image modality conversion.

[0061] The encoder is used to extract multi-scale features from the source modal image; the target modal image generator is used to control the attributes of the generated image using the latent vectors obtained from the multi-scale feature mapping to obtain the target modal image; and the discriminator is used to determine whether the target modal image obtained by the target modal image generator is real or fake.

[0062] Specifically, the MT-PSGAN model can be found here. Figure 2 As shown, it includes: a source image encoder Used to extract content and structural information from MRI source images; target modality image generator. , for use Extracted source image features generate target images for CT modality; target modality image discriminator based on adaptive data augmentation strategy. Used to cooperate Conduct competitive training.

[0063] Specifically, the source image encoder MT-PSGAN is a progressive style encoder used to extract multi-level style features from source images to control the generation process of the Generative Style Transformer (GCT). Similar to other StyleGAN-based network models, the "style" in MT-PSGAN can be understood as the attribute features of the source image at different levels. These attribute features are encoded into latent vectors that control different visual features of the generated image. (Source image encoder) First, a ResNet-based backbone network is used to extract multi-scale feature maps. Then, feature maps of different resolutions are input into different mapping networks. Each mapping network is a small fully convolutional network that progressively reduces the spatial size through a series of convolutional layers with a stride of 2 and the LeakyReLU activation function, ultimately generating a 512-dimensional style vector. Finally, all style vectors are concatenated to form a style latent code w, which is input into Gct to achieve multi-level attribute control of the generated image. Because the image resolution in this embodiment is uniformly set to 256×256, the W+ space corresponding to MT-PSGAN is composed of 14 concatenated style vectors. Low-resolution feature maps generate coarse-grained style vectors {w}. i} 2 i=0 It primarily controls the global features of the image (such as structure and shape); medium-resolution feature maps generate medium-granularity style vectors {w i} 6 i=3 It mainly affects local details (such as texture and color distribution); while high-resolution feature maps generate fine-grained style vectors {w i} 13 i=7 It is used to further refine the subtle features of an image (such as lighting and texture).

[0064] Furthermore, a single 2D slice loses the inherent spatial continuity information of the 3D volume data, thus affecting the model's ability to maintain the consistency of shape and structure before and after image transformation. To address this issue, MT-PSGAN... and adjacent cross sections within the same 3D volume data and Join them together along the channel, then input. This allows us to utilize relevant spatial information to further improve the quality of image conversion. To avoid errors, when When located at the edge of a 3D volume data, MT-PSGAN will copy along the channel. This is to accommodate the input requirements of a three-channel model, rather than using slices of other volumetric data. The reason is that even if cross-sections located in different volumetric data are adjacent, they are artificially placed together and lack spatial continuity and correlation.

[0065] Specifically, MT-PSGAN utilizes a target modality image generator. To generate target CT images . It is based on StyleGAN2, but its style features do not come from random noise like StyleGAN2, but rather from... from Style latent codes extracted from its adjacent cross sections StyleGAN introduces a style-based generation mechanism, achieving fine-grained control over the image generation process by decoupling the latent space and performing adaptive instance normalization (AdaIN) on the feature maps, thus improving the quality of the generated images. However, because AdaIN normalizes each feature map individually, information about the relative amplitudes between features is lost. This allows the generator to dominate statistics by creating local peaks, thereby bypassing the limitations of instance normalization and leaking signal intensity information, and then freely scaling the signal in other regions. This phenomenon often results in blotchy artifacts in the images generated by StyleGAN. StyleGAN2 improves upon AdaIN by using modulated convolution instead of directly operating on the feature maps, thus solving the artifact problem while maintaining fine-grained control over the generated images.

[0066] Depend on Figure 4 As shown, It consists of multiple synthesis modules stacked together, each containing two modulation convolutional layers, utilizing... Extracted latent vectors This controls the attributes of the generated image. In each modulation convolutional layer, First, it will be done through affine transformation. (Fully connected layer) will store style vectors Convert to modulation parameters and through Weights of the current layer convolutional kernel Scaling is performed instead of scaling the style vector directly onto the feature map in the original StyleGAN. The scaling process is represented as follows:

[0067] ;

[0068] in, and These are the channel indices of the input and output feature maps, respectively. For the spatial location of the weights in the convolution kernel and Representing the first The weights of the convolutional kernel before and after modulation. Instance normalization aims to eliminate the influence of modulation operations on the statistics of the output feature map. Instead, a weighted demodulation operation is used to avoid directly manipulating the feature map. The weighted modulation is represented as:

[0069] ;

[0070] in It is a small constant used to avoid numerical problems. Each modulation convolutional layer, after completing weight modulation, weight demodulation, and convolution operations, will pass through... The learned channel-level scaling factor is used to manipulate random noise, which is then broadcast and added to the feature map output by the current convolutional layer. This feature map is then input to the next layer, thereby controlling the flow of noise. Generate random details.

[0071] if Using a simple progressively growing feedforward network to improve the quality of high-resolution image generation results in each resolution stage temporarily serving as the output resolution, causing the generator to generate high-frequency details (such as sharp textures and edges) at each stage. This leads to the generator's intermediate layers retaining too much high-frequency information, thus violating the translation invariance of the convolutional network and causing feature retention. Specifically, some features cannot move smoothly in the image, but instead get stuck at a certain position and suddenly jump to the next generator-preferred position, affecting the continuity and naturalness of the generated results. Therefore, the embodiments of the present invention... It uses a skip connection approach to build the image, eliminating the need for explicit, progressive image generation. Figure 4 (a) shows the specific details, where the upsampling layer uses bilinear filtering, which calculates the new pixel value by weighting the four neighboring pixels to smooth the image, as shown below:

[0072] ;

[0073] in, , , and These are the values ​​of the input image at its four nearest neighbors. and These are two normalization factors, corresponding to the normalized distance values ​​in the horizontal and vertical directions, respectively. The toOut module, like the synthesis module, is based on the modulation convolution mentioned above, with a kernel size of [missing value]. The purpose is to adapt the feature map to the requirements of the number of channels in the output image.

[0074] MT-PSGAN utilizes a discriminator To determine the authenticity of CT images, It is built based on the residual structure. For example... Figure 4 As shown in (b) in the figure, Starting with a high-resolution input image, the image is gradually downsampled to a lower resolution, and different convolutional networks are used to extract features at different resolution stages. Features extracted from different convolutional networks will be fused for the final discrimination decision. This is achieved through a residual structure. It can not only focus on image details (high-resolution parts) but also grasp the overall information of the image (low-resolution parts), thereby improving the ability to judge the realism of the image. The same as the upsampling layer, The downsampling layer also uses bilinear filtering to smooth the image.

[0075] Specifically, embodiments of the present invention propose a two-stage training strategy, such as... Figure 5 As shown. In the first stage, MT-PSGAN is pre-trained based on transfer learning and adaptive data augmentation, mainly including training on the target CT image set in this embodiment of the invention. and Generative adversarial training is performed, and prior knowledge learned on large-scale datasets (such as FFHQ) is utilized through transfer learning to alleviate the lack of training data in the field of medical image modality translation. The second stage involves fixing... The parameters for training the source image encoder This allows the style latent code it encodes to be decoupled from control. Generate a CT image that is identical in content and structure to the source MRI image.

[0076] The first stage involves pre-training based on transfer learning and adaptive data augmentation. For example... Figure 5 As shown in (a) above, in the first stage, and Based on generator loss and counter its losses Conduct competitive training. and They are represented as follows:

[0077] ;

[0078] ;

[0079] in, This indicates that the generator is adversarial against loss. Indicates the regularization loss. express The weights are obtained based on the image resolution in this embodiment of the invention. ; This indicates that the discriminator is susceptible to loss. This represents the gradient penalty loss. express The weight.

[0080] Specifically, the formulas for calculating each type of loss are as follows:

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] ;

[0086] ;

[0087] in, The resolution of the output image; express The current path length, where express In latent vector The Jacobian matrix at the location; The pixel intensity follows a normal distribution. A random vector used to randomly select a direction; This represents the number of pixels in the image. This represents the L2 norm, used to find the square root of the sum of the squares of the elements. It can be simplified into a form that is easier to compute during backpropagation:

[0088]

[0089] Meanwhile, training generative adversarial models with limited data can easily lead to discriminator overfitting, preventing the generator from receiving effective feedback and causing training failure. Medical image data is often difficult to obtain due to technical challenges and the potential for harm to humans. Therefore, an adaptive data augmentation mechanism was employed to stabilize the training of the generative adversarial model in the field of medical image modality transformation. Data augmentation is a common method to prevent model overfitting; however, ordinary data augmentation methods can easily cause the generator to learn the augmented distribution. For example, noise augmentation may introduce noise into the generated images that is not present in the original dataset—a phenomenon known as augmentation leakage. To achieve non-leakage data augmentation, there should be a certain probability (1−p) of skipping the augmentation operation, allowing the generator to ignore the augmentation and learn the original distribution. However, if a fixed value of p is used, a value that is too large will cause the generator to learn an incorrect distribution, while a value that is too small will prevent the discriminator from overfitting. Furthermore, besides the difficulty in choosing a suitable value, a fixed p may also lead to excessively strong data augmentation in the early stages of training and insufficient strength in later stages. To address the above issues, the Adaptive Discriminator Augmentation (ADA) strategy dynamically adjusts the strength of data augmentation (i.e., the probability p of data augmentation) based on the degree of overfitting of the discriminator. This avoids manually setting the p value and allows the ADA strategy to adapt to different stages of model training.

[0090] MT-PSGAN also employs the ADA strategy to augment the data before input. Conduct training. Specifically, this includes the following steps:

[0091] measure overfit , is represented as:

[0092] ;

[0093] ;

[0094] according to Compared with the preset target value The magnitude of η determines the direction in which p should be adjusted (the initial value of p can be preset), and the step size for each adjustment is η:

[0095] ;

[0096] Controls ensure that the value of p is greater than 0:

[0097] .

[0098] In the input Previously, MT-PSGAN would perform a series of enhancement operations on the image, with each enhancement operation having a probability of p. The enhancement operations included: (1) Geometric transformation: horizontal flipping, integer pixel translation, 90-degree rotation, random angle rotation, proportional scaling, and anisotropic scaling of the image. (2) Color transformation: randomly adjusting the brightness, contrast, and saturation of the image; flipping the brightness channel of the image and rotating the hue at random angles. (3) Random perturbation: adding random noise to the image; randomly cutting a rectangular area on the image and setting it to zero. (4) Spatial filtering: applying low-pass, high-pass, and other frequency domain filtering operations to the image with random intensity.

[0099] In addition, during the and Prior to adversarial training, the model parameters pre-trained on the FFHQ dataset were loaded using the concept of transfer learning. The effectiveness of transfer learning depends more on the diversity of the original datasets than on the similarity between datasets. Therefore, even with some differences between the face dataset and the CT image set, transfer learning can still help MT-PSGAN improve model performance in the absence of training data.

[0100] The second stage involves supervised training of the source image encoder. Unlike general image translation tasks, the optimal solution in the field of medical image modality translation is unique; therefore, it is based on... The model is trained to generate images that closely resemble the target in terms of content, structure, and perception. For example... Figure 5 As shown in (b) above, at this stage, The parameters remain fixed and only coordinate during the forward process. The loss function for performing modality conversion on an image. It includes the following four parts: (1) L2 loss : Used to guide the model to output an image whose pixel values ​​are close to those of the target image. (2) Perceptual similarity loss : Guide the model to output an image that is perceptually similar to the target image. (3) Feature embedding similarity loss : Used to guide the model output to an image whose feature embedding is similar to that of the target image. (4) Latent code regularization loss This is used to truncate or reduce the sampling space of latent vectors, thereby improving the average quality of the generated image. In general, Represented as:

[0101] .

[0102] Specifically, the L2 loss is expressed as:

[0103] ;

[0104] ;

[0105] Where N represents the number of pixels in the image; This represents the image output by the generator. Represents the target image;

[0106] Specifically, perceptual similarity loss Represented as:

[0107] ;

[0108] in, express In the Features extracted from layers; vectors Its dimensions and the first The feature maps of each layer have the same number of channels, representing the weight of each channel. This can generally be determined from... Load directly from the middle; and They represent the first The height and width of the layer feature map.

[0109] Specifically, feature embedding similarity loss Represented as:

[0110] ;

[0111] in, This indicates the use of ResNet trained using MOCOv2. This represents the feature embedding extracted from the input image. MOCOv2 is a self-supervised learning method based on contrastive learning. It learns more discriminative feature representations by constructing a dynamic dictionary and is widely used in tasks such as image classification and object detection.

[0112] Specifically, latent code regularization loss Represented as:

[0113] ;

[0114] in, ; It is random noise. for Medium noise A feedforward network mapped to style latent codes. Guided model shrinkage coding The generated latent code and The distance from the center point of the latent code space.

[0115] Verification experiments were conducted on the embodiments of the present invention, including comparative experiments between the method of the present invention and other methods, ablation experiments, and extension experiments. The dataset used came from the publicly available data in the Syn-thRAD2023 large-scale challenge, containing CT images of the brain and pelvis, and rigidly registered MRI images. The brain images were used as the Brain dataset, and the CT images as the Pelvis dataset. A portion of the datasets was randomly selected as the test set for convenient comparison in subsequent work. The Brain dataset contained 180 subjects, and in this embodiment of the invention, 36 subjects were randomly selected to form the test set, containing 7486 slices. The remaining 144 subjects formed the training set, containing 31544 slices. The Pelvis dataset also contained 180 subjects, and 20% was randomly selected as the test set, containing 16765 slices. The remaining subjects formed the training set, containing 66048 slices. The original data were all saved as compressed NiFTI files. For the three-dimensional volume data in each file, two-dimensional slices were read one by one along the first dimension, and the data of each slice was normalized to the range of 0-255 before being saved as PNG images. In the first stage, to facilitate the training of Gct and Dct, single-channel MRI and CT images were copied at the same rate as three-channel images. In the second stage, MRI images were combined with their adjacent slices to form three-channel images, while the loading method for CT images remained unchanged. Furthermore, all images were resized to a fixed size of 256×256 before being input into the model.

[0116] Results and Analysis of Comparative Experiments. MT-PSGAN was compared with four current state-of-the-art modality transfer algorithms, including pix2pix, CycleGAN, AttentionGAN, and MaskGAN. Quantitative comparison results are shown in Tables 1 and 2. SSIM (Structural Similarity) represents structural similarity, evaluating the similarity of two images in brightness, contrast, and structure; a higher SSIM value indicates better transfer performance. MAE (Mean Absolute Error) represents mean absolute error, measuring the pixel-level difference between two images; a lower MAE value indicates better transfer performance. LPIPS (Learned Perceptual Image Patch Similarity) represents learned perceptual image patch similarity, calculating the differences between images in feature space; a lower LPIPS value indicates better transfer performance.

[0117] Table 1 - Quantitative results of different mode transition methods on the Brain test set:

[0118]

[0119] Table 2 - Quantitative results of different mode transition methods on the Pelvis test set:

[0120]

[0121] As shown in Tables 1 and 2, the method of this embodiment achieves optimal results across all three metrics on both datasets. See also... Figure 6 and Figure 7 The diagram illustrates a qualitative comparison between the method of this invention and other methods on two datasets. It shows the output results of each modality conversion method on the two datasets and the corresponding difference heatmaps. Brighter colors in the difference heatmaps indicate greater errors. It can be seen that the method of this invention can effectively maintain the consistency of structural and content details before and after model conversion on both datasets, making the output image as close as possible to the target image in all aspects.

[0122] The results and analysis of the ablation experiments, including ablation experiments in two training phases.

[0123] In the first training phase of the ablation experiment, while keeping other modules unchanged, the model was trained by removing transfer learning and ADA (Adaptive Data Augmentation) separately. After training, Gct was used to generate the same number of CT images as the Brain test set, and the performance of Gct under different training methods was quantitatively evaluated by calculating the FID value. See [link to relevant documentation]. Figure 8 As shown, FID (Fréchet Inception Distance) measures the similarity between the real image distribution and the generated image distribution by calculating the Fréchet distance between them. A smaller FID indicates greater similarity between the two distributions, resulting in better image generation. It can be seen that FID increases significantly after removing transfer learning, indicating that transfer learning has a significant impact on image quality, and ADA also affects the quality of the generated image to some extent. See also Figure 9 As shown, without the prior knowledge from transfer learning, adversarial training suffers from pattern collapse, and the Gct can only repeatedly generate meaningless images. However, with the help of transfer learning and ADA, the Gct trained in the first stage is already able to generate CT images that are indistinguishable from real ones. In addition, this stage also tried different augmentation methods based on transfer learning, including data-free augmentation and augmentation methods of the same series as ADA with fixed probabilities. The results are shown in Table 3.

[0124] Table 3 - Ablation results of enhanced methods:

[0125]

[0126] When using a fixed-probability augmentation method, the experimental results are highly sensitive to the selection of the hyperparameter probability p. When p is 1.0, FID increases significantly, indicating that Gct has learned a completely shifted distribution. When p is set to a smaller value, FID gradually decreases, but when p drops to 0, i.e., no data augmentation, the FID value rebounds. This suggests that when performing fixed-probability augmentation, it is difficult to select a suitable value for probability p, and a fixed p value is not suitable for the entire training process. ADA, by adaptively adjusting the augmentation probability during training, achieved the lowest FID value, meaning that Gct is trained best under this augmentation method.

[0127] The ablation experiment in the second training phase conducted ablation experiments on three key modules (transfer learning, adversarial pre-training ADVP, and continued training Gct), trying all eight possible combinations (i.e., enabling or disabling each module) to analyze the contribution of each module to the final performance. The experimental results are shown in Table 4, where the fourth training strategy is the training strategy adopted in the embodiment of the present invention.

[0128] Table 4 - Ablation results of the second-stage training strategy based on adjacent cross-sectional information:

[0129]

[0130] As can be seen from the table, the training strategy of this invention has achieved optimal results in the three metrics of SSIM, LPIPS and MAE.

[0131] In addition to the training strategies, the importance of utilizing adjacent cross-sectional information from the source image was also explored. The source image was copied at the same value as the three-channel input image, and the model was retrained using each of the eight strategies described above. As shown in Table 5, even without adjacent cross-sectional information, the fourth training strategy still performed best across all three metrics, further validating the superiority and effectiveness of the training strategies in this embodiment.

[0132] Table 5 - Ablation results of the second-stage training strategy without adjacent cross-sectional information:

[0133]

[0134] However, when trained using the same fourth strategy, failing to use adjacent cross-sectional information resulted in a 1.7%, 5.22%, and 8.06% decrease in SSIM, LPIPS, and MAE, respectively, validating the importance of the correlation between cross-sections to the structure and content of the output image. See also Figure 10 As shown, three bar charts visually illustrate the impact of the presence or absence of adjacent cross-sectional information on model performance under different training strategies; for example... Figure 10As shown in (a) and (b), regardless of the training strategy used, adjacent cross-sectional information helps the model improve structural consistency before and after mode transitions, thereby increasing the SSIM value and decreasing the MAE value; this phenomenon is also reflected in the LPIPS metric. Figure 10 As can be seen in (c), under almost all training strategies, utilizing information from adjacent cross sections can help the model achieve lower LPIPS values, making the output image more perceptually similar to the target image.

[0135] While maintaining consistency with MT-PSGAN in model design and training strategy, a certain loss function was removed to verify its effectiveness. As shown in Table 6, when the l2 loss was removed, SSIM, LPIPS, and MAE deteriorated by 1.83%, 2.99%, and 8.06%, respectively, indicating that the l2 loss is significant in improving the pixel-level similarity between the output and target images. When the perceptual similarity loss Llpips was removed, in addition to the severe deterioration of LPIPS, SSIM and MAE deteriorated by 6.53% and 9.68%, respectively, further demonstrating that Llpips is important for the structure and details of the output image. Without Lmoco, SSIM and LPIPS deteriorated by 1.04% and 4.48%, respectively. The experiment also explored the impact of regularizing the latent code on the output image quality by removing Lwreg. Without Lwreg, SSIM, LPIPS, and MAE deteriorated by 1.31%, 0.75%, and 1.61%, respectively.

[0136] Table 6 - Ablation results for different loss functions in the second stage:

[0137]

[0138] This invention also specifically conducted ablation experiments on the weight λwreg of the latent code regularization loss. A larger λwreg indicates that the model pays more attention to the constraints on the sampling space during training. Table 7 shows the training results of the model with different values ​​of λwreg. As can be seen from the table, when λwreg is 0.001, all three indicators achieve the best results, which is also the final experimental setting of MT-PSGAN. When the value of λwreg is too large, it will lead to an excessive truncation of the model's sampling space, thus failing to fully utilize the target loss to fit the data points; while when λwreg is 0, it will affect the overall performance because the model has difficulty learning the features of low-density regions.

[0139] Table 7 - Ablation results of loss function weights:

[0140]

[0141] To explore the versatility of MT-PSGAN in the field of medical image modality transformation, an extended experiment was conducted to enable it to perform modality transformation from CT to MRI. The experiment was first trained on the Brain training set. Except for the change in the direction of modality transformation, the model structure, training strategy, and loss function design remained consistent with the previous methods. In the first training phase, adversarial pre-training based on transfer learning and adaptive data augmentation was performed on the MRI image set in the Brain training set, enabling the generator to produce realistic-looking MRI images. In the second phase, the model was input with source CT images and their adjacent cross-sectional information. The generator's pre-training parameters were kept constant, and supervised training was performed on the source image encoder. After training, the model performance was evaluated on the Brain test set. The quantitative evaluation results are shown in Table 8. The table shows that MT-PSGAN can perform the reverse modality transformation task well without any modifications, even when outputting MRI images with more complex textures than CT images. Figure 11 The results of the extended experiment are shown. As can be seen from the figure, MT-PSGAN can complete the modality transfer task from CT to MRI while maintaining the structural and content consistency with the source image, which further illustrates the versatility and effectiveness of MT-PSGAN.

[0142] Table 8 - Quantitative results of the extension experiment:

[0143]

[0144] This invention also provides a medical image modality conversion device based on pre-trained StyleGAN2, comprising:

[0145] Model building module 1201 constructs the MT-PSGAN model, including a residual network-based encoder, a StyleGAN2-based target modality image generator, and a discriminator. The encoder is used to extract multi-scale features from the source modality image. The target modality image generator uses the latent vectors obtained from the multi-scale feature mapping to control the attributes of the generated image to obtain the target modality image. The discriminator is used to determine whether the target modality image obtained by the target modality image generator is real or fake.

[0146] The model training module 1202 uses a discriminator in conjunction with a target modality image generator for pre-training to obtain a trained target modality image generator; the trained target modality image generator is then used in conjunction with an encoder for training to obtain a trained encoder.

[0147] Model application module 1203. This module utilizes a trained encoder and a trained target modality image generator to achieve modality conversion of medical images.

[0148] This invention proposes the MT-PSGAN medical image modality transfer model, comprising a source image encoder and a generator and discriminator based on StyleGAN2. MT-PSGAN employs a two-stage training strategy: in the first stage, the generator is pre-trained adversarially based on transfer learning and adaptive data augmentation; in the second stage, the generator parameters are fixed, and only the source image encoder is trained. Furthermore, the correlation between adjacent cross-sections is also utilized in MT-PSGAN. Comparative experimental results with other state-of-the-art methods demonstrate that MT-PSGAN can preserve the structural and content details of the source image to the greatest extent during image transfer, thus achieving optimal modality transfer performance. The embodiments of this invention employ extensive ablation experiments to verify the correctness and effectiveness of the key designs of MT-PSGAN; in addition, further extended experiments were conducted to demonstrate the versatility of MT-PSGAN in the field of medical image modality transfer.

[0149] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A medical image modality conversion method based on pre-trained StyleGAN2, characterized in that, Includes the following steps: An MT-PSGAN model is constructed, comprising a residual network-based encoder, a StyleGAN2-based target modality image generator, and a discriminator; the encoder is used to extract multi-scale features from the source modality image. The target modality image generator generates a target modality image using the latent vectors obtained from the multi-scale feature mapping; The discriminator is used in conjunction with the target modality image generator for pre-training to obtain a trained target modality image generator; the trained target modality image generator is then used in conjunction with the encoder for training to obtain a trained encoder. Medical image modality conversion from MRI images to CT images is achieved using a trained encoder and a trained target modality image generator. The trained target modality image generator is used in conjunction with the encoder to obtain a trained encoder. The training function used is... Represented as: ； in, Indicates L2 loss; This represents the loss of perceived similarity. This represents the feature embedding similarity loss; This represents the loss due to latent code regularization. , , and , respectively, represent the weights of L2 loss, perceptual similarity loss, feature embedding similarity loss, and latent code regularization loss; Among them, feature embedding similarity loss Represented as: ； in, This indicates that feature embeddings are extracted from images using a ResNet trained with MOCOv2. This represents the image output by the generator. Represents the target image. Represents the L2 norm; The source modal image is a three-channel image obtained by stitching the original image and its left and right adjacent slices along the channels; if the original image is located at the edge of the three-dimensional volume data, the three-channel image is obtained by copying the original image and stitching it together.

2. The medical image modality conversion method based on pre-trained StyleGAN2 according to claim 1, characterized in that, The encoder extracts multi-scale feature maps through a ResNet-based backbone network; feature maps of different scales are passed through different mapping networks to obtain latent vectors of different scales; each mapping network includes convolutional layers and the LeakyReLU activation function.

3. The medical image modality conversion method based on pre-trained StyleGAN2 according to claim 1, characterized in that, The target modality image generator includes several synthesis modules connected in sequence. The number of synthesis modules is the same as the number of latent vectors. Each synthesis module receives a latent vector and generates a corresponding feature map under the control of the latent vector. The input of the first synthesis module is a randomly generated constant, the input of other synthesis modules is the feature map output by the previous synthesis module, and the output feature map of the last synthesis module is the final generated target modality image. The process of generating feature maps by the synthesis module includes the following steps: The latent vector is received and transformed into modulation parameters using affine transformation. The modulation parameters are then used to scale the weights of the convolutional kernel of the current convolutional layer. The weights of the scaled convolutional kernels are modulated, and then the current convolutional layer is used to perform a convolution operation on the input feature map. The random noise is processed by the learned channel-level scaling factor through the feedforward network, broadcast and added to the feature map output by the current convolutional layer to obtain the output feature map of the current synthesis module.

4. The medical image modality conversion method based on pre-trained StyleGAN2 according to claim 1, characterized in that, The discriminator receives the target modality image, gradually downsamples it to different low-resolution images, and uses different convolutional networks to extract features at different resolution stages. The extracted features are then fused for the final discrimination decision. The downsampling layer uses bilinear filtering to smooth the image.

5. The medical image modality conversion method based on pre-trained StyleGAN2 according to claim 1, characterized in that, The discriminator is used in conjunction with the target modality image generator for pre-training to obtain a trained target modality image generator. The pre-training adopts a strategy of transfer learning combined with adaptive discriminator enhancement. Transfer learning is implemented by loading model parameters of a target modality generator pre-trained on the FFHQ dataset. The adaptive discriminator enhancement strategy performs several enhancement operations on the target modality image before it is input into the discriminator. Each enhancement operation has a probability of being executed. , is represented as: ； in, This refers to the sign function. Discriminator The degree of overfitting, Indicates the preset target value. This indicates an adjustment of the step size.

6. The medical image modality conversion method based on pre-trained StyleGAN2 according to claim 5, characterized in that, The enhancement operations include: Geometric transformations can be used to perform horizontal flipping, integer pixel translation, 90-degree rotation, random angle rotation, proportional scaling, and anisotropic scaling on images. Color transformation, randomly adjusting the brightness, contrast, and saturation of an image; flipping the image's luminance channel and rotating the hue at random angles; Random perturbation: Add random noise to the image; randomly cut a rectangular region from the image and set it to zero; Spatial filtering applies low-pass or high-pass frequency domain filtering to an image, with random intensity.

7. The medical image modality conversion method based on pre-trained StyleGAN2 according to claim 1, characterized in that, The discriminator is used in conjunction with the target modality image generator for pre-training to obtain a trained target modality image generator. The training function used includes the generator loss. With discriminator loss , respectively represented as: ； ； in, This indicates that the generator is adversarial against loss. Indicates the regularization loss. express The weights; This indicates that the discriminator is susceptible to loss. This represents the gradient penalty loss. express The weight.

8. A medical image modality conversion device based on pre-trained StyleGAN2, characterized in that, include: The model building module constructs the MT-PSGAN model, including a residual network-based encoder, a StyleGAN2-based target modality image generator, and a discriminator; the encoder is used to extract multi-scale features of the source modality image. The target modality image generator generates a target modality image using the latent vectors obtained from the multi-scale feature mapping; The model training module uses a discriminator in conjunction with a target modality image generator for pre-training to obtain a trained target modality image generator; then, the trained target modality image generator is used in conjunction with an encoder for training to obtain a trained encoder. The model application module utilizes a trained encoder and a trained target modality image generator to achieve medical image modality conversion from MRI images to CT images; The trained target modality image generator is used in conjunction with the encoder to obtain a trained encoder. The training function used is... Represented as: ； in, Indicates L2 loss; This represents the loss of perceived similarity. This represents the feature embedding similarity loss; This represents the loss due to latent code regularization. , , and , respectively, represent the weights of L2 loss, perceptual similarity loss, feature embedding similarity loss, and latent code regularization loss; Among them, feature embedding similarity loss Represented as: ； in, This indicates that feature embeddings are extracted from images using a ResNet trained with MOCOv2. This represents the image output by the generator. Represents the target image. Represents the L2 norm; The source modal image is a three-channel image obtained by stitching the original image and its left and right adjacent slices along the channels; if the original image is located at the edge of the three-dimensional volume data, the three-channel image is obtained by copying the original image and stitching it together.

Images