An Adaptive Mask-Guided Image Modality Transformation Method Based on Recurrent Generative Adversarial Networks
By using an adaptive mask-guided image modality transformation method, combined with the Unet encoder/decoder and PatchGAN discriminator, the problems of inconsistent topological structure and single grayscale features in medical image modality transformation are solved, achieving diversification of modality transformation results and structural consistency.
Patent Information
- Application Number
- CN202211220715.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-08
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-10-08
AI Technical Summary
Existing recurrent generative adversarial network models cannot guarantee the consistency of topological structure before and after medical image modality conversion, and the conversion results only have a single grayscale feature, which cannot adapt to the diverse grayscale features under different devices and parameters.
An adaptive mask-guided image modality conversion method is adopted. By constructing an adaptive mask network, combining the Unet encoder/decoder and the PatchGAN discriminator, the mask feature generation module modulates multi-scale feature maps at skip connections, and the features of the original image and the reference image are fused by adaptive normalization to generate a modality conversion image.
This method achieves medical image modality conversion results that simultaneously possess the modal attributes of the target domain and the grayscale attributes of the reference image, while maintaining the consistency of the topological structure before and after image conversion.
Smart Images

Figure CN115587924B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing, and in particular to an adaptive mask-guided image modality conversion method based on a recurrent generative adversarial network. Background Technology
[0002] Image-to-image modality transformation is one of the key research tasks in the field of deep learning. In this task, we map an image from one modality to its corresponding modality; for example, obtaining the T2 modality map of a brain image from its T1 modality. In 2017, Zhu et al. proposed an image-to-image transformation network, applying conditional generative adversarial networks (GANs) to the image transformation task. In the same year, Zhu et al. further proposed Cyclic Generative Adversarial Networks (CyGANs), which, by adding an extra pair of generators and discriminators to form a cyclically consistent structure and adding a cyclically consistent loss during training, completely solved the problem of needing ground truth in image-to-image transformation tasks, achieving unsupervised image modality transformation.
[0003] However, existing methods for modal transformation of medical images using generative adversarial network models such as CycleGAN often fail to guarantee strict consistency in the topological structure of the medical images before and after the transformation. Moreover, the transformation result only has one grayscale feature, while in reality, different devices and parameters may produce a variety of grayscale features, which has certain shortcomings. Summary of the Invention
[0004] The purpose of this invention is to provide an adaptive mask-guided image modality transformation method based on a recurrent generative adversarial network, so that the transformed result can simultaneously possess the modal attributes of the target domain and the appearance grayscale attributes of the reference image, while ensuring the consistency of the topological structure before and after the image transformation.
[0005] To address the aforementioned technical problems, this invention provides an adaptive mask-guided image modality conversion method based on a recurrent generative adversarial network, comprising the following steps:
[0006] Image partitioning: the first ratio of original images is randomly selected to form the training set, and the remaining second ratio of original images forms the test set with ground truth values.
[0007] Image preprocessing involves normalizing the grayscale of all images in the training and test sets.
[0008] Network construction: Selecting a recurrent generative adversarial network as the overall network architecture, and constructing an adaptive mask network;
[0009] Network training utilizes preprocessed images from the training set for network training.
[0010] For quality testing, the preprocessed images in the test set are input into the trained network to perform modality transformation on the images in the test set, and the transformed images are compared with the ground truth.
[0011] Furthermore, the overall network architecture includes a mask feature generation module, a Unet codec, and a PatchGAN discriminator.
[0012] Furthermore, the specific steps of the network training include:
[0013] The training set is input into the Unet encoder and decoder for feature extraction, and multi-scale feature maps are obtained at skip connections.
[0014] The mask feature generation module is inserted into the skip connection of the Unet codec to modulate the multi-scale feature map of the test set and obtain the multi-scale mask feature map.
[0015] The obtained multi-scale mask feature maps are gradually fused in the decoding stage of the Unet codec, and transformed into a single-channel adaptive mask through convolution. This mask is then directly added to the original image to obtain the transformed image.
[0016] The generated transformed image is used as a fake sample. An image is randomly selected from the test set as a real sample. The generator is fixed while the discriminator is trained so that it can judge the authenticity of the input image. Then the discriminator is fixed and the generator is trained so that the generated transformed image cannot be judged as real or fake by the discriminator. The above steps are repeated in the manner of training a generative adversarial network.
[0017] Furthermore, the training set and the test set data include two modalities. The difference between the two modalities of the same data is only in the appearance grayscale, and the topological structure is consistent.
[0018] Furthermore, the mask feature generation module has three inputs: the input feature map to be modulated, the original image to be converted, and the reference image.
[0019] Furthermore, the mask feature generation module performs interpolation on the original image to be transformed based on the scale of the input feature map to be modulated, and performs feature extraction at the same scale.
[0020] Furthermore, the multi-scale feature map modulation step includes the following steps:
[0021] Based on the scale of the input feature map to be modulated, the original image is interpolated to the same scale as the input vector and then fed into a CNN to extract the modulation mask feature map at that scale. The mean and variance matrix are obtained by convolving the modulation mask feature map.
[0022] The reference map is input into the pre-trained VGG network to obtain a reference feature map of the same scale, and its mean and variance are calculated.
[0023] Adaptive instance normalization is performed on the mean-variance matrix using the obtained mean and variance.
[0024] Finally, the mean, variance matrix, and modulation mask feature map are spatially adaptively normalized to obtain the multi-scale mask feature map.
[0025] Furthermore, the transformed image is obtained by directly adding the mask to the original image.
[0026] Compared with the prior art, the present invention has at least the following beneficial effects:
[0027] The adaptive mask generation module proposed in this invention enables the constructed mask to adaptively adjust the appearance of medical images based on specific reference images. This allows the modality transformation result to simultaneously possess the modal attributes of the target domain and the grayscale attributes of the reference image. This module can be flexibly embedded into any feature extraction network. Furthermore, the mask-guided modality transformation method for medical images can maintain strict consistency of anatomical structures while completing the modality transformation. Attached Figure Description
[0028] Figure 1 This is a flowchart of the adaptive mask-guided image modality conversion method based on a recurrent generative adversarial network according to the present invention;
[0029] Figure 2 This is a schematic diagram of the modality conversion network structure of the adaptive mask-guided image modality conversion method based on a recurrent generative adversarial network according to the present invention;
[0030] Figure 3 This is a schematic diagram of the MFGM module in the modality conversion network of the adaptive mask-guided image modality conversion method based on recurrent generative adversarial networks of the present invention;
[0031] Figure 4 This is a diagram of the adaptive mask-guided recurrent adversarial network architecture based on recurrent generative adversarial networks, as described in this invention.
[0032] Figure 5 This is a schematic diagram comparing the transformation results of the adaptive mask-guided transformation based on a recurrent generative adversarial network according to the present invention;
[0033] Figure 6 This is a schematic diagram showing the transformation results corresponding to different reference images based on the adaptive mask guidance of the recurrent generative adversarial network of the present invention. Detailed Implementation
[0034] The adaptive mask-guided image modality conversion method based on a recurrent generative adversarial network of the present invention will be described in more detail below with reference to the schematic diagrams, which illustrate preferred embodiments of the present invention. It should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being of broad knowledge to those skilled in the art and is not intended to limit the present invention.
[0035] The invention is described more specifically by way of example in the following paragraphs with reference to the accompanying drawings. The advantages and features of the invention will become clearer from the following description and claims. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the invention.
[0036] like Figure 1 As shown, this embodiment of the invention proposes an adaptive mask-guided image modality conversion method based on a recurrent generative adversarial network, comprising the following steps:
[0037] Image partitioning: the first ratio of original images is randomly selected to form the training set, and the remaining second ratio of original images forms the test set with ground truth values.
[0038] Image preprocessing involves normalizing the grayscale of all images in the training and test sets.
[0039] Network construction: Selecting a recurrent generative adversarial network as the overall network architecture, and constructing an adaptive mask network;
[0040] Network training utilizes preprocessed images from the training set for network training.
[0041] For quality testing, the preprocessed images in the test set are input into the trained network to perform modality transformation on the images in the test set, and the transformed images are compared with the ground truth.
[0042] The following are preferred embodiments of the adaptive mask-guided image modality conversion method based on recurrent generative adversarial networks in medical image processing to clearly illustrate the content of the present invention. It should be understood that the content of the present invention is not limited to the following embodiments, and other improvements made by conventional technical means by those skilled in the art are also within the scope of the present invention.
[0043] S100. Divide the images into training and testing sets.
[0044] Specifically, taking a first ratio of 60% as an example, the second ratio is 40%. 60% of the images to be converted are randomly selected to form the training set, and the remaining 40% form the test set with ground truth values, which are the corresponding images of the images to be converted in the target domain. It is understood that the first and second ratios can also be other values; for example, the first ratio can be above 50%.
[0045] S200, Preprocess all images.
[0046] Specifically, due to differences in equipment, equipment parameters, and lighting conditions during the acquisition of MR images, the grayscale of the images is uneven. Significant grayscale differences can affect feature extraction and recognition. Therefore, the grayscale of all images needs to be normalized during the preprocessing stage. Assuming that (x, y) are the pixels in image I, each pixel is transformed to between 0 and 1 using the grayscale stretching method according to the following formula a, resulting in the normalized image I′:
[0047]
[0048] In the formula: max(I) and min(I) represent the maximum and minimum gray values among all pixels in image I, respectively.
[0049] S300, Construct a segmentation network with an encoder-decoder structure.
[0050] Specifically, we chose Unet combined with a Mask Feature Generation Module (MFGM) as the generator for the recurrent generative adversarial network, embedding our proposed MFGM at the skip connections of Unet. Unet, widely used in deep learning for medical images, has a multi-level skip connection encoding / decoding structure, enabling the fusion of multi-scale features; therefore, it is more suitable for generating masks consistent with the anatomical structure of the image to be converted. We selected a PatchGAN structure as the discriminator, which consists of three convolutional blocks composed of convolutional layers, instance normalization layers, and activation layers in sequence. The activation layers use the LeakyReLU activation function. The output is an N×N×1 tensor, where N=16, and the mean of this tensor is used to determine whether the image is a real image.
[0051] S400. Train the network using the preprocessed images from the training set. The specific training steps are as follows:
[0052] S401. The generator in the modality conversion network uses the Unet encoder-decoder structure as the basic feature extraction network. The mask feature generation module is inserted into the skip connection structure of the encoder to process the multi-scale features of the map to be converted. Assuming that the number of skip connection layers in Unet is L, then L skip connection feature maps can be obtained in the encoding stage.
[0053] S402. The L feature maps obtained in the encoding stage are input into the MFGM module respectively. The module fuses the anatomical structure features of the original image to be converted and the grayscale features of the reference image to obtain encoded skip connection mask feature maps at different scales. Its calculation is as follows: (Formula b)
[0054]
[0055] In the formula: I is the original image to be transformed, and R is an arbitrarily selected reference image in the target domain.
[0056] See Figure 2 Unet embeds multiple MFGM modules, each of which can share the same I and R. The L skip connection mask feature maps generated during the encoding stage are progressively decoded from the bottom of the decoding part and superimposed to produce the final mask feature map. in Calculate as shown in formula c below:
[0057]
[0058] In the formula: up represents interpolation upsampling, and conv represents 3×3 convolution. This indicates that tensors are concatenated along the channel dimension; the missing element in the above formula is... in The feature map extracted from the bottom of Unet.
[0059] See Figure 3 The MFGM module requires spatial adaptive normalization modulation of a feature map to ensure it retains both the topological structure of the original image to be transformed and the grayscale features of the reference image. Therefore, the input to this module includes the feature map to be modulated, the original image I to be transformed, and the reference image R. Assume an input feature tensor F... n,c,y,x The calculation process in MFGM is as follows: (Equation d)
[0060]
[0061] In the formula: N represents the batch size of the input feature map, C represents the number of channels, and H and W represent the height and width. γ n,c,y,x (I,R) and β n,c,y,x (I,R) are modulation parameters learned from the target feature map and the reference map; they have the same scale as the target feature map, μ. c and σ c These are the mean and variance of the input feature map calculated by channel c. The calculation process for the mean and variance is as follows: equations e and f:
[0062]
[0063]
[0064] γ n,c,y,x and β n,c,y,x These are two parameters required for spatial adaptive normalization modulation. On one hand, since they are tensors obtained by convolving the original image to be transformed, they contain the topological information of the original image. On the other hand, during their computation, the style attributes of the reference image are fused into these two parameters through the adaptive instance normalization method (Adain), thus including the grayscale attributes of the reference image. Their computation process is as follows: (g, h)
[0065]
[0066]
[0067] I γk ,I βk =conv(reshape(I))
[0068] R fi =VGG(R), i = 1, 2, ..., L
[0069] In the formula I γk ,I βk The mean and variance matrices are obtained by transforming the image to be transformed (I) to the same scale as the input feature map to be modulated through bilinear interpolation and then convolving it; μ n,c and σ n,c Indicate I γk ,I βk The mean and variance are calculated based on the batch size n and channel c, and their calculation process is similar to that of equations e and f; R fi This refers to the i-th feature map among L feature maps obtained by passing the reference image through a pre-trained VGG network, where its scale is the same as that of the feature map to be modulated. n,c and σ n,c These represent the mean and variance calculated based on the batch quantity n and channel c, and their calculation process is similar to that of equations e and f.
[0070] S403: When the mask feature maps of the encoding and decoding ends are obtained... and Then, the images are concatenated by channel and input into a 1×1 convolution to generate a mask with 1 channel. Finally, this mask is directly added to the original image I to be converted to generate the mode transformation image I′. The calculation process is as follows: Equation I:
[0071]
[0072] In the formula, conv represents 1×1 convolution. This indicates a tensor splicing operation.
[0073] See Figure 4 A recurrent generative adversarial network consists of two generators and two discriminators. The generator is a mode switching network, and the generator and discriminator are trained alternately.
[0074] S404: Use the generated transformed image as a fake sample, and randomly select an image from the target domain as a real sample. While keeping the generator fixed, train the discriminator to distinguish between real and fake input images. Then, keep the discriminator fixed and train the generator so that its generated images cannot be judged as real or fake by the discriminator. Repeat the above steps according to the traditional generative adversarial network training method. The main objective functions optimized during training are generative adversarial loss, cycle consistency loss, and style similarity loss. The generative adversarial loss is as follows: (Equations for j and k are provided in the original text.)
[0075]
[0076]
[0077] Where: G X→Y G represents the X-mode to Y-mode conversion network. Y→X Similarly; D Y This represents a discriminator that determines whether an image belongs to the Y mode, D. X Similarly.
[0078] The cycle-consistent loss is as follows:
[0079]
[0080] Style similarity loss is calculated by using the mean and variance similarity of the feature maps generated by the VGG network from the constrained generated map and the reference map, as shown in the following formulas for n and m:
[0081]
[0082]
[0083] In the formula μ n,c σ represents the mean of a tensor. n,c This indicates that the variance of the tensor is calculated; VGG indicates that a pre-trained VGG network is used for feature extraction.
[0084] The total loss function is the sum of the above loss functions, as shown in equation o below:
[0085]
[0086] In the formula, λ represents the weight value of each part of the loss function, with λ1 being 10, λ2 being 5, and λ3 being 0.5.
[0087] L is calculated through forward propagation. allThe model parameters are updated through backpropagation until the objective function converges, at which point training stops.
[0088] S405: Use the trained network to perform modality transformation on images in the test set.
[0089] Input the preprocessed images from the test set into the trained network, repeat steps S401 to S403, perform mode transformation on the images in the test set, compare the transformed images with the ground truth, and test metrics such as MAE, SSIM, and PSNR.
[0090] See Figure 5 ,in Figure 5 (a) is the T1 modality image to be converted from the Brats18 database; Figure 5 (b) is the transformation result obtained using CycleGAN (published by Zhu et al. at ICCV 2017); Figure 5 (c) is the transformation result obtained using Drit (Lee et al., published at IJCV 2019); Figure 5 (d) is the conversion result obtained using the present invention; Figure 5 (e) is the true value of the T2 mode with the transformation diagram. It can be seen that the transformation result of the present invention is closer to the true result and the topology is better preserved.
[0091] See Figure 6 The first row of the schematic diagram contains T2 mode reference images with different grayscale features; the first image in the second row of the schematic diagram is the original image to be converted; the second to fifth images in the second row of the schematic diagram are corresponding T2 mode conversion images generated using the model of this invention and based on the different reference images in the first row. They belong to the T2 mode and are consistent with the grayscale attributes of the reference images.
[0092] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. An adaptive mask-guided image modality transfer method based on a recurrent generative adversarial network, characterized in that, Includes the following steps: Image partitioning: the first ratio of original images is randomly selected to form the training set, and the remaining second ratio of original images forms the test set with ground truth values. Image preprocessing involves normalizing the grayscale of all images in the training and test sets. For network construction, a recurrent generative adversarial network was chosen as the overall network architecture, and an adaptive mask network was constructed. Network training utilizes preprocessed images from the training set for network training. For quality testing, the preprocessed images in the test set are input into the trained network, modality transformation is performed on the images in the test set to obtain the transformed images, and then compared with the ground truth. The specific steps of the network training include: inputting the training set into the Unet encoder-decoder to extract features, and obtaining multi-scale feature maps at skip connections; The mask feature generation module is inserted into the skip connections of the Unet codec to modulate the multi-scale feature maps of the test set, obtaining multi-scale mask feature maps. The obtained multi-scale mask feature maps are gradually fused in the decoding stage of the Unet codec and transformed into a single-channel adaptive mask through convolution. This mask is then directly added to the original image to obtain the transformed image. The generated transformed image is used as a fake sample, and an image is randomly selected from the test set as a real sample. The discriminator is trained while the generator is fixed, so that it can judge the authenticity of the input image. Then, the discriminator is fixed, and the generator is trained so that the generated transformed image cannot be judged as real or fake by the discriminator. The above steps are repeated in the manner of training a generative adversarial network. The mask feature generation module has three inputs: the input feature map to be modulated, the original image to be converted, and the reference image; The multi-scale feature map modulation step includes the following steps: interpolating the original image to the same scale as the input vector according to the scale of the input feature map to be modulated, and then feeding it into a CNN to extract the modulation mask feature map at that scale, and obtaining the mean and variance matrices by convolving the modulation mask feature map; inputting the reference image into a pre-trained VGG network to obtain a reference feature map of the same scale, and calculating its mean and variance; using the obtained mean and variance to perform adaptive instance normalization on the mean and variance matrix; finally, performing spatial adaptive normalization on the mean and variance matrix and the modulation mask feature map to obtain the multi-scale mask feature map.
2. The adaptive mask-guided image modality conversion method based on recurrent generative adversarial networks as described in claim 1, characterized in that, The overall network architecture includes a mask feature generation module, a Unet encoder / decoder, and a PatchGAN discriminator.
3. The adaptive mask-guided image modality conversion method based on recurrent generative adversarial networks as described in claim 1, characterized in that, The training set and the test set data include two modalities. The only difference between the two modalities of the same data is the appearance grayscale, and the topological structure is consistent.
4. The adaptive mask-guided image modality conversion method based on recurrent generative adversarial networks as described in claim 1, characterized in that, The mask feature generation module performs interpolation on the original image to be transformed based on the scale of the input feature map to be modulated, and extracts features at the same scale.
5. The adaptive mask-guided image modality transfer method based on recurrent generative adversarial networks as described in claim 1, characterized in that, The transformation image is obtained by directly adding the mask to the original image.
Citation Information
Patent Citations
MRI-PET image mode conversion method and system based on cyclic generative adversarial network
CN112508775A
Multi-scale image restoration method, storage medium and terminal
CN115018727A
