Medical image synthesis method based on spatial domain and frequency domain attention mechanism

Abstract

The invention provides a medical image synthesis method based on a spatial domain and frequency domain attention mechanism, belongs to the field of computer vision and medical image generation, solves the defects of an existing medical image synthesis method in the aspect of processing high-frequency detail information, and improves the definition and detail performance of an image. The method comprises the following steps: S1, acquiring a data set and performing data preprocessing; s2, constructing a generative adversarial network; s3, inputting paired CT and MRI data sets into the designed generative adversarial network for training; s4, synthesizing an image; and S5, multiple experiments are carried out, and the synthesized image is evaluated. According to the method, the spatial domain information and the frequency domain information of the image can be extracted in the training process, the mapping relation between medical images of different modalities can be well learned, then synthesis of the medical images is accurately and rapidly completed, and a high-quality synthesized image containing lesion information is obtained.

Description

Technical Field

[0001] The present invention relates to a medical image synthesis method based on spatial domain and frequency domain attention mechanism, belonging to the field of computer vision and medical image generation. Background Art

[0002] In the process of clinical diagnosis and treatment, medical imaging is an important auxiliary tool for doctors to make accurate diagnoses and plays an indispensable role. By using different imaging techniques to scan the same part of the patient, different modality images reflecting the specific information of the region can be obtained, such as multi-contrast MRI (magnetic resonance imaging), CT (computerized tomography) and PET (positron emission tomography). However, in practical applications, high costs, physiological incompatibility, and potential radiation hazards of radioactive tracers to the human body have limited the widespread acquisition of certain modality images. Therefore, the development of cross-modality conversion technology for medical images has become an effective strategy to solve the above problems. This technology aims to generate target modality images using existing source modality images while ensuring the consistency of pathological features between the two modalities. This is not only a very challenging task, but also one of the current research hotspots.

[0003] Early methods for solving the problem of medical image generation mainly relied on manually designed rules and traditional graphics techniques, such as rendering technology based on physical models and basic texture mapping. However, these methods were limited by the computing resources and model expression capabilities at the time and could not cope with the increasingly complex needs of medical image generation. With the rapid development of deep learning technology, especially the breakthrough achievements of convolutional neural networks (CNNs) in fields such as image classification, unprecedented opportunities have been brought to the field of medical image generation. By leveraging the powerful representation ability of deep learning, the accuracy and efficiency of medical image generation have been significantly improved. Subsequently, the advent of variational autoencoders (VAEs) opened up a new way to learn data distribution in latent space by combining probability theory with neural networks. This not only enables the model to generate more diverse and natural images, but also provides a more flexible solution for medical image generation. Soon after, generative adversarial networks (GANs) achieved a major leap in image generation quality through the adversarial training mechanism between the generator and the discriminator. GANs can generate images that are close to reality and of extremely high quality, which has greatly promoted the development of image generation technology, especially in the field of medical images. On the basis of the above progress, the rise of diffusion models has added new vitality to medical image generation technology. Diffusion models demonstrate the ability to generate high-quality, high-resolution images through a novel noise addition and removal process, that is, gradually adding noise to the data and then learning how to recover the original data from the noise. It is particularly noteworthy that diffusion models perform well when processing data sets with long-tail distributions, which is particularly important for medical image generation because medical images often contain a large number of rare but important features. In addition, the introduction of the Transformer architecture has brought revolutionary changes to the development of medical image generation technology. Unlike traditional convolutional neural networks, Transformer can efficiently process sequence data and capture long-distance dependencies through the self-attention mechanism, which is particularly important when processing medical images. In medical image generation, Transformer can not only be used for efficient feature extraction, but also help the model better understand and generate the contextual information of the image through its excellent modeling ability, thereby generating more accurate and realistic medical images.

[0004] Although deep learning technology has made significant progress in medical image generation in recent years, it still faces some key issues that need to be addressed. Most existing methods have shortcomings in processing detailed textures, especially in capturing and expressing high-frequency detail information. This lack of detail information may have an adverse impact on subsequent medical diagnosis and treatment planning. To address this problem, we designed a medical image generation method based on spatial and frequency domain attention mechanisms. The model in this study first uses Transformer and ResNet modules to extract features from the input image to capture the spatial structure and semantic information of the image. Subsequently, the image is analyzed from a frequency domain perspective by combining wavelet transform with Transformer, focusing on extracting and retaining high-frequency detail information in the image. In this way, our model can not only enhance the expression of detail information in the image, but also ensure the overall quality and consistency of the generated image, thereby improving the value of medical images in clinical applications. Summary of the invention

[0005] The present invention provides a medical image synthesis method based on spatial domain and frequency domain attention mechanism, and solves the technical problem of how to achieve cross-modal conversion of medical images while ensuring that the synthesis result is close to the real target image in terms of quality and perception.

[0006] The present invention provides a medical image synthesis method based on spatial domain and frequency domain attention mechanism, comprising the steps of:

[0007] S1. Obtain the data set and perform data preprocessing, screen the images to obtain images that meet the training requirements, make paired CT and MRI data sets, and divide the data sets into training set, test set, and validation set;

[0008] S2. Construct a generative adversarial network, which includes a generator G and a discriminator D;

[0009] S3. Paired CT and MRI datasets are input into the designed generative adversarial network for training. The generator and discriminator evolve together in the alternating iterative game process, and the generator learns the data distribution of real samples.

[0010] S4, saving the trained model, inputting the two-dimensional slices in the png format of the training set into the trained model, and obtaining a synthesized CT image;

[0011] S5. Conduct multiple experiments to explore the optimal network parameters and perform quantitative and qualitative evaluations on the synthesized images.

[0012] Furthermore, obtaining the data set and performing data preprocessing in step S1 specifically includes the following contents:

[0013] Select the SynthRAD2023 brain MRI-CT dataset and the GoldAtlas male pelvis MRI-CT dataset, which are common datasets in the field of medical images, convert the medical datasets into png format, convert the image size to 256×256, and splice the CT and MRI datasets to obtain paired CT and MRI datasets;

[0014] The specific process of preprocessing is as follows:

[0015] Use the SimpleITK library to convert the original DICOM format CT data and MRI data into NIFIT format;

[0016] Use FSL tools to linearly register MRI images to CT images to obtain paired CT and MRI data;

[0017] The window width of the CT image was adjusted to 4000, the window level was adjusted to 1000, and the CT image and MRI image were normalized to between [-1,1];

[0018] The CT and MRI images were uniformly converted to 256×256 and concatenated for training.

[0019] Furthermore, the architecture of the generator G in step S2 includes an encoder, a bottleneck layer and a decoder. Specifically, the encoder includes three layers of downsampling processing, and the decoder correspondingly includes three layers of upsampling operations.

[0020] The bottleneck layer is designed in two branches. One branch uses six Resnet blocks and three Transformer blocks to extract spatial domain features. Resnet blocks and Transformer blocks can effectively capture the global dependencies of images and obtain the spatial domain global features of images. The other uses nine frequency domain attention mechanism modules, WAformer blocks, which obtain diverse frequency information through wavelet transform and use Transformer to interactively combine high and low frequency information to extract the frequency domain features of images. Finally, the fusion module SFFM is used to fuse the spatial domain and frequency domain features.

[0021] Furthermore, the discriminator D in step S2 is based on the conditional PatchGAN architecture, and the discriminator distinguishes between real images and synthetic images at the patch level of the image, thereby enhancing the sensitivity to local details related to high spatial frequency information. This approach enables the discriminator to more effectively detect subtle differences in the image, thereby improving the overall identification performance.

[0022] Furthermore, in step S3, the total loss L of the generated adversarial network is total Calculated by the following formula:

[0023] L total =λ adv L adv +λ pix L pix +λ mse L mse

[0024] Among them, L adv represents the adversarial loss between the generator G and the discriminator D, L pix is the pixel loss, L mse is the mean square error loss, λ adv , pix , mse They are the weights of adversarial loss, pixel loss, and mean square error loss respectively;

[0025] L adv , L pix and L mse The formula is as follows:

[0026] L adv =G adv +D adv

[0027] G adv =E[LOG(1-D(G(x,syn y )))]

[0028] D adv =E[logD(x)]+E[log(1-D(G(x,syn y )))]

[0029] L pix =E[||y real -G(x,syn y )||1]

[0030]

[0031] Among them, G adv represents the adversarial loss of the generator G, D adv represents the adversarial loss of the discriminator D, x represents the input MRI image, y real represents the real CT image, syn y Represents a synthetic CT image.

[0032] Furthermore, in step S5, we use indicators such as PSNR and SSIM to quantitatively evaluate network performance, and the formula is as follows:

[0033]

[0034] in, represents the real image value, represents the pixel value of the synthetic image, N represents the total number of pixels, MSE represents the mean square error, PSNR represents the peak signal-to-noise ratio, SSIM represents the structural similarity performance, μ x μ y represents the average value of x and y, C1 represents a constant, σ xy represents the covariance of x and y, C2 represents a constant, represents the variance of x, Represents the variance of y.

[0035] The experimental development environment is pytorch2.1.2. On the NVIDIA RTX 4090 graphics processor, the Adam optimizer is used, the learning rate is 0.001, the number of model training rounds is 50, and the batch size is set to 1.

[0036] The beneficial effects of the present invention are:

[0037] The present invention significantly improves the ability to capture and express high-frequency detail information in medical images. By adopting advanced imaging technology and image processing algorithms, the present invention can significantly improve the spatial resolution and contrast of medical images, making fine structures more visible, helping doctors to more accurately identify and analyze lesions, thereby improving the accuracy of disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of a medical image synthesis method based on spatial and frequency domain attention mechanisms;

[0039] Figure 2 The generator framework diagram of the present invention is

[0040] Figure 3 This is the framework diagram of the generator fusion module SFFM of the present invention

[0041] Figure 4 The difference between the synthetic image of the present invention and the real image DETAILED DESCRIPTION

[0042] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the present invention provides a medical image synthesis method based on spatial domain and frequency domain attention mechanism, comprising the following steps:

[0044] Step S1, obtain a data set and perform data preprocessing, screen the images to obtain images that meet the training requirements, make paired CT and MRI data sets, and divide the data sets into a training set, a test set, and a validation set;

[0045] In this step, we select the SynthRAD2023 brain MRI-CT dataset and the GoldAtlas male pelvis MRI-CT dataset, which are common datasets in the field of medical imaging, and preprocess the images. The preprocessing includes format unification, size unification, and color channel order unification. For images whose formats do not meet the requirements, we perform format conversion and convert the formats to PNG. We perform cropping operations according to the image processing library and crop the images to 256×256 size images to ensure that the area of ​​interest is retained. We perform channel conversion on the cropped images, and finally splice the images to create paired CT and MRI datasets, and divide the datasets into training set, test set, and validation set.

[0046] The specific processing process is as follows:

[0047] Use the SimpleITK library to convert the original DICOM format CT data and MRI data into NIFIT format;

[0048] Use FSL tools to linearly register MRI images to CT images to obtain paired CT and MRI data;

[0049] The window width of the CT image was adjusted to 4000, the window level was adjusted to 1000, and the CT image and MRI image were normalized to between [-1,1];

[0050] The CT and MRI images were uniformly converted to 256×256 and concatenated for training.

[0051] In this step, the architecture of the generator G includes an encoder, a bottleneck layer, and a decoder. Its specific structure is as follows Figure 2 Specifically, the encoder contains three layers of downsampling processing, and the image downsampling process dimension changes from 1×1×256×256 to 1×256×64×64, while the decoder correspondingly contains three layers of upsampling operations, and the image upsampling process dimension changes from 1×256×64×64 to 1×1×256×256.

[0052] The bottleneck layer is designed in two branches. One branch uses six Resnet blocks and three Transformer blocks to extract spatial domain features. Resnet blocks and Transformer blocks can effectively capture the global dependencies of the image, thereby obtaining the spatial domain global features of the image. The other uses nine frequency domain attention mechanism modules, WAformer blocks, which obtain diverse frequency information through wavelet transform, and use Transformer to interactively combine high and low frequency information to extract the frequency domain features of the image. Finally, the fusion module SFFM is used to fuse the spatial domain and frequency domain features. The specific structure of the fusion module is as follows: Figure 3 .

[0053] The discriminator D in step S2 is based on the conditional PatchGAN architecture. The discriminator distinguishes between real and synthetic images at the patch level of the image, thereby enhancing the sensitivity to local details related to high spatial frequency information. This approach enables the discriminator to more effectively detect subtle differences in the image, thereby improving the overall identification performance;

[0054] Step S3: Input the paired CT and MRI data sets into the designed generative adversarial network for training. The generator and the discriminator evolve together in the alternating iterative game process, and the generator learns the data distribution of the real samples.

[0055] In this step, the generator and the discriminator compete with each other through alternating training: first, the discriminator optimizes its own discrimination ability by distinguishing between real data and fake data generated by the generator; then, the generator deceives the discriminator by generating more realistic data, making it difficult to distinguish between real and fake. This process is repeated until the generator can generate high-quality synthetic data and the discriminator's discrimination ability reaches a balanced state. The total loss L of the generative adversarial network total Calculated by the following formula:

[0056] L total =λ adv L adv +λ pix L pix +λ mse L mse

[0057] Among them, L adv represents the adversarial loss between the generator G and the discriminator D, L pix is the pixel loss, L mse is the mean square error loss, λ adv , pix , mseThey are the weights of adversarial loss, pixel loss, and mean square error loss respectively;

[0058] L adv , L pix and L mse The formula is as follows:

[0059] L adv =G adv +D adv

[0060] G adv =E[log(1-D(G(,syn y )))]

[0061] D adv =E[logD(x)]+E[log(1-D(G(,syn y )))]

[0062] L pix =E[||y real -G(x,syn y )||1]

[0063]

[0064] Among them, G adb represents the adversarial loss of the generator G, D adv represents the adversarial loss of the discriminator D, x represents the input MRI image, y real represents the real CT image, syn y Represents a synthetic CT image. adv , pix , mse The values ​​of are 100, 10, and 1 respectively.

[0065] Step S4, saving the trained model, inputting the two-dimensional slices in the png format of the training set into the trained model, and obtaining a synthesized CT image;

[0066] Step S5, conduct multiple experiments to explore the optimal network parameters, and perform quantitative and qualitative evaluation on the synthesized images;

[0067] We use indicators such as PSNR and SSIM to evaluate the performance of the network. The formula is as follows. We also calculate the difference between the synthetic image and the real image to make a reasonable evaluation of the image.

[0068]

[0069] in, represents the real image value, represents the pixel value of the synthetic image, N represents the total number of pixels, MSE represents the mean square error, PSNR represents the peak signal-to-noise ratio, SSIM represents the structural similarity performance, μ x μ y represents the average value of x and y, C1 represents a constant, σ xy represents the covariance of x and y, C2 represents a constant, represents the variance of x, Represents the variance of y.

[0070] The quality evaluation of the images generated by the generator is shown in Table 1. The evaluation indicators include peak signal-to-noise ratio and structural similarity index. The peak signal-to-noise ratio is the human perception of the quality of the reconstructed image, and the higher the value, the better. The structural similarity index shows the overall similarity between the source image and the synthesized image. The higher the value, the more similar the source image and the synthesized image are. According to the evaluation indicators given in Table 1, it can be seen that the peak signal-to-noise and structural similarity between the synthesized image and the reference image are relatively high, indicating that the synthesized CT image is of high quality and is similar to the real CT image. At the same time, the quantitative results show that the proposed model has excellent performance and high accuracy.

[0071] Table 1 Evaluation indexes of the present invention

[0072] Evaluation Metrics for Synthetic Images SynthRAD2023 Header Dataset GoldAtlas Pelvis Dataset PSNR 26.666 29.183 SSIM 0.883 0.921

[0073] Table 2 is a comparison of the indicators of the present invention and the advanced model

[0074]

[0075] It can be seen that the CT images synthesized by the present invention are slightly higher than those of other models in terms of PSNR and SSIM.

[0076] Make a difference map between the real image and the synthetic image, such as Figure 4 As shown in the figure, and other models are selected for comparison, it can be seen that the image quality synthesized by our model is slightly higher.

[0077] In this experiment, the dataset was randomly divided into training set, test set and validation set in the ratio of 7:2:1. The experimental development environment was pytorch2.1.2. On the NVIDIA RTX 4090 graphics processor, the Adam optimizer was used, the learning rate was 0.001, the number of model training rounds was 50, and the batch size was set to 1.

[0078] The above calculation examples of the present invention are only used to explain the model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A medical image synthesis method based on spatial domain and frequency domain attention mechanism, characterized in that: The following steps are involved: S1. Obtain the data set and perform data preprocessing, screen the images to obtain images that meet the training requirements, make paired CT and MRI data sets, and divide the data sets into training set, test set, and validation set; S2. Construct a generative adversarial network, which includes a generator G and a discriminator D; S3. Paired CT and MRI datasets are input into the designed generative adversarial network for training. The generator and discriminator evolve together in the alternating iterative game process, and the generator learns the data distribution of real samples. S4, saving the trained model, inputting the two-dimensional slices in the png format of the training set into the trained model, and obtaining a synthesized CT image; S5. Conduct multiple experiments to explore the optimal network parameters and perform quantitative and qualitative evaluations on the synthesized images.

2. According to claim 1, a medical image synthesis method based on spatial domain and frequency domain attention mechanism is characterized in that: In step S1, the common data sets in the field of medical images, the SynthRAD2023 brain MRI-CT data set and the GoldAtlas male pelvis MRI-CT data set, are selected, the medical data sets are converted into png format, the image size is uniformly converted to 256×256, and the CT and MRI data sets are spliced ​​to obtain paired CT and MRI data sets.

3. According to claim 1, a medical image synthesis method based on spatial domain and frequency domain attention mechanism is characterized in that: In step S2, the architecture of the generator G includes an encoder, a bottleneck layer, and a decoder; specifically, the encoder includes three layers of downsampling processing, and the decoder correspondingly includes three layers of upsampling operations; The bottleneck layer is designed in two branches. One branch uses six Resnet blocks and three Transformer blocks to extract spatial domain features. Resnet blocks and Transformer blocks can effectively capture the global dependencies of images and obtain the spatial domain global features of images. The other uses nine frequency domain attention mechanism modules, WAformer blocks, which obtain diverse frequency information through wavelet transform and use Transformer to interactively combine high and low frequency information to extract the frequency domain features of images. Finally, the fusion module SFFM is used to fuse the spatial domain and frequency domain features.

4. According to claim 1, a medical image synthesis method based on spatial domain and frequency domain attention mechanism is characterized in that: In step S2, the discriminator D is based on the conditional PatchGAN architecture. The discriminator distinguishes between real images and synthetic images at the patch level of the image, thereby enhancing the sensitivity to local details related to high spatial frequency information; this approach enables the discriminator to more effectively detect subtle differences in the image, thereby improving the overall identification performance.

5. According to claim 1, a medical image synthesis method based on spatial domain and frequency domain attention mechanism is characterized in that: In step S3, the total loss L of the adversarial network is generated total Calculated by the following formula: L total =λ adv L adv +λ pix L pix +λ mse L mse Among them, L adv represents the adversarial loss between the generator G and the discriminator D, L pix is the pixel loss, L mse is the mean square error loss, λ adv , pix , mse They are the weights of adversarial loss, pixel loss, and mean square error loss respectively; L adv , L pix and L mse The formula is as follows: L adv =G adv +D adv G adv =E[log(1-D(G(x,syn y )))] D adv =E[logD(x)]+E[log(1-D(G(x,syn y )))] L pix =E[‖y real -G(x,syn y )||1] Among them, G adv represents the adversarial loss of the generator G, D adv represents the adversarial loss of the discriminator D, x represents the input MRI image, y real represents the real CT image, syn y Represents a synthetic CT image.

6. The medical image synthesis method based on spatial domain and frequency domain attention mechanism according to claim 1, characterized in that: In step S5, we use indicators such as PSNR and SSIM to quantitatively evaluate the performance of the network. The formula is as follows: in, represents the real image pixel value, represents the pixel value of the synthetic image, N represents the total number of pixels, MSE represents the mean square error, PSNR represents the peak signal-to-noise ratio, SSIM represents the structural similarity performance, μ x μ y represents the average value of x and y, C1 represents a constant, σ xy represents the covariance of x and y, C2 represents a constant, represents the variance of x, Represents the variance of y.

7. The medical image synthesis method based on spatial domain and frequency domain attention mechanism according to claim 1, characterized in that: In step S5, the difference between the synthetic CT and the real CT is visualized, and a reasonable qualitative evaluation of the image quality is made.

Images