A method for generating post-aortic dissection surgery imagery

Abstract

The application discloses a kind of postoperative image generation method of aortic dissection, proposes MARTF-GAN model for unpaired image generation, generates the postoperative image of patient based on preoperative image, provides reliable guidance for surgical plan making.The model generator is composed of attention encoder, RTF module, attention decoder.Attention encoder extracts relatively rich image features through global context attention;RTF module is located between attention encoder and attention decoder, and performs deep global and local information extraction and fusion on the output feature map of attention encoder, extracts high-level semantic information;The role of attention decoder is to generate the final postoperative image according to high-level semantic information, and under the guidance of attention mask, it can generate an image with relatively rich semantics.The overall performance of the method is better than that of Munit, AttGan, QsaGan and CycleGan model.

Description

Technical Field

[0001] This invention relates to a method for generating postoperative images of aortic dissection. Background Technology

[0002] Aortic dissection (AD) is a serious vascular disease caused by a rupture of the aortic intima, which, under the influence of blood pressure, separates the aorta into a true lumen and a false lumen. The presence of the false lumen makes patients prone to problems such as insufficient blood supply to the true lumen and rupture of the false lumen. Aortic treatment often involves implanting a stent into the true lumen to compress the false lumen. Postoperative follow-up imaging shows that the proximal end of the false lumen and rupture disappears, and the aortic margin appears gear-shaped. Because surgical treatment only addresses the proximal dissection, in cases where the dissection involves the descending aorta and below, the distal aorta may still be dissected. This leads to a gradually increasing incidence of distal aortic dilation and subsequent aortic rupture, making aortic dissection a high-mortality aortic disease. Studies have shown that its incidence and in-hospital mortality are increasing year by year. Figure 1 The images show the main features of different sites in the same case before and after surgery.

[0003] Doctors often use computed tomography (CT) images to plan surgeries, guide aortic cannulation, and conduct postoperative follow-up assessments. However, a single CT scan for a single patient can generate 600-800 CT slices that require manual screening, significantly reducing diagnostic efficiency. To alleviate the burden on doctors, research has begun to incorporate deep learning algorithms for automatic segmentation and detection of aortic dissections in preoperative images. Postoperative images are the primary basis for evaluating treatment effectiveness. Zhu et al. combined postoperative CT image reconstruction and hydrodynamic analysis to attempt to predict the risk of postoperative dissection expansion. Other studies utilize clinical examination data, combined with predictive models such as neural networks and logistic regression, to predict postoperative complications and avoid overtreatment. However, there are no publicly published studies on generating postoperative images from preoperative images. This invention uses a dataset of unpaired preoperative and postoperative images to train a generative model, aiming to generate corresponding high-fidelity postoperative images based on the patient's preoperative images. This provides doctors with an estimate of the postoperative recovery of the aortic region, which helps in surgical planning and predicting the risk of distal aortic enlargement. Furthermore, advancements in medical image generation algorithms have made postoperative image generation feasible.

[0004] Currently, algorithms for medical image generation are mainly based on the Generative Adversarial Network (GAN) framework. A GAN typically consists of a generator network and a discriminator network, based on a zero-sum game concept, where the generator and discriminator learn adversarially. The generator tries to deceive the discriminator by generating images that can fool it, while the discriminator tries to distinguish between generated and real images, ultimately reaching a Nash equilibrium. Pix2Pix is ​​a classic GAN-based framework that performs well in CT denoising and MR image reconstruction tasks, but it requires paired data for model training. However, differences in medical imaging equipment, contrast agent concentration, and the real-time state of the human body result in variations in medical images of the same body part acquired at different times. This makes obtaining paired medical images extremely difficult. CycleGAN addresses this problem of model dependence on paired datasets by introducing a cycle consistency loss to constrain network training and learn the mapping relationship from the source domain to the target domain. It is mainly used for modality transformation and image segmentation.

[0005] A long-standing challenge in medical image synthesis is the low resolution of regions of interest (ROIs), which can result in blurred boundaries. RegGAN, based on the "loss verification" theory, treats unpaired target images as noise labels and adaptively fits the unpaired noise distribution using a registration network. While methods that align the source and target domains through registration networks consistently and effectively improve performance, they struggle to achieve significant improvements on unpaired datasets due to the widespread noise distribution in real-world unpaired images.

[0006] The above research reveals two problems in current medical image generation: firstly, the resolution of the region of interest is low; and secondly, unpaired images are difficult to align effectively. Therefore, it is necessary to design a method for generating images after aortic dissection surgery. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method for generating postoperative images of aortic dissection. This method for generating postoperative images of aortic dissection adopts a brand-new mode and algorithm, which can obtain postoperative images that are closer to reality.

[0008] The technical solution of the invention is as follows:

[0009] A method for generating postoperative images after aortic dissection uses the MARTF-GAN model (MA stands for Multiple Attention, GAN stands for Generative Adversarial Network, and RTF stands for Residual Transformer Fusion, a fusion module) for generating unpaired images to generate postoperative images of patients based on preoperative images.

[0010] The MARTF-GAN model includes an attention encoder, an RTF module (RTF stands for Residual Transformer Fusion, which is located in the bottleneck layer between the encoder and decoder and is a two-branch feature extraction and fusion module), and an attention decoder.

[0011] (1) The attention encoder extracts image features through global contextual attention to obtain the output feature map;

[0012] (2) The RTF module is located between the attention encoder and the attention decoder. It performs deep global and local information extraction and fusion on the output feature map of the attention encoder to extract high-level semantic information.

[0013] (3) The function of the attention decoder is to generate the final postoperative image based on high-level semantic information. It can generate images under the guidance of attention mask.

[0014] The attention encoder consists of a downsampling layer structure and a global context attention module;

[0015] The downsampling layer structure consists of a 3×3 convolution, InstanceNorm, and ReLU activation function.

[0016] The structure of the global context attention module is as follows: Figure 3 As shown in section (b), the global context attention module consists of three parts: global average pooling, feature transformation, and feature fusion. The global average pooling operation is implemented using a 1x1 convolution and a softmax activation function. The feature transformation uses a bottleneck structure consisting of a 1x1 convolution, layer normalization (LayerNorm), and a ReLU activation function. Feature fusion employs a broadcast mechanism to perform element-wise addition.

[0017] The attention decoder consists of a content branch and an attention branch. The content branch is used to compute the content mask, while the attention branch focuses on the main information in the image construction process. The generated attention mask is used to guide the generation of an image by fusing the feature content mask of the input.

[0018] The generator takes a grayscale image with dimensions of 1×256×256 as input (in 1x256x256, the first '1' indicates the number of channels, and 1x256x256 represents a single-channel grayscale image; color...) Figure 1Typically, the input image is a three-channel 3x256x256 matrix (meaning it consists of three 256x256 matrices). It passes through a convolutional block to generate a 64×256×256 feature map. This convolutional block performs reflection padding on the boundaries, preserving boundary information while ensuring the output feature map is the same size as the input image. Then, a 7×7 kernel is used for convolution operations to retain as much of the input image's feature information as possible.

[0019] The downsampling layer structure consists of a 3×3 convolution, InstanceNorm, and ReLU activation function. Assuming the input dimension of downsampling is C×H×W, the output dimension of the downsampling feature map is 2C×H / 2×W / 2. Global contextual attention is introduced after each downsampling layer to extract more diverse features without changing the feature map size. After two downsampling layers and attention, the output feature dimension is 256×64×64. The RTF module adopts a structure of parallel residual convolutional blocks and transformer blocks to fuse global and local features and reconstruct the feature dimension to 256×64×64 using feature compression.

[0020] The decoding layer employs two branches: one for calculating the attention mask and the other for calculating the content mask. Both branches use deconvolution to upsample the features, resulting in an output feature dimension of 64×256×256 after two upsampling operations. Finally, convolution is used to compress the feature dimension, and the content mask is concatenated with the model input. Guided by the attention mask, image reconstruction is completed, yielding an output result of size 1×256×256.

[0021] The processing procedure for the global context attention block is as follows:

[0022] First, a 1x1 convolution is used to process the downsampled feature map. Generate an attention mask Then, after dimension swapping (first, multiplying H and W of Ma to get 1xH1W1, then swapping two positions to get W1H1x1, and finally performing a dimension increase operation to get W1H1x1x1), we obtain... f down Dimensional exchange (f down Multiplying H and W together, we get C1xH1W1. Then, contextual features are obtained under the guidance of attention masks.

[0023] f down With m a Perform matrix multiplication to obtain f context , because m aThis is the weight matrix generated after a 1x1 convolution and softmax activation function. Its weights guide the network to focus its attention on a specific region of the image. The bottleneck transform (BT) module is used to transform features and capture the dependencies between channels. The dependency is explained as follows: because the input to this module is CxHxW, where C represents the number of channels (which can also be understood as image features), there should be some relationship between the image features; they are not isolated. Therefore, these features are extracted. This dependency is the global context feature. Subsequently, fusion is performed. Finally, the global context feature f is... context with f down Fusion is performed; fusion refers to adding corresponding positions of the feature maps.

[0024] f down The dimensions are C1xH1xW1, and the dimensions of Fcontext are C1x1x1. First, f will be broadcast... context Dimension becomes with f down The same process is repeated, and then the elements at corresponding positions are added together to obtain the final output.

[0025] Let Φ denote a 1×1 convolution, σ denote the Softmax activation function, and LN denote LayerNorm, which is layer normalization. Representing matrix multiplication, ⊕ indicates element-wise addition, and global context attention can be represented as:

[0026] m a =σ(Φ(f) down ))

[0027]

[0028] f out =f down ⊕Φ(ReLU(LN(Φ(f context ReLU is an activation function.

[0029] This invention, from the perspective of feature extraction, relies on cycle consistency loss to improve the extraction performance of key features in the source domain and refine feature fusion methods to enhance the quality of generated images. Therefore, this work proposes a MARTF-GAN model for unpaired image generation. It learns the mapping from the source domain to the target domain to generate new medical image data and applies it to downstream tasks.

[0030] Beneficial effects:

[0031] Postoperative complications such as distal aortic enlargement are common after aortic dissection. Postoperative images can be used to analyze postoperative outcomes and patient recovery. Currently, no research focuses on the generation of postoperative images. This invention proposes a MARTF-GAN model for generating unpaired images, which generates postoperative images of patients based on preoperative images, providing reliable guidance for surgical planning. The model consists of an attention encoder, an RTF module, and an attention decoder. The attention encoder extracts rich image features through global contextual attention; the RTF module, located between the attention encoder and the attention decoder, performs deep global and local information extraction and fusion on the output feature map of the attention encoder to extract high-level semantic information; the attention decoder generates the final postoperative image based on the high-level semantic information, and can generate semantically rich images under the guidance of attention masks. This invention constructs a dataset consisting of 2199 pairs of unpaired preoperative and postoperative images. On this dataset, Wasserstein, FID, and MMD distance metrics were calculated between the generated image set and the postoperative image set in the test set. The overall performance of this model outperformed the Munit, AttGan, QsaGan, and CycleGan models, achieving improvements of 2.0748, 3.1144, and 0.1808 respectively compared to CycleGan, indicating that this model more closely approximates real postoperative images. To further objectively evaluate the model's performance, it was tested on the publicly available BraTS (2018) dataset, achieving the best performance compared to other models, demonstrating its generalization ability.

[0032] The main contributions of this invention are as follows:

[0033] A novel unpaired image generation model, MARTF-GAN, is proposed. This model can generate detailed images of the aorta in postoperative images based on preoperative images, demonstrating certain clinical value.

[0034] An RTF module was designed. This module is used for the fusion of global and local features in the generator, making more comprehensive use of the semantic information in the preoperative images.

[0035] It integrates multiple attention mechanisms. An attention encoder is used to preserve global information in preoperative images; an attention decoder guides the generation of images focused on the lesion area.

[0036] Extensive experiments were conducted on pre- and post-operative image datasets and publicly available datasets for aortic dissection. Experimental results show that the method of this invention achieves better lesion generation results compared to existing methods. Attached Figure Description

[0037] Figure 1These are images of aortic dissection taken from five sites before and after surgery in the same case. (a)-(c) are sections of the proximal dissection; (d)-(e) are sections of the distal dissection.

[0038] Figure 2 This is a diagram of the overall training framework and network overview of MARTF-GAN;

[0039] Figure 3 It is a MARTF-GAN generator;

[0040] Figure 4 It is a module that combines residual and Transformer;

[0041] Figure 5 It is the MARTF discriminator network structure; S represents the convolution stride, and P represents padding.

[0042] Figure 6 This is a dataset display. (a) is the pre- and post-operative image dataset for aortic dissection; (b) is the BraTS (2018) dataset.

[0043] Figure 7 Comparison of postoperative images of aortic dissection generated from various models;

[0044] Figure 8 The generation results of each model on the BraTS (2018) dataset are displayed;

[0045] Figure 9 This is a thermal diagram of the ablation model. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0047] Example 1:

[0048] The challenge of unpaired image-to-image generation lies in the fact that it is an unsupervised training mode, and the learned X-to-Y mapping contains countless possibilities. To enable the generated images to have more refined lesion details, this invention proposes the MARTF-GAN model for generating pre- and post-operative images of unpaired aortic dissection. It aims to fully extract semantic information from the images to maintain the realism of medical images. The training framework and model architecture are as follows: Figure 2 As shown. In this invention, A and B represent the source domain and the target domain, respectively, and G... AB and G BA For two generators in opposite directions, D A D B Let x∈A and y∈B be the discriminators that distinguish between images in domains A and B, respectively. Let x∈A and y∈B be the images in the corresponding domains, which can also be represented as x~p. data (x), y~pdata (y). Simultaneously, Y' = G AB (x) represents the image transformed from the image in domain A; X' = G BA Let (y) represent the image transformed from the image in domain A. By feeding X' and Y' into generator G, we obtain y' = G. AB (X') and x' = G BA (Y'). During training, a pair of images (x, y) are input and used for G. AB G BA D A D B The training utilizes the cycle consistency loss L cycle Ensure the stability of the generated image structure. Adversarial loss L adv This allows the distribution of the generated images to continuously approximate the distribution of the training images. cycle and L adv The loss function expression is as follows:

[0049]

[0050]

[0051] The MARTF-GAN generator employs a multi-attention structure, consisting of an attention encoder (EA) and an attention decoder (DA). It primarily introduces global contextual attention after downsampling in the encoder to supplement the network's semantic information, and introduces an attention branch in the decoder to guide image synthesis. The generator's bottleneck layer is designed as a dual-branch structure, combining the advantages of Transformer and residual blocks to construct a feature fusion module to compensate for the limitation of convolutional operations focusing only on local areas. The discriminator uses a fully convolutional (FCN) structure, patchGAN, and evaluates the generated images using an N2×N2 matrix. This improves the discriminator's focus on specific regions, enabling the generator to produce more detailed images.

[0052] Generator Design

[0053] like Figure 3As shown, the generator of MARTF-GAN (a generative adversarial network consisting of a generator and a discriminator) for unpaired image generation adopts an encoder-decoder structure. The encoder extracts features from the input image. The decoder's main role is to reconstruct an image similar to the target domain based on the encoded features. Therefore, the key to image generation lies in the extraction of encoded features and minimizing the loss of key information during decoding. MARTF-GAN introduces multiple attention mechanisms (MA), consisting of an attention encoder and an attention decoder. The attention encoder combines downsampling with global contextual attention in the encoder to extract richer semantic information. The attention decoder consists of a content branch and an attention branch. The attention branch focuses on the main information during image construction, and the generated attention mask is used to guide the generation of an image by fusing the input feature content mask. In addition, an RTF module is proposed, which is designed as a dual-branch structure, located as a bottleneck layer between the encoder and decoder. This module is mainly used to fuse the image information extracted from the residual block and the Ttransformer encoded block to enhance the extraction of encoded features.

[0054] The generator takes a 1×256×256 grayscale image as input and passes it through a convolutional block to generate a 64×256×256 feature map. This convolutional block performs reflection padding on the boundaries, preserving boundary information while ensuring the output feature map is the same size as the input image. Then, a 7×7 convolutional kernel is used for convolution operations; this large convolution operation preserves as much of the input image's feature information as possible. The downsampling layer structure is as follows... Figure 3 As shown in (a), the feature map consists of a 3×3 convolution, InstanceNorm, and ReLU activation function. Assuming the input dimension of the downsampling is C×H×W, the output dimension of the downsampling is 2C×H / 2×W / 2. Global contextual attention is introduced after each downsampling layer to extract more diverse features without changing the feature map size. Therefore, in this invention, after two downsampling layers and attention, the output feature dimension is 256×64×64. The RTF module adopts a structure of parallel residual convolutional blocks and transformer blocks, fusing global and local features and reconstructing the feature dimension to 256×64×64 using feature compression. The decoding layer uses two branches: one for calculating the attention mask and the other for calculating the content mask. Both use deconvolution operations for feature upsampling, and after two upsampling operations, the output feature dimension is 64×256×256. Finally, convolution is used to compress the feature dimension, the content mask is concatenated with the model input, and under the guidance of the attention mask, the image reconstruction is completed, resulting in an output of size 1×256×256.

[0055] Attention Encoder

[0056] The encoder of this invention uses convolution for feature extraction while downsampling. However, convolution can only establish pixel relationships in local regions and capture local information of the image. To improve the model's long-distance dependencies while downsampling, this invention introduces a global contextual attention block after downsampling, forming the attention encoder block, as shown below. Figure 3 As shown in the dashed box on the left. Global context attention is as follows. Figure 3 As shown in (b), a 1×1 convolution is first applied to the downsampled feature map. Generate an attention mask Then, after dimension transformation, we get... f down Dimensional exchange Then, contextual features are obtained under the guidance of attention masks. Feature transformations are performed using the bottleneck transform (BT) module to capture dependencies between channels. Finally, the global context features f are transformed. context with f down The components are fused together to obtain the final output. In this invention, Φ represents 1×1 convolution, σ represents the Softmax activation function, and LN represents LayerNorm. Representing matrix multiplication, ⊕ indicates element-wise addition, and global context attention can be represented as:

[0057] m a =σ(Φ(f) down ))

[0058]

[0059] f out =f down ⊕Φ(ReLU(LN(Φ(f context ))))

[0060] Residual and Transformer Fusion Module

[0061] The Residual Transformer Fusion (RTF) module is located in the bottleneck layer between the encoder and decoder, and is a two-branch feature extraction and fusion module. For example... Figure 4 As shown, the upper branch consists of N residual network blocks, focusing on the extraction of local features; the lower branch mainly consists of L Transformer encoding blocks, used to extract global information dependencies. Let the input feature map of the RTF module be... The output feature map is

[0062] The Residual Block (RB) first performs a 3×3 convolution operation on the input feature map after reflection padding to fully capture the edge information of the feature map. Then, it uses Instance Norm and ReLU activation functions to improve the performance of style transfer. The residual block structure is as follows: Figure 4 As shown in (a), the general formula for the operation of the residual block can be expressed as: in These represent the input and output of the i-th residual block, respectively.

[0063]

[0064] Therefore, the final output of the upper branch local feature extraction network after passing through N residual blocks is:

[0065] Due to computational performance limitations, the transformer coding block expects a smaller input feature map size. Therefore, it is first necessary to optimize the input I of the RTF module. RTF Feature maps are obtained by performing downsampling (DS). f m Flattening the feature dimensions into one dimension yields Then, by permuting the dimensional position... Position encoding is used for embedding. Position encoding is achieved through a linear mapping. Embedded into the flattened vector to obtain To ensure that the output feature dimension in the transformer encoding block is sufficient to reconstruct the input feature dimension, further feature dimensionality reduction is required. This is achieved by using a linear layer to remap the feature dimension back to 384, i.e. Next, the cascaded transformer coding blocks are processed by multi-head attention (MSA) and a multilayer perceptron (MLP) to analyze f. e′ Processing is then performed. Using LN to denote LayerNorm, the general formula for a Transformer coded block can be expressed as:

[0066]

[0067]

[0068] in Indicates intermediate output. Let represent the output and input of the i-th transformer coded block, and have .

[0069]

[0070] The output after L cascaded transformer coding blocks is Before fusing with local features, the reverse operation of feature processing before the input transformer encoding layer is performed. Perform Patch Flattening (PF), then upsample to the same level. Equal dimensionality. The `concat` operation is used to add channel dimensions, enriching the features that describe the image itself. Channel compression, represented as CC, is a combination of 1×1 and 3×3 convolution operators, aiming to reduce the feature dimension to [missing value]. This invention uses This indicates the concat operation; the final output of the RTF module is O. RTF It can be represented as:

[0071]

[0072] attention decoder

[0073] like Figure 3 The dashed box on the right shows the attention decoder module, which consists of an attention branch (AB) and a content branch (CB). Both use the same deconvolutional blocks for upsampling; the only difference is the number of channels in the output feature during the last convolutional operation of each branch. Assume the model's input image is... The number of output channels for the attention branch is C a If the attention mask is used, then the number of output channels for the content branch is C. c =C a -C2 content mask. This invention inputs the model image. The content mask is added to the channel mask, and then the final output image is constructed under the guidance of the attention mask. This calculation process can be represented as:

[0074]

[0075] Discriminator

[0076] The discriminator uses a PatchGAN-based structure, such as... Figure 5 As shown. This network is a fully convolutional structure, and the final output is an n×n matrix. Each value in the matrix corresponds to a receptive field region of the input image, called a patch. In this invention, the discriminator input is... Except for the first convolutional downsampling block, which does not use regularization, the other three downsampling blocks consist of convolution, regularization, and LeakyReLU, with the last downsampling layer having a stride of 1. Finally, a 4×4 convolution is used to reduce the number of channels to 1, outputting a 16×16 matrix.

[0077] Experiment and Results Analysis

[0078] Dataset

[0079] This invention selects two medical image datasets to evaluate the proposed MARTF-GAN. Details of the datasets are as follows:

[0080] The preoperative and postoperative image dataset for aortic dissection consists of CT images, each 512×512 pixels. It comprises preoperative and postoperative images from 78 patients collected by the Second Xiangya Hospital of Central South University. A total of 2199 unpaired preoperative and postoperative image pairs were selected, mainly including sections of the aortic arch and ascending and descending aorta. Figure 6 (a) The area within the dashed box represents the most important postoperative image feature. After stent implantation, the original aortic dissection will appear as a gear-like structure, and the false lumen and tear in the proximal vessel will disappear. In this experiment, the training set, validation set, and test set contained 1540, 330, and 329 image pairs, respectively.

[0081] The BraTS (2018) dataset is a publicly available dataset containing MRI images of 240×240 pixels. Figure 6 As shown in (b). The dataset of this invention underwent preprocessing, taking 68 images each from the T1 and T2 modalities of each patient, totaling 6528 pairs of paired image data. During training, 6x cross-validation was used, with the training set containing 4352 image pairs, the validation set containing 1088 image pairs, and the test set containing 1088 image pairs.

[0082] Training setup and evaluation metrics

[0083] To ensure fairness, this invention maintained a consistent experimental environment. Hyperparameter settings were also kept identical. The best-performing model on the validation set across 80 training epochs was selected for evaluation. During training, the target domain image index was randomized to avoid using fixed image pairs for training, and the model was trained with unpaired images. Normalized mean error (NMAE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) were selected as evaluation metrics to calculate the model's generative performance on the brain MRI image dataset. To avoid inflated results, image backgrounds were excluded from the calculation. In addition, based on Xu et al.'s research on GAN network evaluation metrics, this invention also selected Wasserstein distance, Fréchet Inception Distance (FID), Kernel Maximum Mean Discrepancy (MMD), and The 1-Nearest Neighbor classifier (1-NN) in the Resnet43 convolutional feature space to evaluate the distance between the generated image set and the target domain image set in the aortic dissection dataset.

[0084] Experimental results

[0085] This invention trains MARTF-GAN, baseline models CycleGan and Munit, and the latest AttentionGan and QsaGan models on unpaired image generation datasets of brain MRI images and pre- and post-operative images of aortic dissection. The generation performance of MARTF-GAN is compared with that of the other four models.

[0086] 1.1.1 Experimental Results of Pre- and Post-operative Image Datasets for Aortic Dissection

[0087] Since the test set of the aortic dissection dataset lacked perfectly paired preoperative and postoperative images, this experiment evaluated the model's generation performance by calculating the distance score between the generated postoperative image set and the postoperative image set in the test set. A smaller score indicates that the generated images are closer to the real dataset. As shown in Table 1, the proposed model had the smallest Wasserstein distance, indicating better model training. A small MMD distance indicates that the image data distribution generated by the proposed model is closer to the real image data distribution. Although 1-NN is slightly inferior to AttGan, MARTF-GAN achieved a lower FID score, indicating that it can generate images with higher quality and diversity.

[0088] Table 1 compares the distance metrics between the image sets generated by each model and the target domain image set on the aortic dissection dataset.

[0089]

[0090] Figure 7 The presented models demonstrate their generation performance on an aortic dissection dataset. This invention focuses on generating the aortic region while maintaining as much structural similarity as possible to the preoperative images. From Figure 7 As can be seen, the CycleGan and QsaGan models generate images with poor clarity. The Munit model generates gears that are closer to the actual images, but its generation lacks diversity, resulting in some postoperative images that completely deviate from the preoperative images, such as... Figure 7 (a3) and (d3). AttGan generates relatively realistic aortic arch slices, but its postoperative image generation performance for ascending and descending aortic slices is very poor, such as... Figure 7 (a5). Our proposed MARTF-GAN model performs significantly better than other models in generating images of the aorta shown in the red box. It can correctly generate postoperative images of the corresponding target, and the gear-shaped traces of the postoperative stent in the target area are more obvious. The false lumen and rupture of the proximal vessel disappear, which is very close to the actual image.

[0091] 1.1.2. Experimental Results of BraTS(2018) Dataset

[0092] To verify the generalization performance of the proposed model, we scrambled the target domain image indices on a public dataset by randomizing them to avoid using fixed image pairs for training. We trained MARTF-GAN on unpaired images alongside other unpaired image generation models. Since the test set in the BraTS2018 dataset consists of paired images, we calculated MAE, PSNR, and SSIM. Table 2 shows the performance of each model after 6x cross-validation; the data are the averages of 6 experiments, and the values ​​in parentheses are the overall standard deviation of the samples. It can be seen that MARTF-GAN, while maintaining structural similarity, reduced MAE to below 0.08, a decrease of 0.0054 compared to CycleGan; and improved the peak signal-to-noise ratio to above 25, an improvement of 0.7847 compared to the baseline model CycleGan. Figure 8 The presentation showcases some representative results from brain images generated by five different models. T1 is the original image, T2 is the corresponding source domain image, and the model input is image T1, generating an image as similar as possible to T2. Figure 8 The results show that AttGan maintains high realism in edge information, but its effect on lesion region generation is average. QsaGan generates lesion regions well, but the edges in the first two images are relatively blurry. Our proposed MARTF-GAN outperforms other models in generating key areas marked by the dashed red box, while also preserving edge information well.

[0093] Table 2 Comparison of image quality evaluation metrics generated by different models on the BraTS 2018 dataset

[0094]

[0095] ablation experiment

[0096] The ablation experiments were primarily conducted to verify the roles of each component in the proposed model. As shown in Table 3, the baseline model used was Cyclic GAN. The addition of modules was indicated by a '+' sign, such as Baseline+EA, which adds an attention encoder to the baseline model. Base+EA+RTF+DA is the MARTF-GAN model proposed in this invention. The experimental results in Table 3 show that EA comprehensively improved all three metrics. The RTF module and the DA model both contributed to MAE and PSNR.

[0097] Table 3 Ablation experimental data of MARTF-GAN

[0098]

[0099] To clearly illustrate the focus of each component, the model's heatmap was visualized. For example... Figure 9 As shown, the attention encoder (EA) focuses on a wider range than the baseline model. The addition of the RTF module allows for the fusion of local and global features, while still concentrating on the lesion site. After adding the attention encoder (DA), the attention across the entire lesion contour is more significant, enabling the model to focus more on the generation of the lesion site.

[0100] Table 4 Analysis of Feature Fusion Methods and Number of Coding Blocks

[0101]

[0102]

[0103] For the feature fusion module, this invention analyzes the fusion method and the number of branch coding blocks, and experiments demonstrate the impact of the number of coding blocks and the fusion method on model performance. As shown in Table 4, RTF6-3 indicates that the number of residual blocks N=6 and the number of Transformer coding blocks L=3 in the RTF module. The experimental results show that increasing the network depth can improve performance. The Add operation of the feature fusion method does not change the feature dimension, which is equivalent to increasing the amount of feature information describing the image; while the concatenate operation (concat) adds along the channel dimension, which is equivalent to increasing the number of features. Experimental results show that the concatenate fusion method can more fully preserve global and local features.

[0104] In this work, the applicant proposes a MARTF-GAN network model for generating unpaired images. The proposed method can generate postoperative images of aortic dissection with good performance. In quantitative experiments, the proposed model also outperforms current mainstream attention-based GAN networks. Furthermore, the generalization performance of the MARTF-GAN model is demonstrated on the publicly available BraTS (2018) dataset. This work has initially explored the possibilities of postoperative image generation. In the next work, we will further combine aortic vessel diameter, blood flow pressure, and other clinical data that may cause postoperative complications to achieve more refined postoperative generation of aortic sites, enabling the model-generated postoperative images to play a greater role in clinical practice.

Claims

1. A method for generating postoperative images of aortic dissection, characterized in that, The MARTF-GAN model was used to generate postoperative images of patients based on preoperative images. The MARTF-GAN model includes an attention encoder, an RTF module, and an attention decoder; (1) The attention encoder extracts image features through the global context attention module to obtain the output feature map; (2) The RTF module is located between the attention encoder and the attention decoder. It performs deep global and local information extraction and fusion on the output feature map of the attention encoder to extract high-level semantic information. (3) The function of the attention decoder is to generate the final postoperative image based on high-level semantic information. It can generate images under the guidance of the attention mask. The model takes a grayscale image with dimensions of 1×256×256 as input, and generates a feature map with dimensions of 64×256×256 through a convolutional block. This convolutional block performs reflection padding on the boundaries to preserve boundary information while ensuring that the output features are the same size as the input image. Then, a convolution operation is performed using a convolutional kernel of size 7×7 to preserve the feature information of the input image as much as possible. The downsampling layer structure consists of a 3×3 convolution, InstanceNorm, and ReLU activation function. Assuming the input dimension of the downsampling layer is C×H×W, the output dimension is a feature map of 2C×H / 2×W / 2. A global context attention module is introduced after each downsampling layer to extract more diverse features without changing the feature map size. After two downsampling layers and the global context attention module, the output feature dimension is 256×64×64. The RTF module uses a parallel structure of residual convolutional blocks and transformer blocks to fuse global and local features, and reconstructs the feature dimension to 256×64×64 using feature compression. The attention decoder employs two branches: one for calculating the attention mask and the other for calculating the content mask. Both branches use deconvolution to upsample the features, resulting in an output feature dimension of 64×256×256 after two upsampling operations. Finally, convolution is used to compress the feature dimension, and the content mask is concatenated with the model input. Guided by the attention mask, image reconstruction is completed, yielding an output result of size 1×256×256. The processing procedure of the global context attention module is as follows: First, a 1×1 convolution is applied to the downsampled feature map. Generate an attention mask ， Then, after dimension transformation, we get... ; will f down Dimensional exchange Then, contextual features are obtained under the guidance of attention masks. The bottleneck transform module is used to perform feature transformations to capture the dependencies between channels; finally, the global context features are transformed. f context and f down The components are fused together to obtain the final output. Let Φ denote a 1×1 convolution, σ denote the Softmax activation function, and LN denote LayerNorm. Represents matrix multiplication. To represent the sum of corresponding elements, the global context attention module can be represented as: m a =σ(Φ(f down ))

Images