An image synthesis method based on transformer-GAN model and its application

Abstract

The present invention relates to an image synthesis method and application based on a transformer-GAN model, and relates to the field of artificial intelligence image generation. The method comprises: S1, image acquisition: acquiring multiple imaging sequences; S2, data processing: using the medical image registration module in the 3D Slicer software to perform image registration on the imaging sequences, completing image cropping and brightness unification operations, and obtaining an image pair data set; S3, model training: constructing a multi-head self-attention generative adversarial network model, and deploying it on a server, training the image pair data set to obtain an optimal training model; S4, image synthesis: using the optimally trained multi-head self-attention generative adversarial network model to perform image synthesis. The application is the application in the identification of vascular stenosis and plaques in intracranial atherosclerotic diseases. The present invention provides a more convenient, low-cost, and high-recognition image synthesis method for the treatment of intracranial atherosclerotic diseases based on an image transformation model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence image generation, and specifically relates to a method and application of image synthesis based on the transformer-GAN model. Background Art

[0002] Intracranial atherosclerotic disease (ICAD) is the leading cause of ischemic stroke worldwide, accounting for 46.6% of acute cerebral ischemia in my country. ICAD is associated with high rates of stroke recurrence, dementia, and Alzheimer's disease, leading to cognitive decline in patients and significantly increasing their economic burden.

[0003] Chinese patent CN118711795A (publication date: September 27, 2024) discloses a method for identifying intracranial vascular characteristics of cerebral atherosclerotic disease. By obtaining the basic lipid data of the pathological sample of the blood vessel to be tested, the basic lipid data is input into the trained recognition model to obtain the target identification result. From the perspective of lipidomics, a random forest training model is used to identify the donor and recipient blood vessels of cerebral atherosclerotic disease. This invention only targets the specific lipid metabolism abnormalities in the vascular tissue of ICAD patients, and lipidomics data needs to be obtained.

[0004] Vascular stenosis and plaque identification are key diagnostic indicators for intracranial coronary artery disease (ICAD). Early detection of vascular stenosis and intracranial plaques, as well as accurate assessment of plaque composition, are crucial for the clinical treatment and prevention of stroke. High-resolution magnetic resonance intracranial vessel wall imaging (HR-IVW) is a non-invasive, radiation-free examination method characterized by high resolution and a high signal-to-noise ratio. It is an important tool for diagnosing intracranial arterial stenosis and plaque analysis.

[0005] HR-IVW examination includes time-of-flight magnetic resonance angiography sequence (hereinafter referred to as TOF-MRA sequence), high-resolution vascular wall imaging sequence (hereinafter referred to as HR-IVW sequence), transverse relaxation-weighted fast spin echo imaging sequence (hereinafter referred to as T2 FSE sequence) and other sequences for locating blood vessels. It requires a 3.0T or higher MRI scanner equipped with a special coil, uses three-dimensional sequence imaging, and reconstructs and evaluates vascular conditions based on a post-processing workstation. The equipment requirements are high and the operating environment is strict, and many medical institutions do not have these conditions. The HR-IVW examination has a long scanning time and requires the patient to remain still during the examination, which is prone to motion artifacts, resulting in unusable images, and the examination is expensive.

[0006] For example, Chinese patent CN117174262A (publication date: December 5, 2023) discloses a method for establishing a computer-accurate simulation imaging model of stroke. CTP and CTA examinations of the head and neck are performed on selected patients. The three-dimensional reconstruction software MIMICS is used to analyze the infarct core area and ischemic penumbra of stroke patients, and accurately measure the corresponding volume and density. The imaging hemodynamic assessment lays the foundation for the study of hemodynamic changes in intracranial arteriosclerosis.

[0007] How to complete image generation between different scanning instruments or scanning sequences based on image conversion technology, reduce the number of scans, thereby reducing the time and money costs of inspection and reducing image artifacts requires further research. Summary of the Invention

[0008] To solve the above problems, the present invention provides an image synthesis method and application based on the transformer-GAN model, the method comprising the following steps:

[0009] S1, image acquisition: acquiring multiple imaging sequences;

[0010] S2. Data processing: Use the medical image registration module in 3D Slicer software to perform image registration on the imaging sequence, complete image cropping and brightness uniformity operations, and obtain an image pair dataset;

[0011] S3. Model training: Build a multi-head self-attention generative adversarial network model and deploy it on the server. Train it on the image pair dataset to obtain the optimal training model.

[0012] S4. Image synthesis: Image synthesis is performed using the optimally trained multi-head self-attention generative adversarial network model.

[0013] Specifically, the imaging sequence in step S1 uses a nuclear magnetic resonance scanner to perform a brain scan, directly acquires a TOF-MRA sequence, a HR-IVW sequence, and a proton density weighted image sequence, and extracts a T2 FSE sequence through the proton density weighted image sequence.

[0014] Specifically, step S2 includes the following steps:

[0015] S2.1. Data registration and image cropping: The TOF-MRA sequence was registered with the T2 FSE sequence and the HR-IVW sequence, and the cerebral vascular region was cropped to obtain the TOF-MRA to HR-IVW image pair dataset and the TOF-MRA to T2FSE image pair dataset.

[0016] S2.2. Brightness uniformity: Adjust the brightness values ​​of T2 FSE images and HR-IVW images to achieve overall uniformity.

[0017] S2.3. Dataset division: The two image pair datasets are randomly divided into a training set, a validation set, and a test set.

[0018] Specifically, step S3 includes the following steps:

[0019] S3.1. Model Construction: Construct a multi-head self-attention generative adversarial network model, including a generator and a discriminator; set the transformer-GAN model generator initialization training times, training data batch size, and model weight update pace parameters;

[0020] S3.2. Data training: Train the generators for synthetic HR-IVW images and synthetic T2 FSE images respectively;

[0021] Furthermore, the generator data training for synthesizing HR-IVW images includes the following steps:

[0022] S3.2.1. Data collation: The TOF-MRA sequences in the training set of the TOF-MRA to HR-IVW image pair dataset were used as the initial image data, and the HR-IVW sequences in the same image pair were used as the real image data.

[0023] S3.2.2, Image synthesis: The initial image data of the TOF-MRA sequence is passed into the generator to obtain the synthesized HR-IVW sequence;

[0024] S3.2.3. Image discrimination: The real image data of the synthesized HR-IVW sequence and the TOF-MRA sequence before synthesis in the same image pair are fed into the discriminator, a loss function is set, and the loss function value is output;

[0025] S3.2.4. Model tuning: Based on the loss function value, perform gradient descent and iterate optimization multiple times until the loss function value is minimized and converges, obtaining the optimal generator for synthesizing HR-IVW images;

[0026] S3.2.5. Model validation: Use the TOF-MRA sequence in the validation set of the TOF-MRA to HR-IVW image pair dataset as the initial image data, and the HR-IVW sequence in the same image pair as the real image data. Repeat steps S3.2.2 and S3.2.3 to calculate whether the loss function converges.

[0027] The same steps are taken for the TOF-MRAto T2 FSE image dataset to obtain the optimal generator of synthetic T2 FSE sequences;

[0028] Specifically, the data training of the generator for synthesizing T2 FSE images is consistent with steps S3.2.1-S3.2.5 to obtain the optimal generator for synthesizing T2 FSE images, wherein the TOF-MRA sequence in the training set of the TOF-MRAto T2 FSE image pair data set is the initial image data, and the T2 FSE sequence in the same image pair is the real image data.

[0029] The generator in step S3 includes an encoder, nine aggregated residual transformers, and a decoder, which are used to extract input image features and generate a synthesized image. The encoder consists of three convolutions and their corresponding activation functions; the aggregated residual transformer includes sampling, a multi-head attention mechanism, and convolution; and the decoder includes two transposed convolutions, one convolution, and an activation function.

[0030] Specifically, the discriminator in step S3 is responsible for determining whether the input image is a synthetic image or a real image, and is composed of four convolution and activation functions.

[0031] Specifically, step S4 includes: inputting the TOF-MRA sequences in the test sets divided from the TOF-MRA to HR-IVW image pair dataset and the TOF-MRA to T2FSE image pair dataset into the generator trained in step S3, respectively, to obtain synthesized HR-IVW images and T2 FSE images.

[0032] Specifically, the TOF-MRA sequence selected for image cropping in step S2.1 contains brain blood vessels, as do the HR-IVW sequence and the T2FSE sequence, and the image elements are aligned during cropping.

[0033] An image synthesis method based on the transformer-GAN model is applied to the identification of vascular stenosis and plaques in intracranial atherosclerotic diseases.

[0034] Compared with the prior art, the technical solution of this application has the following advantages and effects:

[0035] 1. The present invention provides an image synthesis method based on the transformer-GAN model. It only uses TOF-MRA sequence, HR-IVW sequence, and T2 FSE sequence to obtain the optimal training image conversion technology transformer-GAN model, avoiding the traditional HR-IVW examination that requires multiple sequences obtained by 3.0T and above nuclear magnetic resonance scanners. It needs to scan nine sequences such as transverse relaxation weighted image sequence and longitudinal relaxation weighted image sequence, which reduces the equipment requirements for nuclear magnetic resonance scanners, avoids motion artifacts, and reduces scanning time and costs.

[0036] 2. The present invention provides an application of an image synthesis method based on the transformer-GAN model. Based on the trained transformer-GAN model, the synthesized HR-IVW images and synthesized T2 FSE images obtained can be used to identify vascular stenosis and plaques in intracranial atherosclerotic diseases, providing a more convenient, effective and high-recognition image acquisition method for the treatment of intracranial atherosclerotic diseases.

[0037] The above is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings.

[0038] Based on the detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without inventive work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0040] Figure 1 This is a flowchart of an image synthesis method based on a transformer-GAN model according to an embodiment of the present invention;

[0041] Figure 2 is a model training flow chart of an embodiment of the present invention, wherein Figure 2(a) is the overall flow chart of transformer-GAN model training, Figure 2(b) is the encoder flow chart, Figure 2(c) is the aggregated residual transformer (ART) flow chart, Figure 2(d) is the decoder flow chart; Figure 2(e) is the discriminator flow chart;

[0042] Figure 3 This is a comparison diagram of image results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for the sake of clarity and brevity, descriptions of known functions and structures are omitted in the embodiments. It should be understood that the reference to "one embodiment" or "this embodiment" throughout the specification means that the specific features, structures, or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the "one embodiment" or "this embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics can be combined in one or more embodiments in any suitable manner.

[0044] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0045] The term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, B exists alone, and both A and B exist. The term " / and" in this document describes another type of association between related objects, indicating that two possible relationships exist. For example, "A / and B" can represent: A exists alone, and both A and B exist alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0046] It should also be noted that, in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include," "comprises," or any other variations thereof are intended to cover non-exclusive inclusion.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment is an image synthesis method based on the transformer-GAN model, comprising the following steps:

[0049] S1, image acquisition: acquiring multiple imaging sequences;

[0050] S2. Data processing: Use the medical image registration module in 3D Slicer software to perform image registration on the imaging sequence, complete image cropping and brightness uniformity operations, and obtain an image pair dataset;

[0051] S3. Model training: Build a multi-head self-attention generative adversarial network model and deploy it on the server. Train it on the image pair dataset to obtain the optimal training model.

[0052] S4. Image synthesis: Image synthesis is performed using the optimally trained multi-head self-attention generative adversarial network model.

[0053] Technical effect of this embodiment: This embodiment synthesizes images based on the transformer-GAN model, providing a method for identifying vascular stenosis and plaques with low equipment requirements, low inspection requirements, short scanning time, and low cost.

[0054] Example 2

[0055] To solve the above problems, the present invention provides an image acquisition and testing method based on the transformer-GAN model, comprising the following steps:

[0056] S1. Image acquisition: A 3.0T magnetic resonance imaging system was used for imaging. A 24-channel cranial coil was used to select 71 cases for brain scanning. Time-of-flight magnetic resonance imaging (TOF-MRA), high-resolution intravenous (HR-IVW), and proton density-weighted image (PDW) sequences were directly acquired. T2 FSE sequences were extracted from the PDW sequences.

[0057] Specifically, the steps of extracting the T2 FSE sequence are as follows: reading the echo time of the proton density weighted image sequence, extracting the images with an echo time greater than 50, and obtaining the T2 FSE sequence;

[0058] S2. Data processing: Use the medical image registration module in 3D Slicer software to perform image registration on the imaging sequence, complete image cropping and brightness uniformity operations, and obtain an image pair dataset;

[0059] S2.1. Data registration and image cropping: The TOF-MRA sequence was registered with the T2 FSE sequence and the HR-IVW sequence, and the cerebral vascular region was cropped to obtain the TOF-MRA to HR-IVW image pair dataset and the TOF-MRA to T2 FSE image pair dataset.

[0060] Specifically, the TOF-MRA to HR-IVW image pair dataset includes a TOF-MRA image and its corresponding HR-IVW image, both of which are pixel-aligned. The TOF-MRA to T2 FSE image pair dataset includes a TOF-MRA image and its corresponding T2FSE image, both of which are pixel-aligned. In most data sets, the TOF-MRA, T2 FSE, and HR-IVW images are pixel-aligned.

[0061] S2.2. Brightness uniformity: Adjust the brightness values ​​of T2 FSE images and HR-IVW images to achieve overall uniformity.

[0062] S2.3. Dataset division: The two image pair datasets were randomly divided in a ratio of 5:2:3. The TOF-MRA to HR-IVW image pair dataset was divided into a training set of 220 cases (including 109,823 image pairs), a validation set of 89 cases (including 41,311 image pairs), and a test set of 135 cases (including 65,016 image pairs); the TOF-MRA to HR-IVW image pair dataset was divided into a training set of 205 cases (including 13,594 image pairs), a validation set of 73 cases (including 4,360 image pairs), and a test set of 112 cases (including 6,921 image pairs).

[0063] As shown in Figure 2, a is the overall flow chart of transformer-GAN model training, b is the encoder flow chart, c is the aggregated residual transformer (ART) flow chart, d is the decoder flow chart, and e is the discriminator flow chart.

[0064] S3. Model training: Build a multi-head self-attention generative adversarial network model and deploy it on the server. Train it on the image pair dataset to obtain the optimal training model.

[0065] Specifically, the graphics card model of the server in step S3 is Nvidia RTX A5000 GPU, and the memory is 24G.

[0066] S3.1. Model Construction: Construct a multi-head self-attention generative adversarial network model, including a generator and a discriminator; set the transformer-GAN model generator initialization training times, training data batch size, and model weight update pace parameters;

[0067] S3.2. Data training: Train the generators for synthetic HR-IVW images and synthetic T2 FSE images respectively;

[0068] The training of the generator data for synthesizing HR-IVW images consists of the following steps:

[0069] S3.2.1. Data collation: The TOF-MRA sequences in the training set of the TOF-MRA to HR-IVW image pair dataset were used as the initial image data, and the HR-IVW sequences in the same image pair were used as the real image data.

[0070] S3.2.2, Image synthesis: The initial image data of the TOF-MRA sequence is passed into the generator to obtain the synthesized HR-IVW sequence;

[0071] Specifically, the generator in step S3.2.2 includes an encoder, nine aggregated residual transformers, and a decoder, which are used to extract input image features and generate a synthesized image. The encoder consists of three convolutions and their corresponding activation functions; the aggregated residual transformer includes sampling, a multi-head attention mechanism, and convolution; the decoder includes two transposed convolutions, one convolution, and an activation function. The image synthesis includes the following steps:

[0072] S3.2.2.1. Encoder operation: Input the TOF-MRA sequence test set into the encoder and extract TOF-MRA sequence features;

[0073] Furthermore, the encoder is configured with multiple layers, first performing a data padding operation, and then performing a convolution operation, a batch normalization operation, and a ReLU linear correction operation on each layer;

[0074] Specifically, the encoder is configured with 3 layers, the first layer is based on a convolution kernel size of 7*7, the second and third layers are based on a convolution kernel size of 3*3, and each layer performs the convolution operation once with a convolution layer having a stride of 1.

[0075] S3.2.2.2. Aggregate Residual Transformer Operation: Input the TOF-MRA sequence features extracted in step S3.2.2.1 into the aggregate residual transformer to perform imaging sequence synthesis;

[0076] The aggregated residual transformer is configured with multiple layers, and the operations of each layer are as follows: downsampler operation; image segmentation and flattening operation; linear projection operation; position encoding operation; multi-head self-attention mechanism generation adversarial network model operation; recovery of spatial information operation; increase data resolution operation; concatenation operation of the input downsampler data and the data obtained by the data resolution increase operation; convolution operation;

[0077] Specifically, the multi-head self-attention mechanism generates an adversarial network model operation, which includes the following steps:

[0078] 1) Input the data obtained from the position encoding operation step of the aggregated residual transformer and perform layer normalization operation;

[0079] 2) Apply attention layers to the TOF-MRA image sequence in parallel;

[0080] 3) adding the data obtained by the position encoding operation step of the aggregated residual transformer input in step 1) and the data obtained in step 2) respectively on a pixel-by-pixel basis;

[0081] 4) Layer normalization operation;

[0082] 5) Multilayer Perceptron operation;

[0083] 6) The data obtained in step 3) and the data obtained in step 5) are added pixel by pixel.

[0084] Specifically, the convolution operation includes the following steps:

[0085] 7) Perform two-step convolution operations on the data obtained from the series operation and add them pixel by pixel;

[0086] Specifically, the first convolution layer with a convolution kernel size of 1*2 and a stride of 1 performs the convolution operation once; the second convolution layer with a convolution kernel size of 3*3 and a stride of 1 performs the convolution operation twice;

[0087] 8) performing multi-layer convolution operations, batch normalization operations, and linear correction operations on the data obtained in step 7);

[0088] Specifically, the step 8) is configured with 2 layers, and each layer performs the convolution operation once based on a convolution layer with a convolution kernel size of 3*3 and a step size of 1.

[0089] 9) Add the data obtained in step 7) to the data obtained in step 8) pixel by pixel, and then perform a linear correction operation. S3.2.2.3, Decoder Operation: Input the imaging sequence synthesized in step S3.2.2.2 into the decoder and extract the synthesized image;

[0090] The decoder is configured with multiple layers. The first few layers perform inverse convolution, batch normalization, and ReLU linear correction operations. The last layer performs data padding, convolution, and mapping using the Tanh hyperbolic tangent activation function. The convolution kernel size and number of operations selected for the convolution operation in each layer can be set independently.

[0091] Specifically, the decoder is configured with 3 layers, the first two layers are based on a convolution kernel size of 3*3, the third layer is based on a convolution kernel size of 7*7, and the convolution operation is performed once on each convolution layer with a step size of 1.

[0092] Specifically, the discriminator in step S3.2.2 is responsible for determining whether the input image is a synthetic image or a real image, and is composed of four convolution and activation functions.

[0093] S3.2.3, Image discrimination:

[0094] Discriminator operation: The real image data of the synthesized HR-IVW sequence obtained in step S3.2.2.3 and the HR-IVW sequence in the same image pair of the TOF-MRA sequence before synthesis are passed into the discriminator. The peak signal-to-noise ratio (PSNR) of the test set images, the mean absolute error (MAE) of the real images, and the feature vector distance (FID) between the generated images and the real images are calculated, and the loss function value is output;

[0095] The discriminator is configured with multiple layers. The first layer performs convolution operations and LeakyReLU linear correction operations, and the subsequent layers all perform convolution operations, batch normalization operations, and LeakyReLU linear correction operations; the last layer performs convolution operations; the convolution kernel size and number of operations selected for the convolution operation in each layer can be set independently.

[0096] Specifically, the discriminator is configured with 5 layers, and each layer performs the convolution operation once based on a convolution layer with a convolution kernel size of 4*4 and a stride of 1.

[0097] S3.2.4. Model tuning: Based on the loss function value, perform gradient descent and iterate optimization multiple times until the loss function value is minimized and converges, obtaining the optimal generator for synthesizing HR-IVW sequences;

[0098] S3.2.5. Model verification: Use the TOF-MRA sequence in the validation set divided from the TOF-MRA to HR-IVW image pair dataset as the initial image data, and the HR-IVW sequence in the same image pair as the real image data. Repeat steps S3.2.2 and S3.2.3 to calculate whether the loss function converges.

[0099] The same steps are taken for the TOF-MRA to T2 FSE image dataset to obtain the optimal generator of synthetic T2 FSE sequences;

[0100] Specifically, the data training of the generator for synthesizing T2 FSE images is consistent with steps S3.2.1-S3.2.5 to obtain the optimal generator for synthesizing T2 FSE images, wherein the TOF-MRA sequence in the training set of the TOF-MRA to T2 FSE image pair dataset is the initial image data, and the T2 FSE sequence in the same image pair is the real image data.

[0101] Finally, the loss function of the TOF-MRA sequence validation set for synthesizing HR-IVW images finally converged to 0.01707; the number of training times of the conversion model parameters of the TOF-MRA sequence validation set for synthesizing T2 FSE images was adjusted to 100, and the loss function finally converged to 0.01981; the optimally trained HR-IVW image synthesis model and T2 FSE image synthesis model were obtained.

[0102] Specifically, step S4 includes: inputting the TOF-MRA sequences in the test sets divided from the TOF-MRA to HR-IVW image pair dataset and the TOF-MRA to T2FSE image pair dataset into the generator trained in step S3, respectively, to obtain synthesized HR-IVW images and T2 FSE images.

[0103] like Figure 3 , which is a comparison chart of the image results obtained in this embodiment, the jth column is the TOF-MRA image in the test set, the kth column is the real HR-IVW image, the lth column is the synthesized HR-IVW image, the mth column is the real T2 FSE image, and the nth column is the synthesized T2 FSE image;

[0104] The a row shows different images of severe stenosis and fibrous plaque in the transverse section at the same lesion, the b row shows different images of mild stenosis and lipid plaque in the transverse section at the same lesion, the c row shows different images of severe stenosis and mixed plaque in the transverse section at the same lesion, the d row shows different images of severe stenosis and fibrous plaque in the sagittal section at the same lesion, the e row shows different images of mild stenosis and lipid plaque in the sagittal section at the same lesion, the f row shows different images of severe stenosis and mixed plaque in the sagittal section at the same lesion, the g row shows different images of mild stenosis and fibrous plaque in the coronary section at the same lesion, the h row shows different images of severe stenosis and lipid plaque in the coronary section at the same lesion, and the i row shows different images of severe stenosis and mixed plaque in the coronary section at the same lesion.

[0105] The resulting HR-IVW image synthesized in this example achieved a peak signal-to-noise ratio of 20.80, a structural similarity of 0.46 with the ground-truth image, a mean absolute error of 0.07, and an eigenvector distance of 176.25. The T2 FSE synthesized image achieved a peak signal-to-noise ratio of 18.93, a structural similarity of 0.42 with the ground-truth image, a mean absolute error of 0.08, and an eigenvector distance of 253.93.

[0106] Technical effect of this embodiment: Based on the image generation model, this embodiment uses a 3.0T or lower MRI scanner to obtain the synthesized HR-IVW images and T2 FSE images, which are highly similar to the images obtained by HR-IVW examination using a 3.0T or higher MRI scanner, and have a smaller error.

[0107] Example 3

[0108] In this embodiment, the image synthesis method of the transformer-GAN model in the above embodiment is applied to the identification of vascular stenosis and plaques in intracranial atherosclerotic diseases, and synthesized HR-IVW images and T2 FSE images are obtained. Two experienced radiologists independently use 625 HR-IVW synthetic images and 125 T2 FSE synthetic images obtained in the above embodiment to diagnose vascular stenosis and plaques. The HR-IVW synthetic images are used to diagnose vascular stenosis and identify plaques, and the type of plaque can be identified in combination with the T2FSE synthetic images. Compared with HR-IVW examinations using 3.0T and above nuclear magnetic resonance scanners, the vascular stenosis and plaque identification method based on the transformer-GAN model synthetic images of the present invention has a vascular stenosis recognition rate of 78.08%, a plaque recognition rate of 86.72%, and a plaque type recognition rate of 75.20%.

[0109] Technical effect of this embodiment: In actual application, the recognition of vascular stenosis and plaques by this embodiment is close to the results of HR-IVW examination using a 3.0T or higher MRI scanner, and the recognition rate is high.

[0110] The foregoing description is merely a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any variation, modification, replacement, integration, or parameter change to these embodiments, which is within the spirit and principles of the present invention and which achieves the same functionality through conventional substitutions, without departing from the principles and spirit of the present invention, falls within the scope of protection of the present invention.

Claims

1. An image synthesis method based on the transformer-GAN model, characterized in that: The following steps are involved: S1, image acquisition: acquiring multiple imaging sequences; S2. Data processing: Use the medical image registration module in 3D Slicer software to perform image registration on the imaging sequence, complete image cropping and brightness uniformity operations, and obtain an image pair dataset; S3. Model training: Build a multi-head self-attention generative adversarial network model and deploy it on the server. Train it on the image pair dataset to obtain the optimal training model. S4. Image synthesis: Image synthesis using the optimally trained multi-head self-attention generative adversarial network model; The step S2 comprises the following steps: S2.

1. Data registration and image cropping: Perform data registration on the time-of-flight MRI angiography sequence with the transverse relaxation-weighted fast spin echo imaging sequence and the high-resolution vascular wall imaging sequence, respectively. Cerebral vascular regions are cropped out to obtain image pair datasets of the time-of-flight MRI angiography sequence and the high-resolution vascular wall imaging sequence, as well as image pair datasets of the time-of-flight MRI angiography sequence and the transverse relaxation-weighted fast spin echo imaging sequence. The step S3 comprises the following steps: S3.

2. Data training: Train the generators for synthesizing high-resolution vessel wall imaging sequences and synthesizing transverse relaxation-weighted fast spin echo imaging sequences respectively; The step S3.2 includes the following steps: S3.2.

1. Data Arrangement: The time-leap MRI angiography sequence in the training set of the image pair data set divided by the time-leap MRI angiography sequence and the high-resolution vascular wall imaging sequence is used as the initial image data, and the high-resolution vascular wall imaging sequence in the same image pair is used as the real image data; S3.2.

2. Image synthesis: transmitting the initial image data of the time-of-flight magnetic resonance angiography sequence to a generator to obtain a synthesized high-resolution vascular wall imaging sequence; S3.2.

3. Image discrimination: Transmit the real image data of the synthesized high-resolution vascular wall imaging sequence and the pre-synthesis time-flight magnetic resonance angiography sequence in the same image pair into the discriminator, set a loss function, and output the loss function value; S3.2.

4. Model tuning: Based on the loss function value, perform gradient descent and iterate optimization multiple times until the loss function value is minimized and converges, thereby obtaining the optimal generator for synthesizing high-resolution vessel wall imaging sequences; S3.2.

5. Model Validation: Use the time-leap MRI angiography sequence from the validation set of the time-leap MRI angiography and high-resolution vascular wall imaging image pair dataset as the initial image data, and the high-resolution vascular wall imaging sequence from the same image pair as the real image data. Repeat steps S3.2.2 and S3.2.3 to calculate whether the loss function converges. The data training of the generator of the synthetic transverse relaxation weighted fast spin echo imaging sequence is consistent with the steps S3.2.1-S3.2.5 to obtain the optimal generator of the synthetic transverse relaxation weighted fast spin echo imaging sequence, wherein the time-flight magnetic resonance vascular imaging sequence in the training set divided by the data set of the time-flight magnetic resonance vascular and transverse relaxation weighted fast spin echo imaging sequence image pairs is the initial image data, and the transverse relaxation weighted fast spin echo imaging sequence in the same image pair is the real image data.

2. The image synthesis method based on the transformer-GAN model according to claim 1, characterized in that: The imaging sequence in step S1 uses a nuclear magnetic resonance scanner to perform a brain scan, directly acquiring a time-of-flight magnetic resonance angiography sequence, a high-resolution vascular wall imaging sequence, and a proton density-weighted imaging sequence, and extracting a transverse relaxation-weighted fast spin echo imaging sequence through the proton density-weighted imaging sequence.

3. The image synthesis method based on the transformer-GAN model according to claim 1, characterized in that: The step S2 further comprises the following steps: S2.

2. Brightness uniformity: Adjust the brightness values ​​of the transverse relaxation-weighted fast spin echo imaging sequence and the high-resolution vessel wall imaging sequence to achieve overall uniformity. S2.

3. Dataset division: The two image pair datasets are randomly divided into a training set, a validation set, and a test set.

4. The image synthesis method based on the transformer-GAN model according to claim 1, characterized in that: The step S3 further comprises the following steps: S3.

1. Model construction: Construct a multi-head self-attention generative adversarial network model, including a generator and a discriminator; set the transformer-GAN model generator initialization training times, training data batch size, and model weight update pace parameters.

5. The image synthesis method based on the transformer-GAN model according to claim 4, characterized in that: The generator in step S3 includes an encoder, 9 aggregated residual transformers and a decoder, which are used to extract the features of the input imaging sequence image and generate a synthetic image; the encoder consists of 3 convolutions and their corresponding activation functions; the aggregated residual transformer includes sampling, multi-head attention mechanism and convolution; the decoder includes two transposed convolutions, one convolution and activation function.

6. The image synthesis method based on the transformer-GAN model according to any one of claims 4-5, characterized in that: The discriminator in step S3 is composed of four convolution and activation functions, and is used to determine whether the input image is a synthetic image or a real image.

7. The image synthesis method based on the transformer-GAN model according to claim 1 or 3, characterized in that: The time-flight magnetic resonance angiography image, the transverse relaxation-weighted fast spin echo image, and the high-resolution vascular wall image selected for image cropping in step S2.1 all contain brain vessels, and the image elements are aligned during cropping.

8. The image synthesis method based on the transformer-GAN model according to claim 1 or 4, characterized in that: The step S4 includes: inputting the time-flight MRI angiography sequence in a test set divided from the data set of time-flight MRI angiography sequence and high-resolution vascular wall imaging sequence image pairs and the data set of time-flight MRI angiography and transverse relaxation weighted fast spin echo imaging sequence image pairs into the generator trained in step S3, respectively, to obtain a synthesized high-resolution vascular wall imaging sequence and transverse relaxation weighted fast spin echo imaging sequence.

Images