Transform-GAN model-based image synthesis method and application

Abstract

The invention relates to an image synthesis method based on a transformer-GAN model and application, and relates to the field of artificial intelligence image generation. The method comprises the following steps: S1, image acquisition: acquiring a plurality of imaging sequences; s2, data processing: performing image registration on the imaging sequence by using a medical image registration module in 3D Slicer software, and completing image cutting and brightness unification operations to obtain an image pair data set; s3, model training: constructing a multi-head self-attention generative adversarial network model, deploying the multi-head self-attention generative adversarial network model to a server, and training the image pair data set to obtain an optimal training model; and S4, image synthesis: performing image synthesis by using the optimal training multi-head self-attention generative adversarial network model. The application is the application in intracranial atherosclerotic disease vascular stenosis and plaque identification. The invention provides an image synthesis method which is more convenient, low in price and higher in recognition rate for treating intracranial atherosclerotic diseases based on the image conversion model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence image generation, and specifically relates to a method and application of image synthesis based on a transformer-GAN model. Background Art

[0002] Intracranial atherosclerotic disease (ICAD) is the leading cause of ischemic stroke worldwide, causing 46.6% of acute cerebral ischemia in my country. ICAD is associated with a high recurrence rate of stroke, dementia, and Alzheimer's disease, leading to cognitive decline in patients and greatly increasing their economic burden.

[0003] Chinese patent CN118711795A (publication date: September 27, 2024) discloses a method for identifying intracranial vascular characteristics of cerebral atherosclerotic disease. By obtaining the basic lipid data of the vascular pathological sample to be tested, the basic lipid data is input into the trained recognition model to obtain the target identification result. From the perspective of lipidomics, the random forest training model is used to identify the donor and recipient blood vessels of cerebral atherosclerotic disease. This invention only targets specific lipid metabolism abnormalities in the vascular tissue of ICAD patients, and lipidomics data needs to be obtained.

[0004] Vascular stenosis and plaque identification are the main diagnostic indicators of ICAD. Early detection of vascular stenosis and intracranial plaques and accurate assessment of plaque composition are of great significance for the clinical treatment and prevention of stroke. High-resolution magnetic resonance intracranial vascular wall imaging (hereinafter referred to as HR-IVW) is a non-invasive, radiation-free examination method with high resolution and high signal-to-noise ratio. It is an important means of diagnosing intracranial arterial lumen stenosis lesions and plaque analysis.

[0005] HR-IVW examination includes time-of-flight magnetic resonance angiography sequence (hereinafter referred to as TOF-MRA sequence), high-resolution vascular wall imaging sequence (hereinafter referred to as HR-IVW sequence), transverse relaxation weighted fast spin echo imaging sequence (hereinafter referred to as T2 FSE sequence) and other sequences for locating blood vessels. It requires a 3.0T or higher MRI scanner equipped with a special coil, uses three-dimensional sequence imaging, and reconstructs and evaluates vascular conditions based on a post-processing workstation. The equipment requirements are high and the operating environment is strict, so many medical institutions do not have these conditions. The HR-IVW examination takes a long time to scan, and the patient is required to remain still during the scan. Motion artifacts are easily generated, resulting in unusable images, and the examination is expensive.

[0006] For example, Chinese patent CN117174262A (publication date: December 5, 2023) discloses a method for establishing a computer-accurate simulation imaging model of stroke. CTP and CTA examinations of the head and neck are performed on selected patients. The three-dimensional reconstruction software MIMICS is used to analyze the infarct core area and ischemic penumbra of stroke patients, and accurately measure the corresponding volume and density. The imaging hemodynamic assessment lays the foundation for the study of hemodynamic changes in intracranial arteriosclerosis.

[0007] How to complete image generation between different scanning instruments or scanning sequences based on image conversion technology, reduce the number of scans, thereby reducing the time and money costs of inspections and reducing image artifacts requires further research. Summary of the invention

[0008] To solve the above problems, the present invention provides an image synthesis method and application based on a transformer-GAN model, the method comprising the following steps:

[0009] S1. Image acquisition: acquiring multiple imaging sequences;

[0010] S2. Data processing: Use the medical image registration module in the 3D Slicer software to perform image registration on the imaging sequence, complete image cropping and brightness unification operations, and obtain an image pair data set;

[0011] S3, model training: build a multi-head self-attention generative adversarial network model and deploy it on the server, train the image pair dataset to obtain the optimal training model;

[0012] S4. Image synthesis: Image synthesis is performed using the optimally trained multi-head self-attention generative adversarial network model.

[0013] Specifically, the imaging sequence in step S1 uses a nuclear magnetic resonance scanner to perform a cranial brain scan, directly acquires a TOF-MRA sequence, a HR-IVW sequence, and a proton density weighted image sequence, and extracts a T2 FSE sequence through the proton density weighted image sequence.

[0014] Specifically, step S2 includes the following steps:

[0015] S2.1. Data registration and image cropping: The TOF-MRA sequence was registered with the T2 FSE sequence and the HR-IVW sequence, and the cerebral vascular area was cropped to obtain the TOF-MRA to HR-IVW image pair dataset and the TOF-MRA to T2FSE image pair dataset;

[0016] S2.2, brightness uniformity: adjust the brightness values ​​of T2 FSE images and HR-IVW images to make the overall brightness consistent;

[0017] S2.3. Dataset division: The two image pair datasets are randomly divided into a training set, a validation set and a test set.

[0018] Specifically, step S3 includes the following steps:

[0019] S3.1. Model construction: Construct a multi-head self-attention generative adversarial network model, including a generator and a discriminator; set the transformer-GAN model generator initialization training times, training data batch size, and model weight update step parameters;

[0020] S3.2, data training: train the generators of synthetic HR-IVW images and synthetic T2 FSE images respectively;

[0021] Furthermore, the generator data training for synthesizing HR-IVW images includes the following steps:

[0022] S3.2.1, Data collation: The TOF-MRA sequence in the training set of the TOF-MRAto HR-IVW image pair dataset is used as the initial image data, and the HR-IVW sequence in the same image pair is used as the real image data;

[0023] S3.2.2, Image synthesis: The initial image data of the TOF-MRA sequence is passed into the generator to obtain a synthesized HR-IVW sequence;

[0024] S3.2.3, Image discrimination: The real image data of the HR-IVW sequence in the same image pair of the synthesized HR-IVW sequence and the TOF-MRA sequence before synthesis is passed into the discriminator, the loss function is set, and the loss function value is output;

[0025] S3.2.4, Model tuning: According to the loss function value, perform gradient descent and iterate optimization multiple times until the loss function value is minimized and converged, and obtain the optimal generator of synthetic HR-IVW images;

[0026] S3.2.5, model verification: Use the TOF-MRA sequence in the validation set divided from the TOF-MRA to HR-IVW image pair dataset as the initial image data, and the HR-IVW sequence in the same image pair as the real image data, repeat steps S3.2.2 and S3.2.3, and calculate whether the loss function converges.

[0027] The same steps are taken for the TOF-MRAto T2 FSE image dataset to obtain the optimal generator of synthetic T2 FSE sequences;

[0028] Specifically, the data training of the generator of synthetic T2 FSE images is consistent with the steps S3.2.1-S3.2.5 to obtain the optimal generator of synthetic T2 FSE images, wherein the TOF-MRA sequence in the training set divided by the TOF-MRAto T2 FSE image pair data set is the initial image data, and the T2 FSE sequence in the same image pair is the real image data.

[0029] The generator in step S3 includes an encoder, 9 aggregated residual transformers and a decoder, which are used to extract input image features and generate synthetic images. The encoder consists of 3 convolutions and their corresponding activation functions; the aggregated residual transformer includes sampling, multi-head attention mechanism and convolution; the decoder includes two transposed convolutions, one convolution and activation function.

[0030] Specifically, the discriminator in step S3 is responsible for determining whether the input image is a synthetic image or a real image, and is composed of four convolution and activation functions.

[0031] Specifically, the step S4 includes: inputting the TOF-MRA sequences in the test set divided by the TOF-MRA to HR-IVW image pair dataset and the TOF-MRA toT2FSE image pair dataset into the generator trained in step S3 respectively to obtain synthesized HR-IVW images and T2 FSE images.

[0032] Specifically, the TOF-MRA sequence selected for image cropping in step S2.1 contains brain blood vessels together with the HR-IVW sequence and the T2FSE sequence, and the image elements are aligned during cropping.

[0033] An image synthesis method based on the transformer-GAN model is applied to the identification of vascular stenosis and plaques in intracranial atherosclerotic diseases.

[0034] Compared with the prior art, the technical solution of the present application has the following advantages and effects:

[0035] 1. The present invention provides an image synthesis method based on the transformer-GAN model, which only uses TOF-MRA sequence, HR-IVW sequence, and T2 FSE sequence to obtain the optimal training image conversion technology transformer-GAN model, avoiding the need for traditional HR-IVW examinations to obtain multiple sequences using 3.0T and above nuclear magnetic resonance scanners, and the need to scan 9 sequences such as transverse relaxation weighted image sequences and longitudinal relaxation weighted image sequences, thereby reducing the equipment requirements for nuclear magnetic resonance scanners, avoiding motion artifacts, and reducing scanning time and costs.

[0036] 2. The present invention provides an application of an image synthesis method based on a transformer-GAN model. Based on the trained transformer-GAN model, the synthesized HR-IVW images and synthesized T2 FSE images obtained can be used to identify vascular stenosis and plaques in intracranial atherosclerotic diseases, providing a more convenient, effective, and highly recognized image acquisition method for the treatment of intracranial atherosclerotic diseases.

[0037] The above is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings as follows.

[0038] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

[0040] Figure 1 is a flow chart of an image synthesis method based on a transformer-GAN model according to an embodiment of the present invention;

[0041] FIG2 is a flowchart of model training according to an embodiment of the present invention, wherein FIG2(a) is a flowchart of the overall training of the transformer-GAN model, FIG2(b) is a flowchart of the encoder, FIG2(c) is a flowchart of the aggregated residual transformer (ART), FIG2(d) is a flowchart of the decoder; FIG2(e) is a flowchart of the discriminator;

[0042] Figure 3 This is a comparison diagram of image results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, rather than all the embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for clarity and brevity, the description of known functions and structures is omitted in the embodiment. It should be understood that "one embodiment" or "this embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "one embodiment" or "this embodiment" appearing in various places throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0044] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0045] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" in this article describes another type of association relationship between associated objects, indicating that there can be two relationships. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0046] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusions.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment is an image synthesis method based on a transformer-GAN model, comprising the following steps:

[0049] S1. Image acquisition: acquiring multiple imaging sequences;

[0050] S2. Data processing: Use the medical image registration module in the 3D Slicer software to perform image registration on the imaging sequence, complete image cropping and brightness unification operations, and obtain an image pair data set;

[0051] S3, model training: build a multi-head self-attention generative adversarial network model and deploy it on the server, train the image pair dataset to obtain the optimal training model;

[0052] S4. Image synthesis: Image synthesis is performed using the optimally trained multi-head self-attention generative adversarial network model.

[0053] Technical effect of this embodiment: This embodiment synthesizes images based on the transformer-GAN model, and provides a method for identifying vascular stenosis and plaques with low equipment requirements, low inspection requirements, short scanning time and low cost.

[0054] Example 2

[0055] To solve the above problems, the present invention provides an image acquisition and testing method based on a transformer-GAN model, comprising the following steps:

[0056] S1. Image acquisition: A 3.0T magnetic resonance imaging system was used for imaging. A 24-channel head coil was used to select 71 cases for brain scanning. TOF-MRA sequence, HR-IVW sequence, and proton density weighted image sequence were directly acquired, and T2 FSE sequence was extracted through the proton density weighted image sequence.

[0057] Specifically, the steps of extracting the T2 FSE sequence are as follows: reading the echo time of the proton density weighted image sequence, extracting the images with echo time greater than 50, and obtaining the T2 FSE sequence;

[0058] S2. Data processing: Use the medical image registration module in the 3D Slicer software to perform image registration on the imaging sequence, complete image cropping and brightness unification operations, and obtain an image pair data set;

[0059] S2.1. Data registration and image cropping: The TOF-MRA sequence was registered with the T2 FSE sequence and the HR-IVW sequence, and the cerebrovascular region was cropped to obtain the TOF-MRA to HR-IVW image pair dataset and the TOF-MRA toT2 FSE image pair dataset;

[0060] Specifically, the TOF-MRA to HR-IVW image pair dataset includes a TOF-MRA image and a corresponding HR-IVW image, and the pixels of the two are aligned; the TOF-MRA to T2 FSE image pair dataset includes a TOF-MRA image and a corresponding T2FSE image, and the pixels of the two are aligned; in most of the data, the pixels of the three images of TOF-MRA, T2 FSE and HR-IVW are all aligned;

[0061] S2.2, brightness uniformity: adjust the brightness values ​​of T2 FSE images and HR-IVW images to make the overall brightness consistent;

[0062] S2.3. Dataset division: The two image pair datasets were randomly divided in a ratio of 5:2:3. The TOF-MRA to HR-IVW image pair dataset was divided into a training set of 220 cases (including 109,823 image pairs), a validation set of 89 cases (including 41,311 image pairs), and a test set of 135 cases (including 65,016 image pairs); the TOF-MRA to HR-IVW image pair dataset was divided into a training set of 205 cases (including 13,594 image pairs), a validation set of 73 cases (including 4,360 image pairs), and a test set of 112 cases (including 6,921 image pairs).

[0063] As shown in Figure 2, a is the overall flow chart of transformer-GAN model training, b is the encoder flow chart, c is the aggregated residual transformer (ART) flow chart, d is the decoder flow chart, and e is the discriminator flow chart.

[0064] S3, model training: build a multi-head self-attention generative adversarial network model and deploy it on the server, train the image pair dataset to obtain the optimal training model;

[0065] Specifically, in step S3, the graphics card model of the server is Nvidia RTX A5000 GPU, and the memory is 24G.

[0066] S3.1. Model construction: Construct a multi-head self-attention generative adversarial network model, including a generator and a discriminator; set the transformer-GAN model generator initialization training times, training data batch size, and model weight update step parameters;

[0067] S3.2, data training: train the generators of synthetic HR-IVW images and synthetic T2 FSE images respectively;

[0068] The training of generator data for synthetic HR-IVW images consists of the following steps:

[0069] S3.2.1, Data collation: The TOF-MRA sequence in the training set of the TOF-MRA to HR-IVW image pair dataset is used as the initial image data, and the HR-IVW sequence in the same image pair is used as the real image data;

[0070] S3.2.2, Image synthesis: The initial image data of the TOF-MRA sequence is passed into the generator to obtain a synthesized HR-IVW sequence;

[0071] Specifically, the generator in step S3.2.2 includes an encoder, 9 aggregated residual transformers and a decoder, which are used to extract input image features and generate a synthetic image. The encoder consists of 3 convolutions and their corresponding activation functions; the aggregated residual transformer includes sampling, multi-head attention mechanism and convolution; the decoder includes two transposed convolutions, one convolution and activation function. The image synthesis includes the following steps:

[0072] S3.2.2.1. Encoder operation: Input the TOF-MRA sequence test set into the encoder and extract the TOF-MRA sequence features;

[0073] Furthermore, the encoder is configured with multiple layers, first performing a data padding operation, and then each layer performs a convolution operation, a batch normalization operation, and a ReLU linear correction operation;

[0074] Specifically, the encoder is configured with 3 layers, the first layer is based on a convolution kernel size of 7*7, the second and third layers are based on a convolution kernel size of 3*3, and each convolution layer performs the convolution operation once with a step size of 1.

[0075] S3.2.2.2, Aggregate Residual Transformer Operation: Input the TOF-MRA sequence features extracted in step S3.2.2.1 into the aggregate residual transformer to perform imaging sequence synthesis;

[0076] The aggregated residual transformer is configured with multiple layers, and the operations of each layer are as follows: downsampler operation; image segmentation and flattening operation; linear projection operation; position encoding operation; multi-head self-attention mechanism to generate adversarial network model operation; recovery of spatial information operation; increase of data resolution operation; concatenation of the data input to the downsampler and the data obtained by the data resolution increase operation; convolution operation;

[0077] Specifically, the multi-head self-attention mechanism generates an adversarial network model operation, including the following steps:

[0078] 1) Input the data obtained from the position encoding operation step of the aggregated residual transformer and perform layer normalization operation;

[0079] 2) Apply attention layers to the TOF-MRA image sequence in parallel;

[0080] 3) adding the data obtained by the position encoding operation step of the aggregated residual transformer input in step 1) and the data obtained in step 2) respectively on a pixel-by-pixel basis;

[0081] 4) Layer normalization operation;

[0082] 5) Multi-layer perceptron operation;

[0083] 6) The data obtained in step 3) and the data obtained in step 5) are added pixel by pixel.

[0084] Specifically, the convolution operation includes the following steps:

[0085] 7) Perform two-step convolution operations on the data obtained by the series operation and add them pixel by pixel;

[0086] Specifically, the first convolution kernel size is 1*2, and the convolution layer with a step size of 1 performs the convolution operation once; the second convolution kernel size is 3*3, and the convolution layer with a step size of 1 performs the convolution operation twice;

[0087] 8) performing multi-layer convolution operation, batch normalization operation, and linear correction operation on the data obtained in step 7);

[0088] Specifically, the step 8) is configured with 2 layers, and each layer performs the convolution operation once based on a convolution layer with a convolution kernel size of 3*3 and a step size of 1.

[0089] 9) Add the data obtained in step 7) to the data obtained in step 8) by pixel, and then perform a linear correction operation. S3.2.2.3, decoder operation: input the imaging sequence synthesized in step S3.2.2.2 into the decoder, and extract the synthesized image;

[0090] The decoder is configured with multiple layers. The first few layers perform inverse convolution operations, batch normalization operations, and ReLU linear correction operations. The last layer performs data padding operations, convolution operations, and mapping operations using the Tanh hyperbolic tangent activation function. The convolution kernel size and number of operations selected for the convolution operation in each layer can be set independently.

[0091] Specifically, the decoder is configured with 3 layers, the first two layers are based on a convolution kernel size of 3*3, and the third layer is based on a convolution kernel size of 7*7, and each convolution layer with a step size of 1 performs the convolution operation once.

[0092] Specifically, the discriminator in step S3.2.2 is responsible for determining whether the input image is a synthetic image or a real image, and is composed of four convolution and activation functions.

[0093] S3.2.3, Image discrimination:

[0094] Discriminator operation: pass the real image data of the HR-IVW sequence in the same image pair of the synthesized HR-IVW sequence and the TOF-MRA sequence before synthesis into the discriminator, calculate the peak signal-to-noise ratio (PSNR) of the test set image between the generated image and the real image, the mean absolute error (MAE) of the real image, and the feature vector distance (FID), and output the loss function value;

[0095] The discriminator is configured with multiple layers. The first layer performs convolution operations and LeakyReLU linear correction operations, and the subsequent multiple layers perform convolution operations, batch normalization operations, and LeakyReLU linear correction operations; the last layer performs convolution operations; the convolution kernel size and the number of operations selected for the convolution operation of each layer can be set independently.

[0096] Specifically, the discriminator is configured with 5 layers, and each layer performs the convolution operation once based on a convolution layer with a convolution kernel size of 4*4 and a step size of 1.

[0097] S3.2.4, Model tuning: According to the loss function value, perform gradient descent and multiple iterations of optimization until the loss function value is minimized and converged, and obtain the optimal generator of synthetic HR-IVW sequences;

[0098] S3.2.5, model verification: use the TOF-MRA sequence in the verification set divided from the TOF-MRAto HR-IVW image pair dataset as the initial image data, and the HR-IVW sequence in the same image pair as the real image data, repeat steps S3.2.2 and S3.2.3, and calculate whether the loss function converges.

[0099] The same steps were taken for the TOF-MRA to T2 FSE image dataset to obtain the optimal generator for synthetic T2 FSE sequences;

[0100] Specifically, the data training of the generator of synthetic T2 FSE images is consistent with the steps S3.2.1-S3.2.5 to obtain the optimal generator of synthetic T2 FSE images, wherein the TOF-MRA sequence in the training set divided by the TOF-MRA to T2 FSE image pair data set is the initial image data, and the T2 FSE sequence in the same image pair is the real image data.

[0101] Finally, the loss function of the TOF-MRA sequence validation set for synthesizing HR-IVW images finally converged to 0.01707; the number of training times for the conversion model parameters of the TOF-MRA sequence validation set for synthesizing T2 FSE images was adjusted to 100, and the loss function finally converged to 0.01981; the optimally trained HR-IVW image synthesis model and T2 FSE image synthesis model were obtained.

[0102] Specifically, the step S4 includes: inputting the TOF-MRA sequences in the test set divided by the TOF-MRA to HR-IVW image pair dataset and the TOF-MRA toT2FSE image pair dataset into the generator trained in step S3 respectively to obtain synthesized HR-IVW images and T2 FSE images.

[0103] like Figure 3 As shown in the figure, it is a comparison diagram of the image results obtained in this embodiment, the jth column is the TOF-MRA image in the test set, the kth column is the real HR-IVW image, the lth column is the synthesized HR-IVW image, the mth column is the real T2 FSE image, and the nth column is the synthesized T2 FSE image;

[0104] The a row shows different images of severe stenosis fibrous plaques in the same lesion in transverse section, the b row shows different images of mild stenosis lipid plaques in the same lesion in transverse section, the c row shows different images of severe stenosis mixed plaques in the same lesion in transverse section, the d row shows different images of severe stenosis fibrous plaques in the same lesion in sagittal section, the e row shows different images of mild stenosis lipid plaques in the same lesion in sagittal section, the f row shows different images of severe stenosis mixed plaques in the same lesion in sagittal section, the g row shows different images of mild stenosis fibrous plaques in the same lesion in coronary section, the h row shows different images of severe stenosis lipid plaques in the same lesion in coronary section, and the i row shows different images of severe stenosis mixed plaques in the same lesion in coronary section.

[0105] The peak signal-to-noise ratio of the HR-IVW image synthesized in this embodiment reaches 20.80; the structural similarity with the real image reaches 0.46; the mean absolute error with the real image is 0.07; and the feature vector distance is 176.25. The peak signal-to-noise ratio of the T2 FSE synthesized image is 18.93; the structural similarity with the real image is 0.42; the mean absolute error with the real image is 0.08; and the feature vector distance is 253.93.

[0106] Technical effect of this embodiment: This embodiment is based on an image generation model, and the synthetic HR-IVW images and T2 FSE images obtained by using a MRI scanner below 3.0T are highly similar to the images obtained by HR-IVW examination using a MRI scanner of 3.0T and above, and the errors are relatively small.

[0107] Example 3

[0108] In this embodiment, the image synthesis method of the transformer-GAN model in the above embodiment is applied to the identification of vascular stenosis and plaques in intracranial atherosclerotic diseases, and synthesized HR-IVW images and T2 FSE images are obtained. Two experienced radiologists independently use 625 HR-IVW synthetic images and 125 T2 FSE synthetic images obtained in the above embodiment to diagnose vascular stenosis and plaques. The HR-IVW synthetic images are used to diagnose vascular stenosis and identify plaques, and the type of plaque can be identified in combination with the T2FSE synthetic images. Compared with HR-IVW examinations using 3.0T and above nuclear magnetic resonance scanners, the vascular stenosis and plaque identification method based on the transformer-GAN model synthetic images of the present invention has a vascular stenosis recognition rate of 78.08%, a plaque recognition rate of 86.72%, and a plaque type recognition rate of 75.20%.

[0109] Technical effect of this embodiment: In actual application, the recognition of vascular stenosis and plaques by this embodiment is close to the results of HR-IVW examination using a 3.0T or higher MRI scanner, and has a high recognition rate.

[0110] The above description is only the preferred embodiment of the present invention, which does not limit the protection scope of the present invention. For those skilled in the art, the present invention can be modified and varied in various ways. Within the spirit and principle of the present invention, any change, modification, replacement, integration and parameter change of these embodiments by conventional substitution or capable of achieving the same function without departing from the principle and spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. An image synthesis method based on a transformer-GAN model, characterized in that: The following steps are involved: S1. Image acquisition: acquiring multiple imaging sequences; S2. Data processing: Use the medical image registration module in the 3D Slicer software to perform image registration on the imaging sequence, complete image cropping and brightness unification operations, and obtain an image pair data set; S3, model training: build a multi-head self-attention generative adversarial network model and deploy it on the server, train the image pair dataset to obtain the optimal training model; S4. Image synthesis: Image synthesis is performed using the optimally trained multi-head self-attention generative adversarial network model.

2. The image synthesis method based on the transformer-GAN model according to claim 1, characterized in that: The imaging sequence in step S1 uses a nuclear magnetic resonance scanner to perform a brain scan, directly obtains a time-of-flight magnetic resonance angiography sequence, a high-resolution vascular wall imaging sequence, and a proton density weighted image sequence, and extracts a transverse relaxation weighted fast spin echo imaging sequence through the proton density weighted image sequence.

3. The image synthesis method based on the transformer-GAN model according to claim 1, characterized in that: The step S2 comprises the following steps: S2.

1. Data registration and image cropping: The time-of-flight MRI angiography sequence is registered with the transverse relaxation weighted fast spin echo imaging sequence and the high-resolution vascular wall imaging sequence, and the cerebral vascular region is cropped to obtain a dataset of image pairs of the time-of-flight MRI angiography sequence and the high-resolution vascular wall imaging sequence, and a dataset of image pairs of the time-of-flight MRI angiography sequence and the transverse relaxation weighted fast spin echo imaging sequence; S2.

2. Brightness uniformity: adjust the brightness values ​​of the transverse relaxation weighted fast spin echo imaging sequence and the high-resolution vascular wall imaging sequence to make the overall brightness consistent; S2.

3. Dataset division: The two image pair datasets are randomly divided into a training set, a validation set and a test set.

4. The image synthesis method based on the transformer-GAN model according to claim 1, characterized in that: The step S3 comprises the following steps: S3.

1. Model construction: Construct a multi-head self-attention generative adversarial network model, including a generator and a discriminator; set the number of initialization training times of the transformer-GAN model generator, the training data batch size, and the model weight update step parameters; S3.

2. Data training: Train the generators for synthesizing high-resolution vascular wall imaging sequences and synthesizing transverse relaxation-weighted fast spin echo imaging sequences respectively.

5. The image synthesis method based on the transformer-GAN model according to claim 4, characterized in that: The generator data training for synthesizing high-resolution vascular wall imaging sequences in step S3.2 comprises the following steps: S3.2.

1. Data collation: using the time-leap MRI angiography sequence in the training set divided by the image pair data set of the time-leap MRI angiography sequence and the high-resolution vascular wall imaging sequence as initial image data, and using the high-resolution vascular wall imaging sequence in the same image pair as real image data; S3.2.2, image synthesis: transferring the initial image data of the time-flight magnetic resonance angiography sequence into a generator to obtain a synthesized high-resolution vascular wall imaging sequence; S3.2.3, image discrimination: the real image data of the high-resolution vascular wall imaging sequence in the same image pair of the synthesized high-resolution vascular wall imaging sequence and the time-flight magnetic resonance angiography sequence before synthesis are input into the discriminator, a loss function is set, and the loss function value is output; S3.2.4, model tuning: according to the loss function value, gradient descent is performed, and multiple iterations of optimization are performed until the loss function value is minimized and converged, thereby obtaining the optimal generator for synthesizing high-resolution vascular wall imaging sequences; S3.2.5, model verification: use the time-leap MRI angiography sequence in the validation set divided from the image pair data set of the time-leap MRI angiography sequence and the high-resolution vascular wall imaging sequence as the initial image data, use the high-resolution vascular wall imaging sequence in the same image pair as the real image data, repeat steps S3.2.2 and S3.2.3, and calculate whether the loss function converges; The data training of the generator of the synthetic transverse relaxation weighted fast spin echo imaging sequence is consistent with the steps S3.2.1-S3.2.5 to obtain the optimal generator of the synthetic transverse relaxation weighted fast spin echo imaging sequence, wherein the time-flight magnetic resonance vascular imaging sequence in the training set divided by the data set of the time-flight magnetic resonance vascular and transverse relaxation weighted fast spin echo imaging sequence image pairs is the initial image data, and the transverse relaxation weighted fast spin echo imaging sequence in the same image pair is the real image data.

6. The image synthesis method based on the transformer-GAN model according to claim 4 or 5, characterized in that: The generator in step S3 includes an encoder, 9 aggregated residual transformers and a decoder, which are used to extract the image features of the input imaging sequence and generate a synthetic image; the encoder is composed of 3 convolutions and their corresponding activation functions; the aggregated residual transformer includes sampling, multi-head attention mechanism and convolution; the decoder includes two transposed convolutions, one convolution and an activation function.

7. The image synthesis method based on the transformer-GAN model according to any one of claims 4 to 6, characterized in that: The discriminator in step S3 is composed of four convolution and activation functions, and is used to determine whether the input image is a synthetic image or a real image.

8. The image synthesis method based on the transformer-GAN model according to claim 1 or 3, characterized in that: The time-flight magnetic resonance vascular image selected for image cropping in step S2.1, the transverse relaxation weighted fast spin echo image and the high-resolution vascular wall image all contain brain vessels, and the image elements are aligned during cropping.

9. The image synthesis method based on the transformer-GAN model according to claim 1 or 4, characterized in that: The step S4 comprises: inputting the time-flight MRI angiography sequence in the test set divided from the time-flight MRI angiography sequence and high-resolution vascular wall imaging sequence image pair data set and the time-flight MRI angiography and transverse relaxation weighted fast spin echo imaging sequence image pair data set into the generator trained in step S3 respectively, to obtain a synthesized high-resolution vascular wall imaging sequence and transverse relaxation weighted fast spin echo imaging sequence.

10. Application of the image synthesis method based on the transformer-GAN model as described in any one of claims 1 to 9 in the identification of vascular stenosis and plaques in intracranial atherosclerotic diseases.

Images