MRI-to-PET medical image synthesis method and device

Abstract

The invention discloses an MRI (Magnetic Resonance Imaging)-PET (Positron Emission Tomography) medical image synthesis method and device, and the method comprises the steps: constructing a forward modal attribute decoupling mechanism: decoupling a corresponding appearance vector and an anatomical structure vector from an MRI modal image and a PET modal image, and synthesizing the PET modal image through the appearance vector of the PET modal image and the anatomical structure vector of the MRI modal image; constructing a backward modal attribute decoupling mechanism: learning distribution of appearance vectors of PET modal images by using Gaussian vectors, and guiding MRI to PET medical image synthesis by using the Gaussian vectors as the appearance vectors of the PET modal images in a test stage; and designing a diagnosis-oriented modal attribute optimization strategy: predicting more accurate diagnosis information by effectively polymerizing complementary disease-related information between the MRI and the PET modal image so as to guide the network to synthesize a high-quality PET modal image. The device comprises a processor and a memory. According to the method, modal attribute information contained in different imaging modes is fully mined to guide the network to synthesize a high-quality PET modal image.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and medical image synthesis, and in particular to a method and device for synthesizing medical images from MRI to PET. Background Art

[0002] With the advancement of medical imaging technology, magnetic resonance imaging (MRI) and positron emission computed tomography (PET) technologies play a vital role in the diagnosis of brain diseases. Considering that MRI modal images can capture the structural information of the human brain and PET modal images can provide functional information of the human brain, the combined use of MRI and PET modal images can provide complementary multimodal information for the diagnosis of brain diseases such as Alzheimer's Disease (AD), thereby improving the accuracy of disease diagnosis. However, in clinical applications, considering the high radiation risk of PET imaging and the high cost of image acquisition, PET modal images are often missing, which limits the performance of combined use of MRI and PET modal images for AD diagnosis. In this context, studying a method for synthesizing MRI to PET medical images can help assist doctors in more efficient AD diagnosis in clinical applications, which is of great research significance.

[0003] In recent years, thanks to the powerful feature extraction and learning capabilities of convolutional neural networks, the use of convolutional neural networks to achieve MRI to PET medical image synthesis has gradually become a hot research direction. Hu et al. proposed a bidirectional mapping generative adversarial network, which uses PET modality information to guide the conversion of MRI modality images to PET modality images. In order to further improve the quality of synthesized PET modality images in disease-related areas, Pan et al. used a pre-trained disease diagnosis network to provide disease prior knowledge to the synthesis network, effectively assisting the synthesis network to synthesize PET modality images containing more disease-related information. Although the above method improves the quality of synthesized PET modality images, there are still two problems. First, MRI and PET modality images are obtained by different instruments shooting the same part of the subject, and the two have anatomical consistency and appearance differences. The existing methods do not fully exploit the modality attribute information contained in different imaging methods to assist image synthesis, which limits the quality of synthesized PET modality images. In addition, although the above method uses diagnostic information to guide the synthesis network, it improves the synthesis quality of disease-related areas. However, the existing methods ignore the difference in the contribution of different modality images to disease diagnosis tasks, and cannot provide accurate diagnostic information for the synthesis network, and thus it is difficult to guide the network to synthesize high-quality PET modality images. Summary of the invention

[0004] The present invention provides a method and device for synthesizing medical images from MRI to PET. The present invention comprehensively utilizes the anatomical structure consistency and appearance difference of MRI modality images and PET modality images to improve the synthesis quality of PET modality images. The present invention effectively aggregates complementary disease-related information between MRI and PET modality images to predict more accurate diagnostic information to guide the network to synthesize high-quality PET modality images, as described below:

[0005] A method for synthesizing MRI to PET medical images, the method comprising:

[0006] Construct a forward modality attribute decoupling mechanism: decouple the corresponding appearance vector and anatomical structure vector from MRI and PET modality images respectively, and synthesize the PET modality image using the appearance vector of the PET modality image and the anatomical structure vector of the MRI modality image;

[0007] Construct a backward modality attribute decoupling mechanism: use Gaussian vectors to learn the distribution of appearance vectors of PET modality images, and use Gaussian vectors as appearance vectors of PET modality images to guide MRI to PET medical image synthesis during the test phase;

[0008] Design a diagnosis-oriented modality attribute optimization strategy: Utilize the complementary disease-related information between MRI and PET modality images, predict diagnostic information to optimize the modality attributes of MRI and PET modality images, and then synthesize PET modality images.

[0009] The decoupling of the corresponding appearance vector and anatomical structure vector from the MRI and PET modality images is specifically as follows:

[0010] The MRI modality image I is obtained by the appearance vector encoder and the anatomical structure vector encoder respectively. MRI and PET modality images I PET The appearance vector and anatomical structure vector are as follows:

[0011]

[0012] in, and Represent the anatomical structure vector encoders of MRI modality images and PET modality images respectively, which are composed of three stacked convolutional layers, and They represent the appearance vector encoders of MRI and PET modality images, respectively, which are composed of four stacked convolutional layers, two fully connected layers, and reparameterization operations. MRI and a PET They represent the anatomical structure vectors obtained after decoupling the MRI modality image and the PET modality image, respectively. MRI andPET They represent the appearance vectors obtained after decoupling the MRI modality image and the PET modality image, respectively.

[0013] The method of synthesizing the PET modality image by using the appearance vector of the PET modality image and the anatomical structure vector of the MRI modality image is specifically as follows:

[0014] A MRI and PET As the PET modality generator G P The input is used to synthesize PET modality images Will s MRI and a PET As the MRI modality generator G M The input is used to synthesize MRI modality images as follows:

[0015]

[0016] Among them, GP and G M Respectively, they represent the generators for MRI modality images and PET modality images consisting of three stacked convolutional layers and adaptive instance normalization layers;

[0017] A MRI and MRI As the MRI modality generator G M The input is used to synthesize MRI modality images A PET and sP ET As input to the PET modality generator GP to synthesize PET modality images The formula is as follows:

[0018]

[0019] The method of using Gaussian vectors to learn the distribution of appearance vectors of PET modality images and using Gaussian vectors as appearance vectors of PET modality images to guide MRI to PET medical image synthesis in the test phase is specifically as follows:

[0020] The anatomical structure vector a MRI and a randomly sampled Gaussian vector N(z) as the PET modality generator G P The input to predict PET modality images as follows:

[0021]

[0022] The synthesized PET modality image As the input of the PET modality image appearance vector encoder GP, the appearance vector

[0023] The method of using complementary disease-related information between MRI and PET modality images to predict diagnostic information to optimize modality properties of MRI and PET modality images and then synthesizing PET modality images is specifically as follows:

[0024] The appearance vector s of the MRI modality image MRI and the appearance vector s of the PET modality image PET The disease-related features are enhanced by the self-attention mechanism to obtain the enhanced s MRI and PET , using the mutual attention mechanism to enhance the s MRI and PET The complementary appearance vector sa is obtained by fusion ll , the formula is as follows:

[0025] s all =cross_att(self_att(s MRI ), self_att(s PET ))

[0026] Among them, self_att(·) represents the self-attention mechanism, cross_att(·) represents the mutual attention mechanism, based on the complementary appearance vector s all and MRI anatomical structure vector a MRI , the disease diagnosis feature F is generated by using the adaptive instance normalization operation, and the disease diagnosis feature F is used as the input of the disease classifier to achieve accurate AD diagnosis.

[0027] Wherein, the method further comprises: a training mechanism for training an MRI to PET medical image synthesis network;

[0028] The first stage trains the forward modality attribute decoupling mechanism and the backward modality attribute decoupling mechanism to synthesize PET modality images. The overall loss function of this stage is:

[0029]

[0030] Among them, α 1 , α 2 , α 3 , α 4 Represents the weight coefficients of different loss functions;

[0031] The second stage trains the diagnosis-oriented modality attribute optimization strategy to improve the quality of the synthetic PET modality images. The overall loss function of this stage is:

[0032] L second =β 1 L D

[0033] Among them, β 1 Represents the weight coefficient of the loss function;

[0034] The third stage trains the forward modal attribute decoupling mechanism and the backward modal attribute decoupling mechanism. The overall loss function of this stage is L all =L first +L second .

[0035] The method further comprises: using the MRI modality image as an anatomical structure vector encoder The randomly sampled Gaussian vector and the anatomical structure vector of the MRI modality image are used as the PET modality generator G P The final PET modality image is synthesized from the input.

[0036] In a second aspect, a device for synthesizing MRI to PET medical images comprises: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0037] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0038] The beneficial effects of the technical solution provided by the present invention are:

[0039] 1. The present invention constructs a forward modality attribute decoupling mechanism to improve the synthesis quality of PET modality images by mining the consistent anatomical structure information and the different appearance information between different modality images; in addition, a backward modality attribute decoupling mechanism is also constructed to learn the appearance information of PET modality images by using Gaussian vectors in the training stage to guide the synthesis of medical images from MRI to PET in the testing stage;

[0040] 2. The present invention designs a diagnosis-oriented modality attribute optimization strategy, which predicts more accurate diagnostic information by comprehensively utilizing the complementary disease-related information between MRI and PET modality images to optimize the modality attributes of MRI and PET modality images, thereby improving the quality of the synthesized PET modality images;

[0041] 3. The present invention improves the performance of medical image synthesis by jointly utilizing the forward modal attribute decoupling mechanism, the backward modal attribute decoupling mechanism and the diagnosis-oriented modal attribute optimization strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a workflow diagram of a method for synthesizing medical images from MRI to PET;

[0043] Figure 2 This is a quantitative comparison chart of this method and other advanced methods. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0045] Example 1

[0046] To overcome the deficiencies of the prior art, an embodiment of the present invention provides a method for synthesizing MRI to PET medical images, the method comprising:

[0047] 1. Constructing a forward modal attribute decoupling mechanism

[0048] Considering that MRI modality images and PET modality images are obtained by shooting the same part of the subject with different instruments, the two have anatomical consistency and appearance difference. Comprehensively utilizing the anatomical consistency and appearance difference of MRI and PET modality images is helpful to synthesize high-quality PET modality images. The embodiment of the present invention designs a forward modality attribute decoupling mechanism, which decouples the corresponding appearance vector and anatomical structure vector from MRI and PET modality images respectively, and then synthesizes the PET modality image by utilizing the appearance vector of the PET modality image and the anatomical structure vector of the MRI modality image. Among them, anatomical structure is a technical term well known in the art, and the anatomical structure vector is the information of the anatomical structure expressed in a vectorized form, which will not be elaborated in the embodiment of the present invention.

[0049] Specifically, the MRI modality image IM is first obtained by the appearance vector encoder and the anatomical structure vector encoder. R I and PET modality images, P ET The appearance vector and anatomical structure vector are expressed as follows:

[0050]

[0051] in, and Represent the anatomical structure vector encoders of MRI modality images and PET modality images respectively, which are composed of three stacked convolutional layers, and They represent the appearance vector encoders of MRI and PET modality images, respectively, which are composed of four stacked convolutional layers, two fully connected layers, and reparameterization operations. MRI and a PETThey represent the anatomical structure vectors obtained after decoupling the MRI modality image and the PET modality image, respectively. MRI and PET They represent the appearance vectors obtained after decoupling the MRI modality image and the PET modality image, respectively.

[0052] Furthermore, a MRI and PET As the PET modality generator G P The input is used to synthesize PET modality images At the same time, MRI and a PET As the MRI modality generator G M The input is used to synthesize MRI modality images The formula is as follows:

[0053]

[0054] Among them, GP and G M Represents the generators for MRI modality images and PET modality images, respectively, which consist of three stacked convolutional layers and adaptive instance normalization layers.

[0055] Furthermore, a MRI and MRI As the MRI modality generator G M The input is used to synthesize MRI modality images At the same time, a PET and PET As the PET modality generator G P The input is used to synthesize PET modality images The formula is as follows:

[0056]

[0057] In order to constrain the synthetic PET modality images and the real PET modality image I PET Consistency between and synthesized MRI modality images and the real MRI modality image I MRI The consistency between them introduces the reconstruction loss and The formula is as follows:

[0058]

[0059] Where E represents the mathematical expectation, ||·|| 1 represents the mean absolute error.

[0060] At the same time, in order to ensure the consistency of the anatomical structure vectors extracted from the MRI modality images and the PET modality images, the anatomical structure consistency loss L is constructed. a The difference between the anatomical structure vectors of the constrained MRI modality image and the PET modality image is expressed as follows:

[0061]

[0062] Among them, D a stands for anatomical structure discriminator.

[0063] In addition, the embodiment of the present invention also introduces the anti-loss and and L 1 loss and To ensure the synthesis of MRI modality images and synthetic PET modality images The authenticity of is expressed as follows:

[0064]

[0065] Among them, D P represents the PET modality image discriminator, D M stands for MRI Modality Image Discriminator.

[0066] 2. Constructing a backward modal attribute decoupling mechanism

[0067] Considering that it is impossible to obtain real PET modality images during the testing phase, PET , which makes it impossible to obtain the appearance vector s of the real PET modality image PET To realize the forward modality attribute decoupling mechanism. Therefore, a backward modality attribute decoupling mechanism is constructed. By using Gaussian vectors to learn PET modality images I PET The distribution of appearance vectors is used to guide MRI to PET medical image synthesis by utilizing Gaussian vectors as appearance vectors of PET modality images in the testing phase.

[0068] Specifically, the anatomical structure vector a of the MRI modality image is obtained by using the forward modality attribute decoupling mechanism. MRI After that, a MRI and a randomly sampled Gaussian vector N(z) as the PET modality generator G P The input to predict PET modality images The formula is as follows:

[0069]

[0070] Furthermore, in order to ensure that the synthesized PET modality images The authenticity of the synthetic PET modality images As the input of the PET modality image appearance vector encoder GP, the appearance vector The latent vector loss L is introduced latent Constrain Appearance Vector The consistency with the Gaussian vector N(z) is expressed as follows:

[0071]

[0072] In order to constrain the appearance vector of the PET modality image to obey the Gaussian distribution prior, the KL divergence constraint function L is introduced. KL Constrained Gaussian vector N(z) and appearance vector s of PET modality image PET The consistency of distribution. The formula is as follows:

[0073] L KL =E[D KL (N(z)||s PET )]

[0074] Among them, D KL represents the KL divergence.

[0075] Finally, the adversarial loss is introduced and L 1 Loss Function Ensure synthetic PET modality images and the real PET modality image, P ET The consistency between them is expressed as follows:

[0076]

[0077] 3. Design diagnosis-oriented modal attribute optimization strategy

[0078] Considering that introducing disease-related information during the synthesis process helps to improve the synthesis quality of disease-related areas in PET modal images, an embodiment of the present invention designs a diagnosis-oriented modal attribute optimization strategy. This strategy predicts more accurate diagnostic information to optimize the modal attributes of MRI and PET modal images, thereby improving the quality of the synthesized PET modal images, by comprehensively utilizing complementary disease-related information between MRI and PET modal images.

[0079] Specifically, firstly, the appearance vector s of the MRI modality image is MRI and the appearance vector s of the PET modality image PET The disease-related features are enhanced by the self-attention mechanism to obtain the enhanced s MRI and PET , and then the enhanced s is transformed into MRIand PET Fusion is performed to obtain a complementary appearance vector s all , the formula is as follows:

[0080] s all =cross_att(self_att(s MRI ), self_att(s PET ))

[0081] Among them, self_att(·) represents the self-attention mechanism, and cross_att(·) represents the mutual attention mechanism.

[0082] Based on the complementary appearance vector s all and MRI anatomical structure vector a MRI , the adaptive instance normalization operation is used to generate the disease diagnosis feature F. The disease diagnosis feature F is then used as the input of the disease classifier to achieve accurate AD diagnosis.

[0083] Finally, the cross entropy loss L is introduced D To constrain the training process of the diagnosis-oriented modality attribute optimization strategy, the modality attributes of MRI and PET modality images are optimized. The formula is as follows:

[0084]

[0085] Among them, N means that there are N subjects in the training set, and F i represents the disease diagnostic features of the ith subject, D(·) represents the disease classifier, and D(F i ) represents the predicted diagnosis result of the i-th subject, y i represents the true diagnosis result of the i-th subject.

[0086] 4. Training MRI to PET Medical Image Synthesis Network

[0087] The training of the network in the embodiment of the present invention is divided into three stages. The first stage trains the forward modality attribute decoupling mechanism and the backward modality attribute decoupling mechanism to synthesize PET modality images. The overall loss function of this stage is:

[0088]

[0089] Among them, α 1 , α 2 , α 3 , α 4 Indicates the weight coefficient of different loss functions. When the present invention is implemented, α is set 1 is 8, α 2 is 1, α 3 is 0.1, α 4 is 0.01.

[0090] The second stage trains the diagnosis-oriented modality attribute optimization strategy to improve the quality of the synthetic PET modality images. The overall loss function of this stage is:

[0091] L second =β 1 L D

[0092] Among them, β 1 Represents the weight coefficient of the loss function. When the present invention is implemented, β is set 1 is 1.

[0093] The third stage trains the forward modal attribute decoupling mechanism and the backward modal attribute decoupling mechanism. The overall loss function of this stage is L all =L first +L second . Ultimately, high-quality PET modality images can be synthesized.

[0094] 5. Testing the MRI to PET Medical Image Synthesis Network

[0095] During testing, the MRI modality images and randomly sampled Gaussian vectors in the test set are used as inputs to the backward modality attribute decoupling mechanism to synthesize PET modality images. Specifically, the MRI modality images are first used as anatomical structure vector encoders. The randomly sampled Gaussian vector and the anatomical structure vector of the MRI modality image are then used as the PET modality generator G P The final PET modality image is synthesized from the input.

[0096] Example 2

[0097] An MRI to PET medical image synthesis device, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to perform the following method steps in embodiment 1:

[0098] Construct a forward modality attribute decoupling mechanism: decouple the corresponding appearance vector and anatomical structure vector from MRI and PET modality images respectively, and synthesize the PET modality image using the appearance vector of the PET modality image and the anatomical structure vector of the MRI modality image;

[0099] Construct a backward modality attribute decoupling mechanism: use Gaussian vectors to learn the distribution of appearance vectors of PET modality images, and use Gaussian vectors as appearance vectors of PET modality images to guide MRI to PET medical image synthesis during the test phase;

[0100] Design a diagnosis-oriented modality attribute optimization strategy: Utilize the complementary disease-related information between MRI and PET modality images, predict diagnostic information to optimize the modality attributes of MRI and PET modality images, and then synthesize PET modality images.

[0101] Among them, the corresponding appearance vector and anatomical structure vector are decoupled from the MRI and PET modality images respectively as follows:

[0102] The MRI modality image IM is obtained by the appearance vector encoder and the anatomical structure vector encoder respectively. RI and PET modality images, P ET The appearance vector and anatomical structure vector are as follows:

[0103]

[0104] in, and Represent the anatomical structure vector encoders of MRI modality images and PET modality images respectively, which are composed of three stacked convolutional layers, and They represent the appearance vector encoders of MRI and PET modality images, respectively, which are composed of four stacked convolutional layers, two fully connected layers, and reparameterization operations. MRI and a PET They represent the anatomical structure vectors obtained after decoupling the MRI modality image and the PET modality image, respectively. MRI and PET They represent the appearance vectors obtained after decoupling the MRI modality image and the PET modality image, respectively.

[0105] The PET modality image is synthesized by using the appearance vector of the PET modality image and the anatomical structure vector of the MRI modality image as follows:

[0106] A MRI and PET As input to the PET modality generator GP to synthesize PET modality images Will s MRI and a PET As the MRI modality generator G M The input is used to synthesize MRI modality images as follows:

[0107]

[0108] Among them, GP and G M Respectively, they represent the generators for MRI modality images and PET modality images consisting of three stacked convolutional layers and adaptive instance normalization layers;

[0109] AMRI and MRI As the MRI modality generator G M The input is used to synthesize MRI modality images A PET and PET As input to the PET modality generator GP to synthesize PET modality images The formula is as follows:

[0110]

[0111] Among them, the distribution of the appearance vector of the PET modality image is learned by using the Gaussian vector, and the Gaussian vector is used as the appearance vector of the PET modality image to guide the synthesis of MRI to PET medical images in the test phase. Specifically:

[0112] The anatomical structure vector a MRI and a randomly sampled Gaussian vector N(z) as the PET modality generator G P The input to predict PET modality images as follows:

[0113]

[0114] The synthesized PET modality image As the PET modality image appearance vector encoder G P The input is the appearance vector

[0115] Among them, the complementary disease-related information between MRI and PET modality images is used to predict diagnostic information to optimize the modality properties of MRI and PET modality images, and then synthesize PET modality images specifically as follows:

[0116] The appearance vector s of the MRI modality image MRI and the appearance vector s of the PET modality image PET The disease-related features are enhanced by the self-attention mechanism to obtain the enhanced s MRI and PET , using the mutual attention mechanism to enhance the s MRI and PET Fusion is performed to obtain a complementary appearance vector s all , the formula is as follows:

[0117] s all =cross_att(self_att(s MRI ), self_att(s PET ))

[0118] Among them, self_att(·) represents the self-attention mechanism, cross_att(·) represents the mutual attention mechanism, based on the complementary appearance vector s all and MRI anatomical structure vector a MRI , the disease diagnosis feature F is generated by using the adaptive instance normalization operation, and the disease diagnosis feature F is used as the input of the disease classifier to achieve accurate AD diagnosis.

[0119] The device further includes: a training mechanism for training a MRI to PET medical image synthesis network;

[0120] The first stage trains the forward modality attribute decoupling mechanism and the backward modality attribute decoupling mechanism to synthesize PET modality images. The overall loss function of this stage is:

[0121]

[0122] Among them, α 1 , α 2 , α 3 , α 4 Represents the weight coefficients of different loss functions;

[0123] The second stage trains the diagnosis-oriented modality attribute optimization strategy to improve the quality of the synthetic PET modality images. The overall loss function of this stage is:

[0124] L second =β 1 L D

[0125] Among them, β 1 Represents the weight coefficient of the loss function;

[0126] The third stage trains the forward modal attribute decoupling mechanism and the backward modal attribute decoupling mechanism. The overall loss function of this stage is L all =L first +L second .

[0127] The device also includes: using the MRI modality image as an anatomical structure vector encoder The randomly sampled Gaussian vector and the anatomical structure vector of the MRI modality image are used as the PET modality generator G P The final PET modality image is synthesized from the input.

[0128] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0129] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0130] The data signal is transmitted between the memory and the processor via a bus, which is not described in detail in the embodiment of the present invention.

[0131] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0132] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0133] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0134] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0135] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0136] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0137] Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for synthesizing MRI to PET medical images, characterized in that: The method comprises: Construct a forward modality attribute decoupling mechanism: decouple the corresponding appearance vector and anatomical structure vector from MRI and PET modality images respectively, and synthesize the PET modality image using the appearance vector of the PET modality image and the anatomical structure vector of the MRI modality image; Construct a backward modality attribute decoupling mechanism: use Gaussian vectors to learn the distribution of appearance vectors of PET modality images, and use Gaussian vectors as appearance vectors of PET modality images to guide MRI to PET medical image synthesis during the test phase; Design a diagnosis-oriented modality attribute optimization strategy: Utilize the complementary disease-related information between MRI and PET modality images, predict diagnostic information to optimize the modality attributes of MRI and PET modality images, and then synthesize PET modality images.

2. The method for synthesizing MRI to PET medical images according to claim 1, characterized in that: The decoupling of the corresponding appearance vector and anatomical structure vector from the MRI and PET modality images is specifically as follows: The MRI modality image IM is obtained by the appearance vector encoder and the anatomical structure vector encoder respectively. RI and PET modality images I PET The appearance vector and anatomical structure vector are as follows: in, and Represent the anatomical structure vector encoders of MRI modality images and PET modality images respectively, which are composed of three stacked convolutional layers, and They represent the appearance vector encoders of MRI and PET modality images, respectively, which are composed of four stacked convolutional layers, two fully connected layers, and reparameterization operations. MRI and a PET They represent the anatomical structure vectors obtained after decoupling the MRI modality image and the PET modality image, respectively. MRI and PET They represent the appearance vectors obtained after decoupling the MRI modality image and the PET modality image, respectively.

3. The method for synthesizing MRI to PET medical images according to claim 1, characterized in that: The method of synthesizing the PET modality image by using the appearance vector of the PET modality image and the anatomical structure vector of the MRI modality image is specifically as follows: A MRI and PET As the PET modality generator G P The input is used to synthesize PET modality images Will s MRI and a PET As the MRI modality generator G M The input is used to synthesize MRI modality images as follows: Among them, G P and G M Respectively, they represent the generators for MRI modality images and PET modality images consisting of three stacked convolutional layers and adaptive instance normalization layers; A MRI and MRI As the MRI modality generator G M The input is used to synthesize MRI modality images A PET and PET As the PET modality generator G P The input is used to synthesize PET modality images The formula is as follows:

4. The method for synthesizing MRI to PET medical images according to claim 1, characterized in that: The method of using Gaussian vectors to learn the distribution of appearance vectors of PET modality images and using Gaussian vectors as appearance vectors of PET modality images to guide MRI to PET medical image synthesis in the test phase is as follows: The anatomical structure vector a MRI and the randomly sampled Gaussian vector N(z) as the input of the PET modality generator GP to predict the PET modality image as follows: The synthesized PET modality image As the input of the PET modality image appearance vector encoder GP, the appearance vector 5. The method for synthesizing MRI to PET medical images according to claim 1, characterized in that: The method of using complementary disease-related information between MRI and PET modality images to predict diagnostic information to optimize modality properties of MRI and PET modality images and then synthesizing PET modality images is specifically as follows: The appearance vector s of the MRI modality image MRI and the appearance vector s of the PET modality image PET The disease-related features are enhanced by the self-attention mechanism to obtain the enhanced s MRI and PET , using the mutual attention mechanism to enhance the s MRI and PET The complementary appearance vector sa is obtained by fusion ll , the formula is as follows: s all =cross_att(self_att(s MRI ), self_att(s PET )) Among them, self_att(·) represents the self-attention mechanism, cross_att(·) represents the mutual attention mechanism, based on the complementary appearance vector s all and MRI anatomical structure vector a MRI , the disease diagnosis feature F is generated by using the adaptive instance normalization operation, and the disease diagnosis feature F is used as the input of the disease classifier to achieve accurate AD diagnosis.

6. The method for synthesizing MRI to PET medical images according to claim 1, characterized in that: The method further comprises: a training mechanism for training an MRI to PET medical image synthesis network; The first stage trains the forward modality attribute decoupling mechanism and the backward modality attribute decoupling mechanism to synthesize PET modality images. The overall loss function of this stage is: Among them, α1, α2, α3, and α4 represent the weight coefficients of different loss functions; The second stage trains the diagnosis-oriented modality attribute optimization strategy to improve the quality of the synthetic PET modality images. The overall loss function of this stage is: L second =β1L D Among them, β1 represents the weight coefficient of the loss function; The third stage trains the forward modal attribute decoupling mechanism and the backward modal attribute decoupling mechanism. The overall loss function of this stage is L all =L first +L second .

7. The method for synthesizing MRI to PET medical images according to claim 1, characterized in that: The method further comprises: using the MRI modality image as an anatomical structure vector encoder The randomly sampled Gaussian vector and the anatomical structure vector of the MRI modality image are used as the PET modality generator G P The final PET modality image is synthesized from the input.

8. An MRI to PET medical image synthesis device, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods described in claims 11-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 7.

Images