Longitudinal magnetic resonance image generation model training method, generation method and device

Abstract

The present application provides a training method, a generation method, and an apparatus for a longitudinal magnetic resonance image generation model, and relates to the field of image data processing technology. The training method includes: gradually adding noise to the longitudinal magnetic resonance prediction image during the forward diffusion process to obtain the noise image of each forward step, and according to the last noise image and baseline image that conform to the Gaussian distribution, gradually obtaining the synthetic image of each reverse step based on a learnable conditional encoder and a generator during the reverse generation process, and after each synthetic image of a reverse step is obtained, the synthetic image of the reverse step is posteriorly sampled to obtain a synthetic noise image; and gradually inputting the noise image of each reverse step and the synthetic noise image into a discriminator for discrimination. The present application can improve the training efficiency and effectiveness of the longitudinal magnetic resonance image generation model, can ensure the diversity of the generated synthetic images, and can ensure that the generated synthetic image and the magnetic resonance baseline image belong to the same subject.

Description

Technical Field

[0001] The present application relates to the field of image data processing technology, and in particular to a longitudinal magnetic resonance image generation model training method, generation method and device. Background Art

[0002] Longitudinal MRI image generation is a crucial research area in medical imaging analysis. It aims to utilize advanced generative models (such as generative adversarial networks (GANs), diffusion models, or variational autoencoders (VAEs)) to simulate the evolution of an individual's brain at different time points, generating highly realistic longitudinal MRI composite images that can be used to predict future brain changes. The resulting predicted longitudinal MRI composite images have important applications in areas such as disease progression prediction, treatment efficacy evaluation, data augmentation, and personalized medicine. For example, in disease progression simulations, generative models can predict brain changes at future time points based on a patient's baseline MRI images, thereby helping physicians formulate intervention strategies in advance.

[0003] Currently, the commonly used generative models mentioned above generate high-quality longitudinal MRI composite images by learning the spatial and temporal features of MRI images. However, generative adversarial networks are prone to mode collapse, resulting in low diversity in the generated images, while diffusion models are prone to slow sampling speeds. Furthermore, all of these generative models cannot guarantee that the predicted longitudinal MRI composite images and the baseline MRI images belong to the same subject, thus hindering the reliability and effectiveness of the predicted longitudinal MRI composite images.

[0004] Therefore, there is an urgent need to design a method that can improve the training efficiency of the longitudinal magnetic resonance image generation model, ensure the diversity of the generated synthetic images, and ensure that the generated synthetic images and the magnetic resonance baseline images belong to the same subject. Summary of the Invention

[0005] In view of this, embodiments of the present application provide a longitudinal magnetic resonance image generation model training method, generation method and device to eliminate or improve one or more defects in the prior art.

[0006] One aspect of the present application provides a method for training a longitudinal magnetic resonance image generation model, comprising:

[0007] The diffusion generative adversarial network is iteratively trained for at least one round, and a model training step is performed in each round, wherein the model training step includes: gradually adding noise to the longitudinal magnetic resonance prediction image of the subject in each forward step of the forward diffusion process of the diffusion generative adversarial network to obtain a noise image corresponding to each forward step, and stopping the noise addition after obtaining a noise image that conforms to a Gaussian distribution; based on the noise image that conforms to the Gaussian distribution and the magnetic resonance baseline image of the subject, gradually obtaining a longitudinal magnetic resonance synthetic image corresponding to each backward step based on a learnable conditional encoder and a generator in a backward generation process of the diffusion generative adversarial network, and after each longitudinal magnetic resonance synthetic image corresponding to a backward step is obtained, posterior sampling is performed on the longitudinal magnetic resonance synthetic image corresponding to the backward step to obtain a synthetic noise image corresponding to the backward step; the synthetic noise image of each backward step and the noise image of the forward step corresponding to the backward step are gradually input into a discriminator, so that the discriminator outputs discrimination result data corresponding to each backward step to optimize the discriminator and the generator;

[0008] The learnable conditional encoder and the generator in the iteratively trained diffusion generative adversarial network are used as a longitudinal magnetic resonance image generation model.

[0009] In some embodiments of the present application, according to the noise image conforming to the Gaussian distribution and the magnetic resonance baseline image of the subject, in the reverse generation process of the diffusion generative adversarial network, gradually obtaining the longitudinal magnetic resonance synthetic image corresponding to each reverse step based on the learnable conditional encoder and the generator, and after each acquisition of the longitudinal magnetic resonance synthetic image corresponding to a reverse step, performing posterior sampling on the longitudinal magnetic resonance synthetic image corresponding to the reverse step to obtain the synthetic noise image corresponding to the reverse step, including:

[0010] In each reverse step of the reverse generation process of the diffusion generative adversarial network, performing an image synthesis training sub-step respectively;

[0011] The image synthesis training sub-step includes:

[0012] In the current backward step, the noise image conforming to the Gaussian distribution, the MRI baseline image of the subject, and auxiliary data of the subject are input into the learnable conditional encoder, so that the learnable conditional encoder outputs corresponding encoded data; wherein the auxiliary data includes: the age, gender, and diagnosis label of the subject;

[0013] Inputting the encoded data into the generator so that the generator outputs a longitudinal magnetic resonance composite image corresponding to the current reverse step;

[0014] Performing posterior sampling on the longitudinal magnetic resonance synthetic image corresponding to the current reverse step to obtain a synthetic noise image corresponding to the reverse step;

[0015] Calculating the generator loss corresponding to the current backward step according to the longitudinal magnetic resonance synthetic image corresponding to the current backward step and the longitudinal magnetic resonance predicted image of the subject;

[0016] Update the model parameters of the generator based on the generator loss corresponding to the current backward step.

[0017] In some embodiments of the present application, the generator loss includes: absolute mean error loss and training loss of the generator;

[0018] Correspondingly, calculating the generator loss corresponding to the current backward step based on the longitudinal magnetic resonance synthetic image corresponding to the current backward step and the longitudinal magnetic resonance predicted image of the subject includes:

[0019] Obtaining an absolute average error between the longitudinal magnetic resonance synthesized image of the current reverse step and the longitudinal magnetic resonance predicted image of the current round as the absolute average error loss corresponding to the current reverse step;

[0020] And, according to the longitudinal magnetic resonance prediction image of the current round and the noise image of the current reverse step, the training loss of the generator of the current reverse step is obtained.

[0021] In some embodiments of the present application, the generator loss further includes: a correlation relationship loss between time interval and image difference;

[0022] Correspondingly, calculating the generator loss corresponding to the current backward step based on the longitudinal magnetic resonance synthetic image corresponding to the current backward step and the longitudinal magnetic resonance predicted image of the subject further includes:

[0023] The loss of the correlation relationship between the time interval corresponding to the current reverse step and the image difference is obtained based on the longitudinal magnetic resonance synthetic image of the current reverse step, the longitudinal magnetic resonance predicted image of the current round, the age of the subject corresponding to the magnetic resonance baseline image when the magnetic resonance baseline image is acquired, the age of the subject corresponding to the longitudinal magnetic resonance predicted image of the current round, and the preset maximum age and minimum age values.

[0024] In some embodiments of the present application, updating the model parameters of the generator based on the generator loss corresponding to the current backward step includes:

[0025] The generator loss corresponding to the current backward step is determined according to the training loss of the generator corresponding to the current backward step, the product of the absolute mean error loss and a preset first hyperparameter, and the product of the correlation relationship loss between the time interval and the image difference and a preset second hyperparameter, and the model parameters of the generator are updated based on the generator loss corresponding to the current backward step.

[0026] In some embodiments of the present application, before performing at least one round of iterative training on the diffusion generative adversarial network, the method further includes:

[0027] Acquiring an original magnetic resonance image of each subject acquired at a historical time point and an original longitudinal magnetic resonance prediction image of each subject acquired at a target time point after the historical time point;

[0028] performing image preprocessing, data cleaning, and normalization on the original magnetic resonance images and the original longitudinal magnetic resonance predicted images of each subject;

[0029] The original magnetic resonance images corresponding to each subject after the image preprocessing, data cleaning and normalization processing are respectively used as the magnetic resonance reference images corresponding to each subject, and the original longitudinal magnetic resonance prediction images corresponding to each subject after the image preprocessing, data cleaning and normalization processing are respectively used as the longitudinal magnetic resonance prediction images corresponding to each subject.

[0030] A second aspect of the present application provides a method for generating a longitudinal magnetic resonance image, comprising:

[0031] A noise image conforming to a Gaussian distribution, a magnetic resonance baseline image of the current target subject, and auxiliary data of the target subject are input into a longitudinal magnetic resonance image generation model, so that a learning conditional encoder and a generator based on the longitudinal magnetic resonance image generation model gradually obtain longitudinal magnetic resonance synthetic images corresponding to each step, and after each longitudinal magnetic resonance synthetic image corresponding to a reverse step is obtained, a posteriori sampling is performed on the longitudinal magnetic resonance synthetic image corresponding to the reverse step to obtain a synthetic noise image corresponding to the reverse step; wherein the longitudinal magnetic resonance image generation model is pre-trained based on the longitudinal magnetic resonance image generation model training method; and the auxiliary data includes: the age, gender, and diagnostic label of the target subject;

[0032] The longitudinal magnetic resonance composite image output by the generator in the last backward step is used as the longitudinal magnetic resonance image prediction result data of the target subject at the target time point.

[0033] A third aspect of the present application provides a longitudinal magnetic resonance image generation model training device, comprising:

[0034] An iterative training module is configured to perform at least one round of iterative training on the diffusion generative adversarial network, and execute a model training step in each round, wherein the model training step includes: gradually adding noise to the longitudinal magnetic resonance prediction image of the subject in each forward step of the forward diffusion process of the diffusion generative adversarial network to obtain a noise image corresponding to each forward step, and stopping the noise addition after obtaining a noise image that conforms to a Gaussian distribution; based on the noise image that conforms to the Gaussian distribution and the magnetic resonance baseline image of the subject, gradually obtaining a longitudinal magnetic resonance synthetic image corresponding to each reverse step in the reverse generation process of the diffusion generative adversarial network based on a learnable conditional encoder and a generator, and after each longitudinal magnetic resonance synthetic image corresponding to a reverse step is obtained, performing posterior sampling on the longitudinal magnetic resonance synthetic image corresponding to the reverse step to obtain a synthetic noise image corresponding to the reverse step; and gradually inputting the synthetic noise image of each reverse step and the noise image of the forward step corresponding to the reverse step into a discriminator, so that the discriminator outputs discrimination result data corresponding to each reverse step to optimize the discriminator and the generator;

[0035] A model determination module is used to use the learnable conditional encoder and the generator in the iteratively trained diffusion generative adversarial network as a longitudinal magnetic resonance image generation model.

[0036] A fourth aspect of the present application provides a longitudinal magnetic resonance image generating device, comprising:

[0037] A model prediction module is configured to input a noise image conforming to a Gaussian distribution, a magnetic resonance baseline image of a current target subject, and auxiliary data of the target subject into a longitudinal magnetic resonance image generation model, so that a learning conditional encoder and generator in the longitudinal magnetic resonance image generation model gradually obtain longitudinal magnetic resonance synthetic images corresponding to each step, and after each acquisition of the longitudinal magnetic resonance synthetic image corresponding to a reverse step, a posteriori sampling is performed on the longitudinal magnetic resonance synthetic image corresponding to the reverse step to obtain a synthetic noise image corresponding to the reverse step; wherein the longitudinal magnetic resonance image generation model is pre-trained based on the longitudinal magnetic resonance image generation model training method; and the auxiliary data includes: the age, gender, and diagnostic label of the target subject;

[0038] A result output module is used to use the longitudinal magnetic resonance synthetic image output by the generator in the last backward step as the longitudinal magnetic resonance image prediction result data of the target subject at the target time point.

[0039] The fifth aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the method for training a longitudinal magnetic resonance image generation model is implemented, and / or the method for generating a longitudinal magnetic resonance image is implemented.

[0040] The sixth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the longitudinal magnetic resonance image generation model training method and / or the longitudinal magnetic resonance image generation method.

[0041] The seventh aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the longitudinal magnetic resonance image generation model training method and / or the longitudinal magnetic resonance image generation method.

[0042] The longitudinal magnetic resonance image generation model training method provided in the present application performs at least one round of iterative training on the diffusion generative adversarial network, and executes the model training steps respectively in each round, wherein the model training steps include: gradually adding noise to the longitudinal magnetic resonance predicted image of the subject in each forward step of the forward diffusion process of the diffusion generative adversarial network to obtain the noise image corresponding to each forward step, and stopping the noise addition after obtaining a noise image that conforms to the Gaussian distribution; according to the noise image that conforms to the Gaussian distribution and the magnetic resonance baseline image of the subject, gradually obtaining the longitudinal magnetic resonance synthetic image corresponding to each reverse step based on the learnable conditional encoder and the generator in the reverse generation process of the diffusion generative adversarial network, and obtaining a reverse step each time. After obtaining the longitudinal magnetic resonance synthetic image corresponding to the forward step, the longitudinal magnetic resonance synthetic image corresponding to the backward step is subjected to posterior sampling to obtain the synthetic noise image corresponding to the backward step; the synthetic noise image of each backward step and the noise image of the forward step corresponding to the backward step are gradually input into the discriminator, so that the discriminator outputs the discrimination result data corresponding to each backward step respectively to optimize the discriminator and the generator; the learnable conditional encoder and the generator in the iteratively trained diffusion generative adversarial network are used as a longitudinal magnetic resonance image generation model; the training efficiency and effectiveness of the longitudinal magnetic resonance image generation model can be improved, the diversity of the generated synthetic images can be ensured, and the generated synthetic images and the magnetic resonance baseline images can be ensured to belong to the same subject.

[0043] Additional advantages, purposes, and features of the present application will be described in part in the following description and will become apparent to those skilled in the art upon study of the following or may be learned from practice of the present application. The purposes and other advantages of the present application may be achieved and obtained by the structures specifically pointed out in the specification and drawings.

[0044] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are intended to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger than other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0046] Figure 1 This is a first flow chart of a method for training a longitudinal magnetic resonance image generation model in one embodiment of the present application.

[0047] Figure 2 This is a first flow chart of the model training step in step 100 in one embodiment of the present application.

[0048] Figure 3 This is a schematic diagram of the training process of a longitudinal magnetic resonance image generation model based on a diffusion generative adversarial network in an example of this application.

[0049] Figure 4 This is a second flow chart of the model training step in step 100 in one embodiment of the present application.

[0050] Figure 5 This is a first flow chart of the image synthesis training sub-step in step 121 in one embodiment of the present application.

[0051] Figure 6 This is a second flow chart of the image synthesis training sub-step in step 121 in one embodiment of the present application.

[0052] Figure 7 This is a second flow chart of the longitudinal magnetic resonance image generation model training method in one embodiment of the present application.

[0053] Figure 8 This is a flowchart of a longitudinal magnetic resonance image generation model training method in an application example of the present application.

[0054] Figure 9 Schematic diagram of the comparison results of the longitudinal data after three years (GT (3 years)) provided in this application.

[0055] Figure 10 Schematic diagram of the comparison results of the longitudinal data after six years (GT (6 years)) provided in this application.

[0056] Figure 11 FIG. 1 is a flow chart of a method for generating a longitudinal magnetic resonance image in an embodiment of the present application.

[0057] Figure 12 Schematic diagram of the structure of a longitudinal magnetic resonance image generation model training device in one embodiment of the present application.

[0058] Figure 13 Schematic diagram of the structure of a longitudinal magnetic resonance image generating device in one embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail in conjunction with the embodiments and drawings. Here, the illustrative embodiments of this application and their descriptions are used to explain this application, but are not intended to limit this application.

[0060] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show structures and / or processing steps that are closely related to the scheme according to the present application, while other details that are not closely related to the present application are omitted.

[0061] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0062] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0063] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0064] Commonly used longitudinal MRI generation methods include generative adversarial networks (such as conditional GANs, CycleGAN, and 3DGANs), diffusion models (such as denoising diffusion probability model DDPM), and variational autoencoders (such as time series VAE and conditional VAE). These methods generate high-quality longitudinal images by learning the spatial and temporal features of MRI. Conditional GANs (i.e., conditional generative adversarial networks) in generative adversarial networks (GANs) generate images at future time points that match the conditions by introducing conditional information such as a baseline MRI image and time point. CycleGAN, a variant of GANs, is designed to learn image-to-image translation between two different domains without paired training data. It does not require corresponding images of the same scene or object in the two different domains. CycleGAN learns the mapping between images at different time points through a cycle consistency loss, without the need for paired data. 3D GANs are three-dimensional generative adversarial networks used to generate three-dimensional geometric data. The denoising diffusion probability model (DDPM) in diffusion models generates high-quality images by gradually adding and removing noise, capable of generating detailed MRI images. The conditional diffusion model further introduces conditional information to generate future images that are consistent with the baseline image and time point. Time series in variational autoencoders (VAEs) combine time series modeling (such as recurrent neural networks (RNNs) and long short-term memory (LSTMs) artificial neural networks (ANNs)) with VAEs to generate continuous longitudinal MRI images. Conditional variational autoencoders (VAEs) introduce conditional information to generate future images that match the baseline image and time point.

[0065] Traditionally, studying brain changes over time relies on the collection and analysis of longitudinal magnetic resonance imaging data. However, data at certain time points may be missing or incomplete, limiting the accuracy of depicting brain changes over time. To address these issues, generative models are used to simulate changes in an individual's brain at different time points to generate realistic magnetic resonance images. In addition, synthetic longitudinal magnetic resonance images are used for disease progression prediction, treatment effect evaluation, and data augmentation. However, generative adversarial networks are prone to mode collapse, resulting in low diversity in generated images; diffusion models have slow sampling speeds; and none of the above generative models can guarantee that the predicted longitudinal magnetic resonance synthetic images and the magnetic resonance baseline images belong to the same subject.

[0066] Based on this, in order to design a method that can improve the training efficiency of the longitudinal magnetic resonance image generation model, ensure the diversity of the generated synthetic images, and ensure that the generated synthetic images and the magnetic resonance baseline images belong to the same subject, the embodiments of the present application respectively provide a longitudinal magnetic resonance image generation model training method, a longitudinal magnetic resonance image generation method, a longitudinal magnetic resonance image generation model training device for executing the longitudinal magnetic resonance image generation model training method, a longitudinal magnetic resonance image generation device for executing the longitudinal magnetic resonance image generation method, a physical device, a computer-readable storage medium and a computer program product, which can improve the training efficiency and effectiveness of the longitudinal magnetic resonance image generation model, ensure the diversity of the generated synthetic images, and ensure that the generated synthetic images and the magnetic resonance baseline images belong to the same subject.

[0067] The details are described in detail through the following examples.

[0068] Based on this, the embodiment of the present application provides a longitudinal magnetic resonance image generation model training method that can be implemented by a longitudinal magnetic resonance image generation model training device, see Figure 1 The longitudinal magnetic resonance image generation model training method specifically includes the following contents:

[0069] Step 100: Perform at least one round of iterative training on the diffusion generative adversarial network, and perform a model training step in each round, wherein the model training step includes: gradually adding noise to the longitudinal magnetic resonance prediction image of the subject in each forward step of the forward diffusion process of the diffusion generative adversarial network to obtain a noise image corresponding to each forward step, and stopping the noise addition after obtaining a noise image that conforms to a Gaussian distribution; based on the noise image that conforms to the Gaussian distribution and the magnetic resonance baseline image of the subject, gradually obtaining a longitudinal magnetic resonance synthetic image corresponding to each backward step based on a learnable conditional encoder and a generator in a backward generation process of the diffusion generative adversarial network, and after each longitudinal magnetic resonance synthetic image corresponding to a backward step is obtained, posterior sampling is performed on the longitudinal magnetic resonance synthetic image corresponding to the backward step to obtain a synthetic noise image corresponding to the backward step; and gradually inputting the synthetic noise image of each backward step and the noise image of the forward step corresponding to the backward step into a discriminator, so that the discriminator outputs discrimination result data corresponding to each backward step to optimize the discriminator and the generator.

[0070] First of all, it should be noted that the longitudinal magnetic resonance image generation model training method provided in the embodiment of the present application refers to a model training method based on a diffusion generative adversarial network, that is, the model training framework of the longitudinal magnetic resonance image generation model is a diffusion generative adversarial network.

[0071] The Diffusion GAN (Diffusion GAN) is a novel generative model that combines diffusion models with generative adversarial networks (GANs). It aims to overcome the limitations of traditional GANs, such as training instability and mode collapse, as well as the slow data generation problem of diffusion models. The core idea of this model is to gradually generate data through a diffusion process, thereby introducing more controllable and stable generation. Specifically, the Diffusion GAN can be divided into two main stages: the forward diffusion process and the backward generation process. In the forward diffusion process, real data samples are converted into a noise distribution through a series of steps of gradually adding Gaussian noise, similar to the noise addition mechanism in the diffusion model. In the backward generation process, the generator network reconstructs real data samples from the learned noise distribution, then generates noisy samples through posterior sampling. Finally, a discriminator is used to perform adversarial training on the noisy samples in each step, ensuring the authenticity of the generated samples at each step. The Diffusion GAN can generate high-quality samples, maintain training stability and diversity, and accelerate sampling.

[0072] That is to say, for the longitudinal magnetic resonance image generation method using a generative adversarial network, since the generative adversarial network directly synthesizes images at one time, its generator only learns to generate a few image patterns and ignores the diversity of the entire data distribution, thus leading to the problem of low diversity of the generated images. The embodiment of the present application adopts a diffusion generative adversarial network, in which the diffusion model gradually denoises and restores the data distribution. Each step models the details of the real distribution and is unlikely to fall into a local optimum that only generates a few patterns. Therefore, it can effectively ensure the diversity of the generated synthetic images.

[0073] In addition, since the noise added at each step in the forward propagation process of the diffusion model obeys a Gaussian normal distribution, each diffusion step is very small and the number of steps is large, so it takes a long time to infer to generate data. Based on this, the longitudinal magnetic resonance image generation method using the diffusion model has the problem of slow sampling speed. The embodiment of the present application adopts a diffusion generative adversarial network. According to the situation that more data in real applications obey a multimodal distribution, the diffusion process of each step of the diffusion generative adversarial network adopted in the embodiment of the present application does not have to obey a Gaussian normal distribution. A conditional generator is used to fit the diffusion process of each step. Since each step is a non-Gaussian normal distribution, the number of diffusion steps can be reduced, thereby achieving fast data generation, thereby improving the training efficiency of the longitudinal magnetic resonance image generation model.

[0074] In step 100, see Figure 2 The model training step in step 100 specifically includes the following contents:

[0075] Step 110: gradually adding noise to the longitudinal magnetic resonance prediction image of the subject in each forward step of the forward diffusion process of the diffusion generative adversarial network to obtain a noise image corresponding to each forward step, and stopping the noise addition after obtaining a noise image that conforms to the Gaussian distribution.

[0076] It can be understood that a forward step refers to a step in the forward diffusion process. In step 100, noise is gradually added to the subject's predicted longitudinal magnetic resonance image in each forward step of the forward diffusion process of the diffusion generative adversarial network until the newly added noise image conforms to the Gaussian distribution. The total number of forward steps at this time can be recorded as T.

[0077] Step 120: According to the noise image that conforms to the Gaussian distribution and the magnetic resonance baseline image of the subject, in the reverse generation process of the diffusion generative adversarial network, the longitudinal magnetic resonance synthetic image corresponding to each reverse step is gradually obtained based on the learnable conditional encoder and the generator, and after each time the longitudinal magnetic resonance synthetic image corresponding to a reverse step is obtained, the longitudinal magnetic resonance synthetic image corresponding to the reverse step is posteriorly sampled to obtain the synthetic noise image corresponding to the reverse step.

[0078] It can be understood that the reverse steps refer to the steps in the reverse generation process, and the number of the reverse steps is the same as the number of the forward steps and corresponds one to one.

[0079] Step 130: gradually input the synthetic noise image of each backward step and the noise image of the forward step corresponding to the backward step into the discriminator, so that the discriminator outputs the discrimination result data corresponding to each backward step respectively to optimize the discriminator and the generator.

[0080] In one example, see Figure 3 , each forward step is A1, A2, A3 and A4 in sequence; in the forward step A1, the forward diffusion process corresponding to the forward step A1 of step 110 is carried out. , the noise image x1 corresponding to the longitudinal magnetic resonance prediction image x0 is obtained; in the forward step A2, the forward diffusion process corresponding to the forward step A2 of step 120 is performed. Based on the noise image x1, the noise image x2 corresponding to the longitudinal magnetic resonance prediction image x0 is obtained; in the forward step A3, the forward diffusion process corresponding to the forward step A3 of step 120 is performed. Based on the noise image x2, the noise image x3 corresponding to the longitudinal magnetic resonance prediction image x0 is obtained; in the forward step A4, the forward diffusion process corresponding to the forward step A3 of step 120 is performed. Based on the noise image x3, the noise image x4 corresponding to the longitudinal magnetic resonance prediction image x0 is obtained. If the noise image x4 is the first noise image that conforms to the Gaussian distribution in the current round, the forward expansion process is stopped. At this time, there are 4 forward steps, that is, T is equal to 4.

[0081] Correspondingly, in this example, the number of reverse steps is also 4, and the reverse steps are B1, B2, B3 and B4 in sequence. In the reverse step B1, the longitudinal magnetic resonance synthesis image corresponding to the longitudinal magnetic resonance prediction image x0 is obtained through step 120. Since the reverse step B1 corresponds to the forward step A4, and the forward step A4 obtains the noise image x4, in the reverse step B1, the longitudinal magnetic resonance synthesis image corresponding to the reverse step B1 is obtained. Perform posterior sampling to obtain the synthetic noise image corresponding to the reverse step B1 ; At this time, the synthetic noise image The noise image x4 forms a set of data to be judged, and the set of data to be judged is input into the discriminator in step 130, so that the discriminator outputs a synthetic noise image True or false judgment result data.

[0082] In the reverse step B2, the longitudinal MRI synthetic image corresponding to the longitudinal MRI prediction image x0 is obtained through step 120. Since the reverse step B2 corresponds to the forward step A3, and the forward step A3 obtains the noise image x3, in the reverse step B2, the longitudinal magnetic resonance synthesis image corresponding to the reverse step B2 is obtained. Perform posterior sampling to obtain the synthetic noise image corresponding to the reverse step B2 ; At this time, the synthetic noise image The noise image x3 forms a set of data to be judged, and the set of data to be judged is input into the judgement device through step 130. , so that the discriminator outputs a synthetic noise image True or false judgment result data.

[0083] In the reverse step B2, the longitudinal MRI synthetic image corresponding to the longitudinal MRI prediction image x0 is obtained through step 120. Since the reverse step B2 corresponds to the forward step A3, and the forward step A3 obtains the noise image x3, in the reverse step B2, the longitudinal magnetic resonance synthesis image corresponding to the reverse step B2 is obtained. Perform posterior sampling to obtain the synthetic noise image corresponding to the reverse step B2 ; At this time, the synthetic noise image The noise image x3 forms a set of data to be judged, and the set of data to be judged is input into the discriminator in step 130, so that the discriminator outputs a synthetic noise image True or false judgment result data.

[0084] In the reverse step B3, the longitudinal MRI synthetic image corresponding to the longitudinal MRI prediction image x0 is obtained through step 120. Since the reverse step B3 corresponds to the forward step A2, and the forward step A2 obtains the noise image x2, in the reverse step B3, the longitudinal magnetic resonance synthesis image corresponding to the reverse step B3 is obtained. Perform posterior sampling to obtain the synthetic noise image corresponding to the reverse step B3 ; At this time, the synthetic noise image The noise image x2 forms a set of data to be judged, and the set of data to be judged is input into the discriminator in step 130, so that the discriminator outputs a synthetic noise image True or false judgment result data.

[0085] In the reverse step B4, the longitudinal MRI synthetic image corresponding to the longitudinal MRI prediction image x0 is obtained through step 120. Since the reverse step B4 corresponds to the forward step A1, and the forward step A1 obtains the noise image x1, in the reverse step B4, the longitudinal magnetic resonance synthesis image corresponding to the reverse step B4 is obtained. Perform posterior sampling to obtain the synthetic noise image corresponding to the reverse step B4 ; At this time, the synthetic noise image The noise image x1 forms a set of data to be judged, and the set of data to be judged is input into the discriminator in step 130, so that the discriminator outputs a synthetic noise image True or false judgment result data.

[0086] Step 200: Using the learnable conditional encoder and the generator in the iteratively trained diffusion generative adversarial network as a longitudinal magnetic resonance image generation model.

[0087] It can be understood that the longitudinal magnetic resonance image generation model refers to a model for generating and outputting a longitudinal magnetic resonance composite image of the subject according to an input magnetic resonance baseline image of the subject.

[0088] In one or more embodiments of the present application, the magnetic resonance baseline image can be simply referred to as the baseline image, and the model of the longitudinal magnetic resonance synthetic image can be simply referred to as the synthetic image; the longitudinal magnetic resonance predicted image is actually a real longitudinal magnetic resonance image that serves as a label of the longitudinal magnetic resonance synthetic image during the model training process, and therefore can be simply referred to as the real image.

[0089] The baseline MRI image is a longitudinal MRI image of the subject at a certain historical time, and the predicted longitudinal MRI image is a longitudinal MRI image of the subject acquired at a target time after the historical time. The time interval between the historical time and the target time can be set according to actual application requirements.

[0090] From the above description, it can be seen that the longitudinal magnetic resonance image generation model training method provided in the embodiment of the present application can improve the training efficiency and effectiveness of the longitudinal magnetic resonance image generation model and ensure the diversity of generated synthetic images.

[0091] At the same time, the diffusion generative adversarial network only considers the synthesis of natural images, but does not consider the validity of the synthesized images and the identity of the subjects. In the process of synthesizing medical images, it is necessary not only to ensure the authenticity of the synthesized images, but also to ensure that the synthesized images come from the same subject. Therefore, the longitudinal magnetic resonance image generation model training method provided in the embodiment adopts a learnable conditional encoder and adds information such as the baseline image of the subject for encoding constraints, so that the generator can generate a synthesized image of the same subject, thereby maintaining the consistency of the subject, that is, it can ensure that the generated synthesized image and the magnetic resonance baseline image belong to the same subject, thereby improving the application effectiveness and reliability of the longitudinal magnetic resonance synthesized image generated by the longitudinal magnetic resonance image generation model obtained through training.

[0092] In order to further ensure that the generated synthetic image and the magnetic resonance baseline image belong to the same subject, in the longitudinal magnetic resonance image generation model training method provided in the embodiment of the present application, see Figure 4 , the step 120 specifically includes the following contents:

[0093] Step 121: In each backward step of the backward generation process of the diffusion generative adversarial network, an image synthesis training sub-step is executed respectively; wherein the image synthesis training sub-step includes: in the current backward step, the noise image conforming to the Gaussian distribution, the magnetic resonance baseline image of the subject and the auxiliary data of the subject are input into the learnable conditional encoder, so that the learnable conditional encoder outputs corresponding encoded data; wherein the auxiliary data includes: the age, gender and diagnostic label of the subject; the encoded data is input into the generator, so that the generator outputs the longitudinal magnetic resonance synthetic image corresponding to the current backward step; the longitudinal magnetic resonance synthetic image corresponding to the current backward step is sampled a posteriori to obtain the synthetic noise image corresponding to the backward step; according to the longitudinal magnetic resonance synthetic image corresponding to the current backward step and the longitudinal magnetic resonance predicted image of the subject, the generator loss corresponding to the current backward step is calculated; and the model parameters of the generator are updated based on the generator loss corresponding to the current backward step.

[0094] Specifically, see Figure 5 The image synthesis training sub-step in step 121 specifically includes the following contents:

[0095] Step 1211: In the current reverse step, the noise image that conforms to the Gaussian distribution, the magnetic resonance baseline image of the subject, and the auxiliary data of the subject are input into the learnable conditional encoder, so that the learnable conditional encoder outputs corresponding encoded data; wherein the auxiliary data includes: the age, gender, and diagnostic label of the subject.

[0096] Step 1212: Input the encoded data into the generator so that the generator outputs the longitudinal magnetic resonance synthesis image corresponding to the current reverse step.

[0097] Step 1213: Perform a posteriori sampling on the longitudinal magnetic resonance synthesis image corresponding to the current reverse step to obtain a synthetic noise image corresponding to the reverse step.

[0098] Step 1214: Calculate the generator loss corresponding to the current backward step based on the longitudinal magnetic resonance synthetic image corresponding to the current backward step and the longitudinal magnetic resonance predicted image of the subject.

[0099] Step 1215: Update the model parameters of the generator based on the generator loss corresponding to the current backward step.

[0100] That is to say, in the generator network of the adversarial generative network, the embodiment of the present application proposes to add a learnable conditional encoder, add the subject's baseline image, age, gender and diagnostic label to encode the constraint model to generate images of the same person, so as to maintain the consistency of the subject.

[0101] In order to further ensure that the generated synthetic image and the magnetic resonance baseline image belong to the same subject and to ensure the continuity of adjacent slice images in the synthetic image, in a longitudinal magnetic resonance image generation model training method provided in an embodiment of the present application, the generator loss includes: absolute mean error loss and the training loss of the generator ;

[0102] For corresponding reference, see Figure 6 , the step 1214 specifically includes the following contents:

[0103] Step 1214 - 1 : Obtain the absolute mean error between the longitudinal magnetic resonance synthesized image of the current backward step and the longitudinal magnetic resonance predicted image of the current round as the absolute mean error loss corresponding to the current backward step.

[0104] Specifically, the loss function corresponding to the absolute mean error loss is shown in the following formula (1):

[0105] (1)

[0106] in, represents the longitudinal magnetic resonance composite image corresponding to the current reverse step, The longitudinal magnetic resonance prediction image of the current round is represented.

[0107] Each step in the reverse generation process constrains the image synthesized by the learnable conditional encoder to be as similar as possible to the original input. This can constrain the entire training process to converge quickly while ensuring that the synthesized image is similar to the original image.

[0108] That is to say, since the initial noise of the diffusion model is random, if the noise evolution path at different positions is unstable or uncoordinated, it will also lead to discontinuity between slice images in the synthesized 3D brain image. Therefore, in view of the problem that the existing technology easily causes discontinuity between adjacent slice images in the synthesized 3D brain image, the embodiment of the present application adopts the method of step 1214-1, firstly uses the generator to directly generate the synthetic image, and adds the noise in each diffusion step. The loss function ensures the reliability of each step of the process, and uses the discriminator to ensure that the results of the generator are more realistic.

[0109] And, step 1214-2: obtaining the training loss of the generator of the current backward step based on the longitudinal magnetic resonance prediction image of the current round and the noise image of the current backward step.

[0110] The loss function corresponding to the training loss of the generator is shown in the following formula (2):

[0111] (2)

[0112] in, represents the noise image corresponding to the t-1th step; represents the noise image corresponding to the t-th step; represents the parameters of the generator network; Represents the real noisy image mid-sampling; Represents the image obtained by posterior sampling from the image predicted by the generator sampling; represents the discriminator.

[0113] In order to further improve the rationality and effectiveness of the synthesized image, the difference between the synthesized image and the baseline image needs to increase as the time interval increases. In a longitudinal magnetic resonance image generation model training method provided in an embodiment of the present application, the generator loss also includes: a correlation loss between the time interval and the image difference;

[0114] For corresponding reference, see Figure 6 , the step 1214 further specifically includes the following contents:

[0115] Step 1214-3: Based on the longitudinal magnetic resonance composite image of the current reverse step, the longitudinal magnetic resonance predicted image of the current round, the age of the subject corresponding to the magnetic resonance baseline image, the age of the subject corresponding to the longitudinal magnetic resonance predicted image of the current round, and the preset maximum age and minimum age, obtain the correlation loss between the time interval corresponding to the current reverse step and the image difference.

[0116] The correlation loss between the time interval and the image difference is The corresponding loss function is shown in the following formula (3):

[0117] (3)

[0118] in, is the age of the subject when the baseline image was collected, and a is the age of the subject at the predicted longitudinal MRI image. They are the maximum and minimum age values in all preset longitudinal MRI prediction images and baseline images, respectively.

[0119] In order to further improve the application effectiveness and reliability of the longitudinal magnetic resonance image generation model, in a longitudinal magnetic resonance image generation model training method provided in an embodiment of the present application, see Figure 6 , the step 1215 specifically includes the following contents:

[0120] Step 1215-1: Determine the generator loss corresponding to the current backward step based on the training loss of the generator corresponding to the current backward step, the product of the absolute mean error loss and a preset first hyperparameter, and the product of the correlation relationship loss between the time interval and the image difference and a preset second hyperparameter, and update the model parameters of the generator based on the generator loss corresponding to the current backward step.

[0121] Among them, the generator loss The corresponding loss function is shown in the following formula (4):

[0122] (4)

[0123] It is the first hyperparameter preset during training; It is the second hyperparameter preset during the training process.

[0124] In order to further improve the effectiveness and reliability of the longitudinal magnetic resonance image generation model training, in a longitudinal magnetic resonance image generation model training method provided in an embodiment of the present application, see Figure 7 The longitudinal magnetic resonance image generation model training method further includes the following contents before step 100:

[0125] Step 010: Acquire the original magnetic resonance image of each subject acquired at a historical time point and the original longitudinal magnetic resonance prediction image of each subject acquired at a target time point after the historical time point.

[0126] Step 020: performing image preprocessing, data cleaning and normalization processing on the original magnetic resonance image and the original longitudinal magnetic resonance prediction image of each subject.

[0127] Image preprocessing can be divided into three parts: structural MRI preprocessing, functional MRI preprocessing, and diffusion tensor imaging preprocessing. Structural MRI preprocessing steps include: motion correction, which corrects image artifacts caused by subject movement; tissue extraction, which removes irrelevant brain areas, such as the skull, from the structural MRI image; and spatial normalization, which normalizes the image to the MNI standard space to facilitate comparisons across subjects.

[0128] Data cleaning can be done by: truncating the data to find the 99.9% threshold of the data used, assigning the number greater than the threshold as the threshold, and normalizing the data to [-1, 1].

[0129] Step 030: The original magnetic resonance images corresponding to each subject after the image preprocessing, data cleaning and normalization processing are respectively used as the magnetic resonance reference images corresponding to each subject, and the original longitudinal magnetic resonance prediction images corresponding to each subject after the image preprocessing, data cleaning and normalization processing are respectively used as the longitudinal magnetic resonance prediction images corresponding to each subject.

[0130] In order to further improve the effectiveness and reliability of the longitudinal magnetic resonance image generation model training, in a longitudinal magnetic resonance image generation model training method provided in an embodiment of the present application, see Figure 4 , the step 130 specifically includes the following contents:

[0131] Step 131: In each reverse step of the reverse generation process of the diffusion generative adversarial network, a discrimination sub-step is executed respectively; wherein the discrimination sub-step includes: in the current reverse step, the synthetic noise image corresponding to the current reverse step and the noise image of the forward step corresponding to the current reverse step are used as a set of data to be discriminated, and the data to be discriminated are respectively input into the discriminator so that the discriminator outputs the discrimination result data corresponding to the current reverse step; the model parameters of the generator are updated according to the discrimination result data corresponding to the current reverse step; and the discriminator loss corresponding to the current reverse step is calculated to update the model parameters of the discriminator based on the discriminator loss.

[0132] The loss function corresponding to the discriminator loss is shown in the following formula (5):

[0133] (5)

[0134] in, represents the forward diffusion process, Represents the reverse generation process; Represents the real noisy image mid-sampling; Represents the forward diffusion process sampling; Represents the image obtained by posterior sampling from the image predicted by the generator Sampling; that is, It refers to the loss of the longitudinal magnetic resonance prediction image when the input is calculated. It is calculated as the loss of longitudinal MR composite images.

[0135] To further illustrate the embodiment of the above-mentioned longitudinal magnetic resonance image generation model training method, the present application also provides a specific application example of the longitudinal magnetic resonance image generation model training method, which can also be regarded as a longitudinal magnetic resonance image generation model training method based on a diffusion generative adversarial network. The application example specifically includes the following content:

[0136] S1: Acquire an original magnetic resonance image of each subject acquired at a historical time point and an original longitudinal magnetic resonance prediction image of each subject acquired at a target time point after the historical time point;

[0137] The file formats of the original magnetic resonance images and the original longitudinal magnetic resonance predicted images are both nii. The nii format (an extension of the NIFTI format) is designed for multidimensional neuroimaging and can truly reflect metadata, including direction information.

[0138] S2: performing image preprocessing on the original magnetic resonance image and the original longitudinal magnetic resonance predicted image of each subject;

[0139] Image preprocessing is divided into three parts: structural MRI preprocessing, functional MRI preprocessing, and diffusion tensor imaging preprocessing. Structural MRI preprocessing steps include: head motion correction, which corrects image artifacts caused by subject movement; tissue extraction, which removes irrelevant brain areas, such as the skull, from the structural MRI image; and spatial normalization, which normalizes the image to the MNI standard space to facilitate comparison across subjects.

[0140] S3: performing data cleaning and normalization processing on the original magnetic resonance image and the original longitudinal magnetic resonance predicted image of each subject;

[0141] Data cleaning can be done by truncating the data to find the 99.9% threshold of the data used, assigning the number greater than the threshold as the threshold, and normalizing the data to [-1, 1].

[0142] S4: Constructing a diffusion generative adversarial network;

[0143] Specifically, the generator in the diffusion generative adversarial network can adopt the U-Net architecture. U-Net is an end-to-end image segmentation model based on a convolutional neural network (CNN) that can include multiple residual (ResNet) modules and attention mechanisms. In addition, a z-conditioning is added for style transfer. This z represents noise and can be transformed through a fully connected network (called a mapping network). The resulting embedding vector (represented by w) is then fed into each adaptive group normalization layer. Each adaptive group normalization layer contains a fully connected layer that takes w as input and outputs the offset and scaling parameters of the normalized group for each channel. The discriminator can be constructed using a convolutional network with a residual (ResNet) block.

[0144] The architecture of the generator consists of the following 25 layers connected in sequence:

[0145] 1) "3*3*3conv3D(stride1, 64", which means: a 64-dimensional 3D convolutional layer with a size of 3*3*3 and a stride of 1;

[0146] 2) "ResNetBlock,64", which means: 64-dimensional residual layer;

[0147] 3) "ResNetBlock,64", which means: 64-dimensional residual layer;

[0148] 4) "Convdown(stride2), 64", which means: a 64-dimensional convolutional downsampling layer with a stride of 2;

[0149] 5) "ResNetBlock,128", which means: 128-dimensional residual layer;

[0150] 6) "ResNetBlock,128", which means: 128-dimensional residual layer;

[0151] 7) "Convdown(stride2), 128", which means: a 128-dimensional convolutional downsampling layer with a stride of 2;

[0152] 8) "ResNetBlock,256", which means: 256-dimensional residual layer;

[0153] 9) "ResNetBlock,256", which means: 256-dimensional residual layer;

[0154] 10) "Convdown(stride2), 256", which means: a 256-dimensional convolutional downsampling layer with a stride of 2;

[0155] 11) "ResNetBlock,512", which means: 512-dimensional residual layer;

[0156] 12) "ResNetBlock,512", which means: 512-dimensional residual layer;

[0157] 13) "Convdown(stride2), 512", which means: a 512-dimensional convolutional downsampling layer with a stride of 2;

[0158] 14) "ResNetBlock,512", which means: 512-dimensional residual layer;

[0159] 15) "ResNetBlock,512", which means: 512-dimensional residual layer;

[0160] 16) "Convtranspose3D,upsampling(stride2),256" indicates a neural network layer that performs a transposed convolution on 3D data and an upsampling layer with a 256-dimensional stride of 2.

[0161] 17) "ResNetBlock,256", which means: 256-dimensional residual layer;

[0162] 18) "ResNetBlock,256", which means: 256-dimensional residual layer;

[0163] 19) "Convtranspose3D,upsampling(stride2),128" means: a neural network layer that performs a transposed convolution operation on the 3D data and a 128-dimensional upsampling layer with a stride of 2;

[0164] 20) "ResNetBlock,128", which means: 128-dimensional residual layer;

[0165] 21) "ResNetBlock,128", which means: 128-dimensional residual layer;

[0166] 22) "Convtranspose3D,upsampling(stride2),64" indicates a neural network layer that performs a transposed convolution on 3D data and a 64-dimensional upsampling layer with a stride of 2.

[0167] 23) "ResNetBlock,64", which means: 64-dimensional residual layer;

[0168] 24) "ResNetBlock,64", which means: 64-dimensional residual layer;

[0169] 25) "Convtranspose3D,upsampling(stride2),1" indicates a neural network layer that performs a transposed convolution operation on 3D data and a 1-dimensional upsampling layer with a stride of 2.

[0170] The architecture of the discriminator consists of the following eight layers connected in sequence:

[0171] 1) "1*1*1conv3D,64" means a 64-dimensional 3D convolutional layer of size 1*1*1;

[0172] 2) "ResNetBlockdown,64", which means: 64-dimensional residual downsampling layer;

[0173] 3) "ResNetBlock down, 128", which means: 128-dimensional residual downsampling layer;

[0174] 4) "ResNetBlock down, 256", which means: 256-dimensional residual downsampling layer;

[0175] 5) “ResNetBlock down, 512”, which means: 512-dimensional residual downsampling layer;

[0176] 6) "minibatch std layer", which represents the mini-batch standard deviation layer;

[0177] 7) "Global Sum Pooling", which represents the global sum pooling layer;

[0178] 8) “FC layer” means fully connected layer.

[0179] S5: Training a Diffusion Generative Adversarial Network

[0180] The training process includes the forward diffusion process and the reverse generation process, see Figure 8 .

[0181] Specifically, first, the forward noise addition process based on the diffusion model gradually adds noise to the real longitudinal magnetic resonance prediction image until the final noise image conforms to the Gaussian distribution. The gradual noise addition process lasts for a total of T steps. In the reverse generation process, the noise image A longitudinal magnetic resonance synthesis image is generated by a generator, and then the generated longitudinal magnetic resonance synthesis image is and the actual longitudinal MRI predicted images Calculate the mean absolute error loss , where the absolute mean error loss The loss function is shown in formula (1) above.

[0182] Generative longitudinal magnetic resonance composite image Use posterior sampling to obtain synthetic noise images , and the noise image in each step and synthetic noise images The noise image is sent to the discriminator for confrontation to ensure that the distribution of the synthetic noise image obtained by each step of posterior sampling is consistent with the real noise image. After T steps of the reverse generation process, the final synthetic longitudinal magnetic resonance image is obtained. In the generator, it is proposed to add a learnable conditional encoder, add the subject's baseline image, age, gender and diagnosis label to encode the constraint model to generate images of the same person, so as to maintain the consistency of the subject. Together with the learnable conditional encoder, it ensures that the synthesized images are images of the same subject.

[0183] Longitudinal MRI composite images based on each step of synthesis and the actual longitudinal MRI predicted images , set the constraint that the difference between the synthetic image and the baseline image becomes larger as the time interval increases. By increasing the time interval, it is ensured that the larger the age interval is, the greater the change is compared with the baseline image. Conversely, the smaller the interval is, the smaller the change is. The loss of the correlation between the time interval and the image difference , as shown in formula (3).

[0184] The loss function for training the conditional generator (the aforementioned generator) is shown in formula (2). The overall loss function for the generator is shown in formula (4). The loss function for the discriminator is shown in formula (5).

[0185] That is, step S5 firstly performs a forward denoising process based on the diffusion model to gradually predict the real longitudinal magnetic resonance image. Add noise until the final noise image conforms to the Gaussian distribution. The process of gradual noise addition lasts for T steps. In the reverse generation process, the noise image An initial baseline image is generated by a condition generator, and then the generated longitudinal MRI composite image is and the actual longitudinal MRI predicted images The absolute mean error was calculated and the longitudinal MRI composite image was generated based on the Obtained using posterior sampling , and based on the original and The noise image is sent to the discriminator for confrontation to ensure that the distribution of the synthetic noise image obtained by each step of posterior sampling is consistent with the real noise image. After T steps of reverse generation process, the final synthetic longitudinal magnetic resonance image is obtained. .

[0186] In addition, in the application example of this application, a Unet structure can also be used to generate original brain images. The Unet structure consists of an encoder and a decoder. The application example of this application uses structural magnetic resonance data. If gray matter volume images or T2 images are used, they can also be considered as alternatives.

[0187] On this basis, the application examples of this application also provide experimental results data between the longitudinal magnetic resonance image generation model (IP-DDGAN) and the conditional generative adversarial network (CGAN), self-attention generative adversarial network (SAGAN), cyclic generative adversarial network (CycleGAN), diffusion model (DDPM) and hidden layer diffusion model (LDM). Figure 9 The results of the comparison of the longitudinal data after 3 years (GT (3 years)) are shown; see Figure 10 The results of the comparison of the longitudinal data after 6 years (GT (6 years)) are shown in the figure; "CN-CN" refers to normal people to normal people; "MCI-MCI" refers to mild cognitive impairment to mild cognitive impairment; "AD-AD" refers to Alzheimer's disease to Alzheimer's disease; "CN-MCI" refers to normal people to mild cognitive impairment patients; "MCI–AD" refers to mild cognitive impairment patients to Alzheimer's disease patients. Figure 9 and Figure 10 In the image, red represents an error greater than 0, and blue represents an error less than 0. However, since absolute values are used, no blue appears in the image. The specific comparison values are shown in Tables 1 to 4:

[0188] Table 1 - Comparison of longitudinal data of each model for "CN-CN" after 3 and 6 years of synthesis

[0189]

[0190] Table 2 - Comparison of the results of each model for the longitudinal data of "MCI to MCI" after 3 and 6 years

[0191]

[0192] Table 3 - Comparison of the results of each model for the longitudinal data of AD to AD after 3 and 6 years

[0193]

[0194] Table 4 - Comparison of the various models for longitudinal data from CN to MCI and MCI to AD

[0195]

[0196] Among them, PSNR refers to the peak signal-to-noise ratio, and the larger the value, the better; SSIM refers to structural similarity, and the closer to 1, the better; MMD refers to the maximum mean difference, and the closer to 0, the better.

[0197] That is to say, the longitudinal magnetic resonance image generation model training method provided in the application example of this application synthesizes longitudinal structural magnetic resonance images of the same subject based on the diffusion generative adversarial network model, and ensures that the synthesized longitudinal images come from the same subject. The data structure currently used is MRI images, but it is not limited to T1 images. Gray matter density images and the like can be input. Among them, the ‌T1 image refers to the T1-weighted image (T1WI) in magnetic resonance imaging (MRI). The absolute mean error loss function is used to constrain the error at each step in the diffusion process, thereby accelerating the rapid convergence of the model. In the diffusion generative adversarial network, it is assumed to obey a multimodal distribution, so the number of diffusion steps is small, which speeds up sampling. It should be effective at different diffusion steps.

[0198] Based on this, the longitudinal magnetic resonance image generation model training method provided in the application example of this application has the following beneficial effects:

[0199] 1) Regarding Problem 1: Generative adversarial networks are prone to mode collapse and the generated images have low diversity;

[0200] The path adopted by the application example of this application to solve the above problem 1 is: GAN directly synthesizes images at one time, and the generator only learns to generate a few image patterns, ignoring the diversity of the entire data distribution. However, the diffusion model gradually "denoises and restores" the data distribution. Each step is modeling the details of the real distribution, making it more difficult to fall into the local optimum of only generating a few patterns.

[0201] 2) Regarding issue 2: Slow sampling speed in the diffusion model;

[0202] The path adopted by the application example of this application to solve the above-mentioned problem 2 is: in the forward propagation process of the diffusion model, the noise added in each step obeys the Gaussian normal distribution, which results in each diffusion step being very small and the number of steps being large, so it takes a long time to infer to generate data. However, in real applications, more data obey a multimodal distribution. The diffusion process of each step of the model proposed in this study does not have to obey the Gaussian normal distribution. A conditional generator is used to fit the diffusion process of each step. Since each step is a non-Gaussian normal distribution, the number of diffusion steps can be reduced, thereby achieving fast data generation.

[0203] 3) Regarding question 3: The current diffusion generative adversarial network only considers the synthesis of natural images, and does not consider the validity of the synthesized images and the identity of the subjects. In the process of synthesizing medical images, it is necessary not only to ensure the authenticity of the synthesized images, but also to ensure that the synthesized images come from the same subject. In addition, the current diffusion generative adversarial network cannot guarantee that the volume of each brain region in the synthesized medical image is very close. The images may look similar, but the residual is large compared with the actual longitudinal direction, thus failing to guarantee the authenticity of the image. Discontinuity between adjacent slices is also prone to occur in the synthesized 3D brain images;

[0204] The solution to the above problem 3 is as follows: the initial noise of the diffusion model is random. If the noise evolution path at different locations is unstable or uncoordinated, it will also lead to discontinuity between slices. In this application, the generator is first used to directly generate the synthetic image, and the absolute mean error loss is added in each diffusion step. , ensuring the reliability of each step of the process, and using the discriminator to ensure that the results of the generator are more realistic.

[0205] That is to say, the application example of this application uses a diffusion generative adversarial network to synthesize the longitudinal images of each subject, and ensures that the synthesized images are from the same subject; based on the constraint of absolute average error in each step of the reverse generation process, the similarity of the microstructure of the medical image is achieved, such as the consistency of gray matter volume, white matter volume, etc., and the continuity of adjacent slice images can be guaranteed; using L IDThe loss function ensures that as the time interval increases, the synthesized brain image changes more compared to the baseline image when the age interval is large, and vice versa, the synthesized brain image changes less when the interval is small.

[0206] Based on the above embodiment of the longitudinal magnetic resonance image generation model training method, the present application also provides a longitudinal magnetic resonance image generation method, see Figure 11 The longitudinal magnetic resonance image generation method specifically includes the following contents:

[0207] Step 300: Input a noise image conforming to a Gaussian distribution, a baseline magnetic resonance image of the current target subject, and auxiliary data of the target subject into a longitudinal magnetic resonance image generation model, so that a learning conditional encoder and generator in the longitudinal magnetic resonance image generation model gradually acquire longitudinal magnetic resonance composite images corresponding to each step, and after each acquisition of the longitudinal magnetic resonance composite image corresponding to a reverse step, perform a posteriori sampling on the longitudinal magnetic resonance composite image corresponding to the reverse step to obtain a composite noise image corresponding to the reverse step; wherein the longitudinal magnetic resonance image generation model is pre-trained based on the longitudinal magnetic resonance image generation model training method; and the auxiliary data includes: the age, gender, and diagnostic label of the target subject;

[0208] Step 400: Using the longitudinal magnetic resonance synthesis image output by the generator in the last backward step as the longitudinal magnetic resonance image prediction result data of the target subject at the target time point.

[0209] It can be understood that the longitudinal magnetic resonance image generation model training method mentioned in the embodiment of the longitudinal magnetic resonance image generation method provided in the present application can specifically adopt the processing flow of the longitudinal magnetic resonance image generation model training method in the above embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned longitudinal magnetic resonance image generation model training method embodiment.

[0210] From the software level, the present application also provides a longitudinal magnetic resonance image generation model training device for executing all or part of the longitudinal magnetic resonance image generation model training method, see Figure 12 The longitudinal magnetic resonance image generation model training device specifically includes the following contents:

[0211] The iterative training module 10 is configured to perform at least one round of iterative training on the diffusion generative adversarial network, and execute a model training step in each round, wherein the model training step includes: gradually adding noise to the longitudinal magnetic resonance prediction image of the subject in each forward step of the forward diffusion process of the diffusion generative adversarial network to obtain a noise image corresponding to each forward step, and stopping the noise addition after obtaining a noise image that conforms to a Gaussian distribution; based on the noise image that conforms to the Gaussian distribution and the magnetic resonance baseline image of the subject, gradually obtaining the longitudinal magnetic resonance synthetic image corresponding to each step in the backward diffusion process of the diffusion generative adversarial network based on a learnable conditional encoder and a generator, and after each time obtaining the longitudinal magnetic resonance synthetic image corresponding to a backward step, performing posterior sampling on the longitudinal magnetic resonance synthetic image corresponding to the backward step to obtain a synthetic noise image corresponding to the backward step; and gradually inputting the synthetic noise image of each backward step and the noise image of the forward step corresponding to the backward step into a discriminator, so that the discriminator outputs discrimination result data corresponding to each backward step to optimize the discriminator and the generator.

[0212] The model determination module 20 is configured to use the learnable conditional encoder and the generator in the iteratively trained diffusion generative adversarial network as a longitudinal magnetic resonance image generation model.

[0213] The embodiment of the longitudinal magnetic resonance image generation model training device provided in this application can be specifically used to execute the processing flow of the embodiment of the longitudinal magnetic resonance image generation model training method in the above-mentioned embodiment. Its functions will not be described in detail here, and reference can be made to the detailed description of the above-mentioned longitudinal magnetic resonance image generation model training method embodiment.

[0214] The portion of the longitudinal magnetic resonance image generation model training apparatus that performs longitudinal magnetic resonance image generation model training can be completed in a server or client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not impose any restrictions on this. If all operations are completed in the client device, the client device may also include a processor for the specific processing of the longitudinal magnetic resonance image generation model training.

[0215] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0216] The server and the client device may communicate using any suitable network protocol, including network protocols that have not yet been developed as of the filing date of this application. Examples of such network protocols include TCP / IP, UDP / IP, HTTP, and HTTPS. Furthermore, examples of such network protocols include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols, which are used on top of the aforementioned protocols.

[0217] From the above description, it can be seen that the longitudinal magnetic resonance image generation model training device provided in the embodiment of the present application can improve the training efficiency and effectiveness of the longitudinal magnetic resonance image generation model, can ensure the diversity of generated synthetic images, and can ensure that the generated synthetic images and magnetic resonance baseline images belong to the same subject.

[0218] From the software level, the present application also provides a longitudinal magnetic resonance image generation device for executing all or part of the longitudinal magnetic resonance image generation method, see Figure 13 The longitudinal magnetic resonance image generating device specifically includes the following contents:

[0219] The model prediction module 30 is configured to input a noise image conforming to a Gaussian distribution, a baseline magnetic resonance image of a current target subject, and auxiliary data of the target subject into a longitudinal magnetic resonance image generation model, so that a learning conditional encoder and generator based on the longitudinal magnetic resonance image generation model gradually obtain longitudinal magnetic resonance composite images corresponding to each step, and after each acquisition of the longitudinal magnetic resonance composite image corresponding to a reverse step, a posteriori sampling is performed on the longitudinal magnetic resonance composite image corresponding to the reverse step to obtain a composite noise image corresponding to the reverse step; wherein the longitudinal magnetic resonance image generation model is pre-trained based on the longitudinal magnetic resonance image generation model training method; and the auxiliary data includes: the age, gender, and diagnostic label of the target subject;

[0220] The result output module 40 is configured to use the longitudinal magnetic resonance synthesis image output by the generator in the last backward step as the longitudinal magnetic resonance image prediction result data of the target subject at the target time point.

[0221] The embodiment of the longitudinal magnetic resonance image generation device provided in the present application can be specifically used to execute the processing flow of the embodiment of the longitudinal magnetic resonance image generation method in the above embodiment. Its functions are not described in detail here, and reference can be made to the detailed description of the embodiment of the above longitudinal magnetic resonance image generation method.

[0222] The part of the longitudinal magnetic resonance image generation device that generates the longitudinal magnetic resonance image can be completed in a server or a client device.

[0223] The present application also provides an electronic device that may include a processor, a memory, a receiver, and a transmitter. The processor is configured to execute the longitudinal magnetic resonance image generation model training method and / or the longitudinal magnetic resonance image generation method described in the above embodiments. The processor and the memory may be connected via a bus or other means, with bus connection being used as an example. The receiver may be connected to the processor and the memory via a wired or wireless means.

[0224] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0225] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the longitudinal magnetic resonance image generation model training method and / or the longitudinal magnetic resonance image generation method in the embodiments of the present application. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the longitudinal magnetic resonance image generation model training method and / or the longitudinal magnetic resonance image generation method in the above-mentioned method embodiments.

[0226] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0227] The one or more modules are stored in the memory, and when executed by the processor, perform the longitudinal magnetic resonance image generation model training method and / or the longitudinal magnetic resonance image generation method in the embodiment.

[0228] In some embodiments of the present application, the user equipment may include a processor, a memory and a transceiver unit, and the transceiver unit may include a receiver and a transmitter. The processor, memory, receiver and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0229] As an implementation method, the functions of the receiver and transmitter in this application can be considered to be implemented through a transceiver circuit or a dedicated transceiver chip, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general-purpose chip.

[0230] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiments of the present application. That is, the program code for implementing the functions of the processor, receiver, and transmitter is stored in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0231] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of the aforementioned longitudinal magnetic resonance image generation model training method and / or longitudinal magnetic resonance image generation method. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0232] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the aforementioned longitudinal magnetic resonance image generation model training method and / or longitudinal magnetic resonance image generation method.

[0233] It should be understood by those skilled in the art that the various exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether it is implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted on a transmission medium or communication link via a data signal carried in a carrier.

[0234] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0235] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0236] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Those skilled in the art will appreciate that various modifications and variations of the present embodiment are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A longitudinal magnetic resonance image generation model training method, characterized in that: include: The diffusion generative adversarial network is iteratively trained for at least one round, and a model training step is performed in each round, wherein the model training step includes: gradually adding noise to the longitudinal magnetic resonance prediction image of the subject in each forward step of the forward diffusion process of the diffusion generative adversarial network to obtain a noise image corresponding to each forward step, and stopping the noise addition after obtaining a noise image that conforms to a Gaussian distribution; based on the noise image that conforms to the Gaussian distribution and the magnetic resonance baseline image of the subject, gradually obtaining a longitudinal magnetic resonance synthetic image corresponding to each backward step based on a learnable conditional encoder and a generator in a backward generation process of the diffusion generative adversarial network, and after each longitudinal magnetic resonance synthetic image corresponding to a backward step is obtained, posterior sampling is performed on the longitudinal magnetic resonance synthetic image corresponding to the backward step to obtain a synthetic noise image corresponding to the backward step; the synthetic noise image of each backward step and the noise image of the forward step corresponding to the backward step are gradually input into a discriminator, so that the discriminator outputs discrimination result data corresponding to each backward step to optimize the discriminator and the generator; The learnable conditional encoder and the generator in the iteratively trained diffusion generative adversarial network are used as a longitudinal magnetic resonance image generation model.

2. The longitudinal magnetic resonance image generation model training method according to claim 1, characterized in that: The method includes: stepwise acquiring, in a reverse generation process of the diffusion generative adversarial network, longitudinal magnetic resonance synthetic images corresponding to respective reverse steps based on the noise image conforming to the Gaussian distribution and the magnetic resonance baseline image of the subject based on a learnable conditional encoder and a generator; and performing posterior sampling on the longitudinal magnetic resonance synthetic image corresponding to the reverse step after each acquisition of the longitudinal magnetic resonance synthetic image corresponding to the reverse step to obtain a synthetic noise image corresponding to the reverse step, including: In each reverse step of the reverse generation process of the diffusion generative adversarial network, performing an image synthesis training sub-step respectively; The image synthesis training sub-step includes: In the current backward step, the noise image conforming to the Gaussian distribution, the MRI baseline image of the subject, and auxiliary data of the subject are input into the learnable conditional encoder, so that the learnable conditional encoder outputs corresponding encoded data; wherein the auxiliary data includes: the age, gender, and diagnosis label of the subject; Inputting the encoded data into the generator so that the generator outputs a longitudinal magnetic resonance composite image corresponding to the current reverse step; Performing posterior sampling on the longitudinal magnetic resonance synthetic image corresponding to the current reverse step to obtain a synthetic noise image corresponding to the reverse step; Calculating the generator loss corresponding to the current backward step according to the longitudinal magnetic resonance synthetic image corresponding to the current backward step and the longitudinal magnetic resonance predicted image of the subject; Update the model parameters of the generator based on the generator loss corresponding to the current backward step.

3. The longitudinal magnetic resonance image generation model training method according to claim 2, characterized in that: The generator loss includes: absolute mean error loss and the training loss of the generator; Correspondingly, calculating the generator loss corresponding to the current backward step based on the longitudinal magnetic resonance synthetic image corresponding to the current backward step and the longitudinal magnetic resonance predicted image of the subject includes: Obtaining an absolute average error between the longitudinal magnetic resonance synthesized image of the current reverse step and the longitudinal magnetic resonance predicted image of the current round as the absolute average error loss corresponding to the current reverse step; And, according to the longitudinal magnetic resonance prediction image of the current round and the noise image of the current reverse step, the training loss of the generator of the current reverse step is obtained.

4. The longitudinal magnetic resonance image generation model training method according to claim 3, characterized in that: The generator loss also includes: a correlation loss between time intervals and image differences; Correspondingly, calculating the generator loss corresponding to the current backward step based on the longitudinal magnetic resonance synthetic image corresponding to the current backward step and the longitudinal magnetic resonance predicted image of the subject further includes: The loss of the correlation relationship between the time interval corresponding to the current reverse step and the image difference is obtained based on the longitudinal magnetic resonance synthetic image of the current reverse step, the longitudinal magnetic resonance predicted image of the current round, the age of the subject corresponding to the magnetic resonance baseline image when the magnetic resonance baseline image is acquired, the age of the subject corresponding to the longitudinal magnetic resonance predicted image of the current round, and the preset maximum age and minimum age values.

5. The longitudinal magnetic resonance image generation model training method according to claim 4, characterized in that: Updating the model parameters of the generator based on the generator loss corresponding to the current backward step includes: The generator loss corresponding to the current backward step is determined according to the training loss of the generator corresponding to the current backward step, the product of the absolute mean error loss and a preset first hyperparameter, and the product of the correlation relationship loss between the time interval and the image difference and a preset second hyperparameter, and the model parameters of the generator are updated based on the generator loss corresponding to the current backward step.

6. The longitudinal magnetic resonance image generation model training method according to any one of claims 1 to 5, characterized in that: Before performing at least one round of iterative training on the diffusion generative adversarial network, the method further includes: Acquiring an original magnetic resonance image of each subject acquired at a historical time point and an original longitudinal magnetic resonance prediction image of each subject acquired at a target time point after the historical time point; performing image preprocessing, data cleaning, and normalization on the original magnetic resonance images and the original longitudinal magnetic resonance predicted images of each subject; The original magnetic resonance images corresponding to each subject after the image preprocessing, data cleaning and normalization processing are respectively used as the magnetic resonance reference images corresponding to each subject, and the original longitudinal magnetic resonance prediction images corresponding to each subject after the image preprocessing, data cleaning and normalization processing are respectively used as the longitudinal magnetic resonance prediction images corresponding to each subject.

7. A method for generating a longitudinal magnetic resonance image, characterized in that: include: A noise image conforming to a Gaussian distribution, a magnetic resonance baseline image of the current target subject, and auxiliary data of the target subject are input into a longitudinal magnetic resonance image generation model, so that a learning conditional encoder and a generator based on the longitudinal magnetic resonance image generation model gradually obtain longitudinal magnetic resonance synthetic images corresponding to each step, and after each acquisition of the longitudinal magnetic resonance synthetic image corresponding to a reverse step, a posteriori sampling is performed on the longitudinal magnetic resonance synthetic image corresponding to the reverse step to obtain a synthetic noise image corresponding to the reverse step; wherein the longitudinal magnetic resonance image generation model is pre-trained based on the longitudinal magnetic resonance image generation model training method according to any one of claims 1 to 6; the auxiliary data includes: the age, gender, and diagnostic label of the target subject; The longitudinal magnetic resonance composite image output by the generator in the last backward step is used as the longitudinal magnetic resonance image prediction result data of the target subject at the target time point.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the longitudinal magnetic resonance image generation model training method according to any one of claims 1 to 6, and / or implements the longitudinal magnetic resonance image generation method according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for training a longitudinal magnetic resonance image generation model according to any one of claims 1 to 6 is implemented, and / or the method for generating a longitudinal magnetic resonance image according to claim 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for training a longitudinal magnetic resonance image generation model according to any one of claims 1 to 6 is implemented, and / or the method for generating a longitudinal magnetic resonance image according to claim 7 is implemented.

Images