Multi-modal medical image conversion method based on conditional potential diffusion model

Abstract

The invention discloses a multi-modal medical image conversion method based on a conditional potential diffusion model. The method comprises the following steps: acquiring a source domain modal image; compressing the source domain modal image from an original full-size image to a potential space by using a pre-trained first encoder to obtain corresponding low-dimensional data; and inputting the low-dimensional data into the trained conditional potential diffusion model, and further converting the low-dimensional data into a target domain modal image. According to the conditional potential diffusion model, low-dimensional data is used as input, shape information features extracted from a source domain modal image and visual information features extracted from a target domain modal image are used as comprehensive guide conditions, noise is gradually added through multiple diffusion steps, a noise image is obtained, then the noise image is subjected to a denoising process, and a final image is obtained. And finally, reconstructing the target domain modal image by using a decoder. According to the method, cross-modal MRI synthesis is realized, the calculation cost is reduced, and the shape consistency of the synthesized image and the source domain image is ensured by introducing shape priori knowledge.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly to a multi-modal medical image conversion method based on a conditional latent diffusion model. BACKGROUND

[0002] Multi-modal magnetic resonance imaging (MRI) images respectively show information of different contrasts of the same organ, while single-modal medical images usually cannot reflect the complex characteristics of diseases. By comprehensively analyzing multiple modal medical images, combining different data features and complementary relationships, a variety of supplementary information including structure and function can be provided for disease diagnosis, and the accuracy of disease diagnosis can be significantly improved. For example, in clinical diagnosis, T1 weighted (T1w), T2 weighted MRI (T2w), T1 weighted contrast-enhanced MRI (T1ce) and fluid attenuated inversion recovery (Flair) are often acquired for brain tumors, and balanced steady-state free precession (bSSFP) cine imaging, late-gadolinium enhancement (LGE) imaging and T2 weighting are relied on for evaluation of the heart. The use of MRI contrast agents can significantly enhance the contrast of images and improve the accuracy and reliability of diagnosis. However, the acquisition of multiple modal images for each patient requires expensive examination costs and time costs. In the face of this challenge, multi-modal image conversion can provide more rich diagnostic information without increasing the burden on patients. The challenges faced by this technology mainly include: the synthesized images have obvious differences in shape and texture and other attributes from the source domain images; the effective information between different modalities is not fully utilized.

[0003] Currently, multi-modality MRI can reflect different characteristics of different regions by using multiple different imaging modalities, providing more abundant tissue information, and is widely used in the examination and diagnosis of heart and brain. For example, in the examination of heart MRI, commonly used imaging modalities include LGE and bSSFP. LGE technology can enhance the display of myocardial infarction (MI) through its significant local brightness contrast, and its good spatial resolution and tissue specificity have unique advantages in the diagnosis of various myocardial diseases. Accurate segmentation of cardiac structures in LGE has important clinical significance for cardiac function assessment and myocardial disease diagnosis. However, a major challenge of LGE imaging is that the contrast agent needs to be injected for 10-20 minutes to take effect, and its special imaging method causes uneven myocardial intensity in the image and blurred boundaries between myocardium and blood pool, affecting the accuracy of subsequent image segmentation and analysis. Unlike LGE images, bSSFP imaging can highlight high signal areas of fluids while presenting uniform signals for other tissues, making it particularly suitable for capturing heart motion and clearly displaying heart boundaries. Converting easily accessible and well-labeled bSSFP images to synthetic LGE images not only has practical significance for making up for the lack of LGE image samples, but also provides an effective way for cross-modality data augmentation, thereby supporting more accurate cardiac disease diagnosis and analysis.

[0004] MRI is also one of the gold standards for brain tumor diagnosis. Glioblastoma (GBM) is one of the most common and aggressive primary brain tumors. Early detection of brain abnormalities is crucial for timely diagnosis and treatment, as tumors can be potentially malignant. Four commonly used standard MRI modalities include T1w, T2w, T1ce, and Flair, which provide complementary information about tumors. T1 imaging sequences are suitable for observing the anatomical structure of the brain, T2 imaging sequences are more sensitive to detecting abnormal water content in brain tissue, and Flair sequences are specifically used to detect high signal areas in white matter. T1ce can clearly distinguish the parenchymal part of brain tumors and has important guiding significance for the diagnosis of tumor-related diseases. In clinical practice, common MRI modalities such as T1, T2, and Flair are relatively easy to obtain, as they can be obtained during the same scan by adjusting specific parameters of the MRI scan. In contrast, T1ce is relatively difficult to obtain because additional interventions, such as contrast agent injection, are required for patients. This contrast agent is usually radioactive and requires careful consideration of its dosage and safety, and some patients may have allergic reactions to it. Therefore, developing a method to synthesize T1ce images from other conventional MRI modalities has important clinical value for improving diagnostic efficiency and patient safety.

[0005] For patients, synthetic multi-modal MRI data not only reduces the need for repeated imaging, improves the efficiency of medical equipment, but also reduces the discomfort and potential risks of patients, and at the same time, reduces the economic burden. In addition, training the network using the synthesized multi-modal data to cope with downstream tasks has broad application prospects.

[0006] At present, in the field of image conversion, deep learning technology has made significant progress. For natural images, Pix2pix performs well on paired image translation, and CycleGAN performs well on non-paired image translation. However, GAN models may face problems such as unstable training and lack of interpretability. Unlike natural images, medical images contain rich anatomical information, so it is crucial to preserve the anatomical structure in multi-modal MRI conversion tasks. Current image synthesis methods often ignore the consistency of contour information, which affects the quality of the final generated conversion images. Recently, diffusion models have become a kind of generative model that can generate high-quality images. However, in the application of multi-modal medical image conversion, existing diffusion models face the challenges of high computational complexity and insufficient preservation of structural information. In particular, during the forward diffusion process, the structural details of the source domain image are easily lost and cannot be fully recovered through learned reverse diffusion, while ensuring the integrity of the anatomical structure is extremely important in medical images.

[0007] Traditional multi-modal image conversion methods, such as sparse coding, patch matching, and atlas analysis, usually require precise alignment between source and target domain images, and have limitations in scalability. In contrast, deep learning-based image synthesis technology is rapidly advancing, especially methods based on generative adversarial networks (GAN). Through adversarial training between the generator and the discriminator, GAN-based methods can effectively learn the distribution characteristics of the data, thereby achieving realistic image generation. For example, using an improved CycleGAN network, the network parameters are optimized through convergence at the beginning, and then the parameter update is continued at a lower learning rate, while taking advantage of the network's dynamic synthesis ability. On a cardiac MRI dataset, multiple target domain images can be generated from a single source domain image for data augmentation. MM-GAN randomly discards several input modalities and uses a selective discriminator for loss calculation, synthesizing high-quality images in the absence of modalities, and can be extended to any combination of available sequences and missing sequences on the BraTS2018 dataset. However, existing deep learning methods are prone to problems such as unstable training or mode collapse, and lack of interpretability. In addition, when modeling complex multi-modal data distribution, they are challenged by the limited ability of adversarial learning.

[0008] Recently, diffusion models have shown excellent capabilities in image translation. For example, patent application CN202211091270.9 discloses a cross-modal medical image completion method based on conditional score, which selects one or more modal image data as the original target modal image data, and the remaining part as the conditional modal image data, and passes the complete diffusion image data through a multi-modal conditional inverse stochastic differential equation to obtain the completed target modal image data. Although the above conditional diffusion guided method has made some progress in cross-modal medical image generation, it still has some limitations, including computational complexity, shape constraint, dependence on training data, and generalization ability. Therefore, it is of significance to construct new conditions to guide the diffusion generation model. SUMMARY

[0009] The purpose of the present application is to overcome the defects of the prior art described above and provide a multi-modal medical image conversion method based on a conditional latent diffusion model. The method comprises the following steps:

[0010] Obtaining a source domain modal image;

[0011] Compressing the original full-size image to a latent space using a pre-trained first encoder to obtain corresponding low-dimensional data;

[0012] Inputting the low-dimensional data into a trained conditional latent diffusion model to convert it into a target domain modal image;

[0013] Wherein, the conditional latent diffusion model takes the low-dimensional data as input, uses shape information features extracted from the source domain modal image and visual information features extracted from the target domain modal image as comprehensive guide conditions, gradually adds noise through multiple diffusion steps to obtain a noise image, and then removes noise from the noise image through a denoising process. Finally, the target domain modal image is reconstructed using a decoder.

[0014] Compared with the prior art, the present application has the advantages that a multi-modal medical image conversion method based on a conditional latent diffusion model is proposed, which realizes multi-modal MRI image conversion for different organs. Not only does it maintain shape consistency by introducing shape prior information, but also enhances the stability and interpretability of the model, effectively realizing the conversion from the known domain of available MRI modalities to the target domain of missing modalities. The conditional latent diffusion model proposed in the present application not only reduces the computational cost and realizes cross-modal MRI synthesis, but also ensures the shape consistency of the synthesized image with the source domain image by introducing shape prior knowledge, improving the accuracy of the synthesized image.

[0015] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments thereof, which is to be taken in conjunction with the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0017] Figure 1 is a flow chart of a multi-modal medical image conversion method based on a conditional latent diffusion model according to an embodiment of the present application;

[0018] Figure 2 is a general process schematic diagram of a multi-modal medical image conversion method based on a conditional latent diffusion model according to an embodiment of the present application;

[0019] Figure 3 is a model general training process schematic diagram according to an embodiment of the present application;

[0020] Figure 4 is a forward diffusion process schematic diagram of a conditional diffusion model according to an embodiment of the present application;

[0021] Figure 5 is a training process schematic diagram of a conditional diffusion model according to an embodiment of the present application;

[0022] Figure 6 is a description diagram of a data set and implemented task according to an embodiment of the present application;

[0023] Figure 7 is a result schematic diagram of a cardiac data set generating pseudo LGE from real bSSFP according to an embodiment of the present application;

[0024] Figure 8 is a result schematic diagram of a brain data set generating pseudo T1ce from real T1+T2+Flair according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of the components and steps set forth in the embodiments, numerical expressions, and numerical values are not limiting to the scope of the present application unless otherwise specifically stated.

[0026] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the application or its applications or uses.

[0027] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered part of the specification as appropriate.

[0028] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0029] It should be noted that like numerals and letters refer to like items throughout the several views, as such, once an item is defined in one view, it need not be discussed further in subsequent views.

[0030] To solve the problem of the lack of current medical image modalities, the application provides a multi-modal medical image conversion method based on a conditional latent diffusion model. In general, the method uses one or more source modalities as samples, compresses them to the latent space through an encoder, and then generates the corresponding target modality image by combining the shape prior knowledge to guide the conditional diffusion model. The application introduces a shape preservation term with the help of the prior formed by the traditional method. It ensures that the generated image is consistent in shape with the source image.

[0031] Specifically, as shown in Figure 1 and Figure 2 The provided multi-modal medical image conversion method based on a conditional latent diffusion model includes the following steps:

[0032] Step S110, obtaining a training data set reflecting the corresponding relationship between the source domain modality image and the target domain modality image.

[0033] The training data set can be a publicly known data set or self-constructed. The training data set reflects the corresponding relationship between the source domain modality image (or simply referred to as the source domain image) and the target domain modality image (or simply referred to as the target domain image).

[0034] For example, the training data set is constructed based on the MS-CMR2019 data set or the BraTs2020 data set. The MS-CMR2019 data set reflects the corresponding relationship between the LGE modality image, the T2 modality image and the bSSFP modality image. The BraTs2020 data set reflects the corresponding relationship between the T1w modality image, the T2w modality image, the Flair modality image and the T1ce modality image. One or more modalities can be selected as the source domain, and the remaining one modality can be selected as the target domain. For example, the training data set reflects the corresponding relationship between the source domain modality image or the target domain modality image.

[0035] Step S120, constructing a conditional latent diffusion model and training based on the training data set until the set loss standard is met.

[0036] In combination with Figure 3As shown, the constructed potential conditional diffusion model (or simply conditional diffusion model) generally includes an encoder module, a conditional potential diffusion model training module, and a decoder module. Based on the constructed training data set, the three modules including the encoder module, the decoder module, and the multi-condition potential diffusion model training module can be trained. Generally, combined with Figure 2 As shown, first, the original image is compressed to the latent space by using the encoder, then the conditional guided training is formed by introducing the shape prior and the target domain image, and finally the cross-modal image is generated by recovering to the original image space by using the decoder.

[0037] Specifically, combined with Figure 4 As shown, let A and B be two image domains representing the source modality and the target modality, respectively. In the forward diffusion process, the pre-trained encoder E is used to compress the original full-size image x A,0 to obtain low-dimensional data z A,0 (referred to as z0 hereinafter), and the data space where the compressed z0 is located is called the latent space. The forward diffusion process is represented by q, and q(z A,t |z A,0 ) represents that z0 is input into the conditional diffusion model, and a series of intermediate representations: {z0,…, z t} are gradually added by noise through T diffusion steps until it is finally converted into a random noise image z t . The noise addition formula is represented as:

[0038]

[0039] wherein the noise added each time is subject to a Gaussian distribution, i.e., ∈ t ~ N(0, 1), α t = 1-β t , β t is the variance. It is worth noting that since z0 is compressed, its size is much smaller than the original image, thereby being able to significantly reduce the consumption of computing resources.

[0040] In training the conditional diffusion model, the Canny operator is used to extract key edge information from the source domain image, and through further morphological processing, refined edges are obtained as the condition y1 to guide the reverse diffusion. This step not only enhances the model's perception of image contours, but also provides clear structural guidance for the subsequent reverse diffusion process. In addition, in order to incorporate the feature information of the target domain, the target domain image is input to the encoder to extract effective information using the properties of convolution and transform it into the condition y2. In this way, it is beneficial to extract effective features in the target domain image to ensure that the generated image not only preserves the shape information of the source domain but also contains the key visual features of the target domain. Finally, by combining the two conditions y1 and y2, a comprehensive guide condition y is formed to guide the diffusion model to accurately reconstruct high-quality target images.

[0041] A neural network needs to be learned to approximate the conditional probability distribution in the reverse diffusion process. In one embodiment, the conditional diffusion model is constructed using a modified Unet network (or the conditional noise model ε θ ). Unet includes a compression path for down-sampling and an expansion path for up-sampling. The compression path and the corresponding expansion path directly splice the two groups of feature maps through short-path.

[0042] As shown in Figure 5 , the input of the conditional noise model mainly includes three parts: z t , which represents the time encoding t, the target modality effective information, and the introduced shape prior information y. A condition y is introduced at each step of diffusion to guide the noise prediction at each time step t. Then, the predicted z0 is sent to the decoder to generate the image of the missing modality. The reverse denoising process is represented by P θ (z A,t-1 |z A,t , y1, y2) aims to guide the training of ∈ θ from z t to z t-1 , as shown in the following formula:

[0043]

[0044] where ∈ θ represents the conditional noise model trained by the network.

[0045] The objective function for training the conditional diffusion model is set as:

[0046]

[0047] where the loss term L E is parameterized to minimize ∈ twith ∈ θ , ∈ t denotes a Gaussian noise term that reparameterizes the input z t at time step t, ∈ θ (z t , t, y) is the noise term predicted by the network trained with the guidance condition y = {ylconcaty2}.

[0048] In summary, the forward diffusion process is a process of gradually adding noise to the original image, and finally making the original image become a noisy image. The reverse denoising process is the opposite of the forward diffusion process, which gradually removes noise from the noisy image to finally obtain the original image. By introducing the condition y, the synthesized image with the same prior distribution is more accurately reconstructed.

[0049] Step S130, using the trained conditional latent diffusion model to realize the conversion of the source domain image to the target domain image.

[0050] The trained conditional latent diffusion model can be used for actual conversion of the source domain modality image to the target domain modality image, and is suitable for various medical image conversion tasks for organs. For example, based on known heart and brain images, one-to-one or many-to-one image conversion tasks can be realized, such as one-to-one conversion from bSSFP to LGE, or many-to-one conversion from T1, T2, Flair to T1ce.

[0051] Step S140, using the converted target domain image to construct a new data set for data augmentation of downstream tasks.

[0052] It should be understood that the conditional latent diffusion model designed by the present application can be further used for data augmentation of downstream tasks in addition to medical image conversion, that is, using the newly synthesized image data set to train a deep learning model for downstream tasks, thereby providing more image information for various clinical-oriented downstream tasks. The downstream tasks include target detection, image segmentation, and classification, and any medical image processing or analysis task.

[0053] To further analyze the effect of the present application, verification was carried out on two known data sets.

[0054] Dataset 1: The MS-CMR2019 dataset involves CMR data (refers to cardiac structure and function information acquired by cardiac magnetic resonance imaging) of 45 patients, each of which is scanned using three CMR sequences (i.e. LGE, T2 and bSSFP). In the experimental verification, only the LGE and bSSFP images among them are used. There are 35 labeled bSSFP images, which are composed of 8-12 slices with an in-plane resolution of 1.25x1.25mm and a slice thickness of 8 to 13mm. There are only 5 labeled LGE images with 10-18 slices, with an in-plane resolution of 0.75x0.75mm and a slice thickness of 5mm. The size of the image ranges from 256x256 to 512x512, and is uniformly adjusted to 256x256 pixels.

[0055] Dataset 2: The BraTs2020 dataset is provided by the multi-modal brain tumor segmentation challenge, which involves a total of 369 patients, each of which acquires images of four modalities (T1w, T1ce, T2w, Flair). In the experimental verification, the training set is randomly sorted and divided into two subsets in the ratio of 9:1, the training set includes 332 samples and the test set includes 37 samples. The 3D MR scan is cut into axial slices, and the range from 80 to 128 of the original 155 slices is selected. In addition, any blank or redundant slice with a pixel value lower than 15 is discarded. All image sizes are 240x240 and are uniformly padded to 256x256 pixels.

[0056] For dataset 1, the source domain A is composed of bSSFP images, and the target domain B is LGE images. For dataset 2, the source domain A contains three types of magnetic resonance images of T1, T2 and Flair, and the target domain B is T1ce image. After determining the source domain and the target domain, the conditional latent diffusion model is trained. Finally, the task is realized as shown in Figure 6 , realizing one-to-one conversion of bSSFP to LGE, and many-to-one conversion of T1, T2, Flair to T1ce.

[0057] To verify the effect of the present application, experiments are carried out on two data sets. Based on the two data sets of heart and brain, one-to-one and multi-to-one modal conversion are verified. For the heart, the synthetic image generated by bSSFP has a pixel distribution close to that of the real LGE image, and has a myocardial anatomical shape similar to the original bSSFP-CMR image, realizing one-to-one image conversion task. For the synthesis of brain MRI, our method performs better in generating complex sulci and tumor boundaries and can better maintain the anatomical structure of the original image, realizing multi-to-one image conversion task. The proposed method is evaluated on two quantitative indicators PSNR and SSIM, and the PSNR of the test set is 25.76±2.72 and the SSIM is 92.60±0.02. Figure 7 is a schematic diagram of the result of generating pseudo LGE from real bSSFP of the heart data set, Figure 8 is a schematic diagram of the result of generating pseudo T1ce from real T1+T2+Flair of the brain data set. By Figure 7 and Figure 8 It can be seen that the generated pseudo LGE image and pseudo T1ce image can clearly display the organ and edge details.

[0058] It should be understood that the model training process involved in the present application can be carried out offline on a server or in the cloud, and the trained model can be embedded into an electronic device to realize conversion of medical images of different modalities. The electronic device can be a terminal device or a server, and the terminal device includes any terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sale (POS), a vehicle-mounted computer, etc. The server includes but is not limited to an application server or a Web server, and can be a standalone server or a cluster server or a cloud server, etc.

[0059] In summary, compared with the prior art, the present application has the following advantages:

[0060] 1) A new conditional guided latent diffusion model is proposed, which significantly reduces the computing power required for the training process and speeds up the model training by operating in the latent space (whose dimension is much smaller than that of the pixel space). And the model constructs a new guidance training method using shape prior and effective information of target domain image to realize conditional generation.

[0061] 2) Compared with the existing GAN method, the conditional latent diffusion model designed in the present application can avoid unstable training and mode collapse, and through the fine tuning of the source image generation process and the likelihood-based optimization strategy in each iteration process, stable training and controllable generation can be obtained.

[0062] 3) To preserve anatomical structures more effectively, the present application introduces edge information extracted from source domain images as a prior of strong structure guidance, which plays a key role in guiding the diffusion process.

[0063] 4) The present application makes full use of semantic information of different images as much as possible, covering source domain and target domain images. This comprehensive information utilization strategy makes the learned network exhibit higher robustness and generalization ability.

[0064] 5) The present application can realize one-to-one and many-to-one modal conversion tasks for different organs respectively. This MRI synthesis is expected to alleviate the challenge of lack of MRI mode in clinical practice and provide data support for various clinical-oriented downstream tasks such as detection, segmentation, classification, etc.

[0065] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0066] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or punched tape, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0067] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0068] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0069] Various aspects of the present application can be described herein with reference to flowchart illustrations and / or block diagrams of machine "devices" and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0070] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all. The computer readable storage medium can also have instructions stored thereon or therein which may

[0071] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0072] The flow diagrams and block diagrams in the accompanying drawings show archi tectures, functional and operational architectures of possible implementations of systems, methods, and computer program products according to the present disclosure. In this regard, each block in the flow diagrams and block diagrams can represent a module, a segment, or a portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (acts). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in some cases, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0073] Having described various embodiments of the application, it is to be understood that the above description is meant to be illustrative only and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art, without departing from the scope and spirit of the described embodiments. The selection of terms to be used in the description is intended to best explain the principles of the embodiments, the practical application, or technical improvement over the prior art, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the application is defined by the claims appended hereto.

Claims

1. A method for multi-modal medical image conversion based on a conditional latent diffusion model, comprising the following steps: obtaining source domain modality images; compressing the original full-size images of the source domain modality images into a latent space using a pre-trained first encoder to obtain corresponding low-dimensional data; inputting the low-dimensional data into a trained conditional latent diffusion model to convert into target domain modality images; wherein the conditional latent diffusion model takes the low-dimensional data as input, uses shape information features extracted from the source domain modality images and visual information features extracted from the target domain modality images as comprehensive guidance conditions, gradually adds noise through multiple diffusion steps to obtain noise images, and then removes noise from the noise images through a denoising process, and finally uses a decoder to reconstruct the target domain modality images.

2. The method of claim 1, wherein, The noise image z t is represented as: where {z0,..., z T} are intermediate representations generated by gradually adding noise through T diffusion steps, z0 is the low-dimensional data output by the encoder, and the noise added at each diffusion step follows a Gaussian distribution denoted as ∈ t ~ N(0, 1), a t = 1 - β t , β t is a variance parameter, and t is the time step index.

3. The method of claim 1, wherein, The comprehensive guidance conditions are obtained according to the following steps: extracting edge information from the source domain modality images using a Canny operator and obtaining shape information features through morphological processing as a first condition y1; inputting the target domain modality images into a second encoder to extract visual feature information and convert into a second condition y2; combining the first condition y1 and the second condition y2 to form a comprehensive guidance condition y; introducing the comprehensive guidance condition y at each step of diffusion to guide noise prediction at each time step t.

4. The method of claim 2, wherein, The reverse denoising process of the conditional latent diffusion model is represented as: where ∈ θ (z t t, y) is a trained conditional noise model, y is a synthetic guiding condition.

5. The method of claim 4, wherein, The objective function for training the conditional latent diffusion model is set as: where L E denotes the loss, ∈ t denotes a Gaussian noise term that reparameterizes the input z t at time step t, ∈ θ (z t , t, y) is the predicted noise term.

6. The method of claim 1, wherein, The source domain modality images contain one or more modalities, and the target domain modality images contain one modality.

7. The method of claim 1, wherein, The source domain modality images are bSSFP modality MRI images, and the target domain modality images are LGE modality MRI images, or the source domain modality images are T1, T2 and Flair modality MRI images, and the target domain modality images are T1ce modality MRI images.

8. The method of claim 1, wherein, Further comprising: using the converted target domain modality images to construct a dataset for downstream tasks.

9. A computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8. The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

Images