Medical image generation method based on cyclic diffusion and target guided sampling and application

Abstract

A medical image generation method based on cycle diffusion and target guided sampling, which generates other target modality medical images from existing source modality medical images by cycle diffusion. In the noise prediction network of the diffusion model of the cycle diffusion model, an additional attention mechanism is introduced to fuse the anatomical structure information in the source modality medical image and the current noise image; at the same time, the anatomical structure segmentation network introduces an anatomical structure shape loss. In training the diffusion model, a perceptual priority weighting P2W training strategy is adopted: by adjusting the loss function weight, the diffusion model is made to preferentially learn the global structure of the image at a high noise level and optimize the image details at a low noise level; at the same time, a target guided classifier for the sampling stage is trained. In the sampling stage: first, the latent code is obtained by reverse sampling from the source modality image; then, under the guidance of the classifier, the target modality probability gradient information is used to guide the sampling process, and the generation of the target modality image is completed.

Description

Medical Image Generation Methods and Applications Based on Cyclic Diffusion and Target-Guided Sampling Technical Field

[0001] This invention relates to the field of medical image processing, specifically to a medical image generation method based on the CycleDiffusion model and a target-guided sampling strategy, for converting CT images to ultrasound images, and is particularly suitable for scenarios where the shape of anatomical structures needs to be preserved. Background Technology

[0002] Medical imaging plays a vital role in disease diagnosis, treatment planning, and efficacy evaluation. Different medical imaging modalities (such as CT, MRI, and ultrasound) provide different information, but data for some modalities may not be available due to cost, radiation, or other limitations. Therefore, generating images of other modalities using existing medical imaging data has significant clinical value.

[0003] In recent years, deep learning technology has made significant progress in the field of medical image generation. Among them, the method based on generative adversarial network (GAN) [1] has been widely used. However, GAN has problems such as unstable training and mode collapse, which makes it difficult to guarantee the quality of the generated images. Diffusion model [2] is an emerging generative model with advantages such as stable training and high generation quality, and has gradually attracted the attention of researchers. CycleDiffusion [3] is an image translation method based on diffusion model, which can realize the conversion between different image domains. It draws on the idea of ​​recurrent neural network CycleGAN and combines the technology of diffusion model. However, there are still some problems when CycleDiffusion model is directly used in medical image generation, such as:

[0004] Shape preservation problem: The generated image has a large proportion of shape distortion and is inconsistent with the anatomical structure of the original image.

[0005] Target information retention problem: The generated image retains features of the source domain that do not completely match the features of the target domain. Summary of the Invention

[0006] To address the above problems, this invention proposes a medical image generation method based on cyclic diffusion and target-guided sampling, aiming to improve the quality and shape preservation of the generated images and effectively simulate the characteristics of ultrasound images. At the same time, this method can effectively simulate the characteristics of medical image modalities such as ultrasound and has important clinical value. In this method, when training the diffusion model (Diffusion Model) used for target domain generation in CycleDiffusion, the perceptual priority weighted (P2W)[4] training strategy is adopted. Specifically, by adjusting the weight of the loss function, the diffusion model prioritizes learning the global structure when the noise level is high and optimizes the details when the noise level is low, thereby improving the quality of the generated images. At the same time, a classifier is trained for target guidance in the subsequent sampling stage.

[0007] Then, in the sampling stage, the present invention incorporates target-guided sampling: first, back-sample from the source modality image to obtain the latent code; then, under the guidance of the classifier, use the probability gradient of the target modality to complete the generation of the target modality.

[0008] Furthermore, this invention introduces an additional attention mechanism into the noise prediction network of the diffusion model, which fuses the structural information of the source modal medical image (such as CT image) with the current noisy image to generate a target modal medical image (such as ultrasound echo and texture) that conforms to physical laws.

[0009] Meanwhile, the present invention introduces the U-Net segmentation network [5] to introduce shape loss and ensure that the (anatomical structure) shape of the generated target modal medical image (such as ultrasound image) is consistent with the source modal medical image (such as original CT image).

[0010] The main contributions of this invention are as follows:

[0011] 1. When training the diffusion model for target modality generation in the CycleDiffusion model, a perceptual priority weighted training strategy is introduced. Specifically, by dynamically adjusting the weight allocation of the loss function, the diffusion model can prioritize learning global structural features in high-noise stages, while further optimizing details in low-noise stages, thereby improving the overall quality of the generated image.

[0012] 2. In the sampling phase, a target-guided sampling strategy was introduced. First, the medical images based on the source modality generated latent codes through the reverse process of the diffusion model; then, guided by the classifier, the sampling process was guided by the probability gradient information of the target modality to generate medical images consistent with the features of the target modality, thereby significantly reducing the residual features of the source modality.

[0013] 3. To ensure that the anatomical structure of the generated images is realistic and the shape remains consistent, an end-to-end segmentation-aware diffusion guidance is introduced during the generation process. This not only involves introducing the segmentation loss as a weighting term, but also deeply coupling the segmentation network with the diffusion generation process. Through a differentiable end-to-end training method, the output of the segmentation network directly feeds back the guidance term of the backdiffusion sampling, dynamically controlling the generation process and improving shape consistency.

[0014] 4. Furthermore, this invention utilizes the high signal-to-noise ratio and detailed anatomical information provided by CT images as prior knowledge to guide the diffusion model in simulating the physical process of ultrasound waves interacting with different tissue structures when generating ultrasound images. By integrating the original CT images, their structural features, and multi-level segmentation mask depth into the network structure of the diffusion model, and designing a joint loss function targeting physical characteristics such as ultrasound texture, echo intensity distribution, and attenuation mode, the model can generate ultrasound echoes and textures that conform to physical laws based on the underlying anatomical structure. Attached Figure Description

[0015] Figure 1 illustrates the overall framework of the algorithm of this invention. Detailed Implementation

[0016] To make the content of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0017] As shown in Figure 1, this invention provides a medical image generation method based on cyclic diffusion and target-guided sampling. The algorithm includes steps such as data preprocessing, model training, sampling generation, and shape / structure preservation. Table 1 shows a portion of the pseudocode implementation flow of this method.

[0018] This embodiment uses kidney CT-converted ultrasound images as an example to illustrate the method, and the steps include:

[0019] First, paired CT and ultrasound kidney images were collected as the raw dataset (source domain images). The raw data underwent preprocessing (including image registration, grayscale normalization, etc.).

[0020] Then, the kidney region is segmented from the CT kidney image to obtain a structurally accurate kidney segmentation mask.

[0021] 1. Training Phase

[0022] P2W Training Strategy: During the CycleDiffusion model training phase, this invention employs a Perception Priority Weighting (P2W) training strategy for the diffusion model used to generate the target domain (ultrasound images).

[0023] This strategy specifically improves the overall quality and detail of the synthesized image by modifying the loss function and assigning different weights to samples at different noise levels. This allows the model to prioritize learning the global structure during high-noise phases and optimize detailed information during low-noise phases.

[0024]

[0025] t: diffusion steps, used to control noise level.

[0026] Original image, Gaussian noise Noise scheduling parameters.

[0027] Noise predicted by the diffusion model.

[0028] Wherein, the weight W is defined as:

[0029]

[0030] At high noise levels (larger t), the weight W is higher, and the diffusion model focuses more on learning important global visual concepts. At low noise levels (smaller t), the weight W is lower, and the diffusion model focuses more on detail recovery. This dynamic weighting mechanism ensures that the diffusion model has a stronger ability to perceive key structures and restore details.

[0031] 2. Sampling stage

[0032] During the sampling phase, firstly, the source domain CT image is encoded into a latent space z using the DPM-Encoder. Then, during backsampling, ultrasound images are iteratively generated starting from z, and in each sampling step:

[0033] The structural segmentation mask output by the joint segmentation network is used to calculate the shape loss of the source CT structural mask in real time, and the sampling parameters or step size are dynamically adjusted to achieve adaptive closed-loop optimization of segmentation-sampling.

[0034] In the reverse iteration of the diffusion model, for each step, the generated image x t Target-guided sampling strategy:

[0035] Starting with the source modality image x, its latent encoding m is obtained through backsampling using a pre-trained diffusion model:

[0036]

[0037] y represents the label of the target modality;

[0038] The classifier is calculated based on the latent encoding. Regarding x t gradient .

[0039] Noise prediction using gradient-adjusted diffusion models:

[0040]

[0041] It is noise predicted by the diffusion model. This is the adjusted noise. Guiding strength coefficient, The structural loss guide coefficients are used. The adjusted noise is then used for the next step of image generation.

[0042] Use the adjusted noise for the next step of image generation.

[0043] 3. Shape and structure preservation

[0044] Considering the significant differences between ultrasound and CT images in imaging principles and tissue structure representation, direct modality conversion may lead to loss of structural information or the appearance of pseudo-structures. To ensure the consistency of the anatomical structure in the generated ultrasound image with that in the original CT image, this invention proposes the following structure preservation mechanism:

[0045] This invention uses CycleDiffusion to generate the skeleton, and the input is the processed kidney CT image and the corresponding structural segmentation mask.

[0046] During the diffusion model training phase, the anatomical structure segmentation network and the diffusion generation network are jointly trained end-to-end. The output of the segmentation network is used to dynamically feed back into the diffusion model to optimize the structural loss. The loss is calculated and fed back in real time at each diffusion step.

[0047] 4. Physical property simulation and diffusion modeling

[0048] To address the shortcomings of existing methods that ignore the unique physical principles of modes, diffusion models... Receive the noise image of the current diffusion step. and time step As the primary input. Within the network, particularly between the encoder and decoder modules of the U-Net structure, cross-modal attention layers are introduced. These layers enable the model to dynamically reference anatomical information provided by CT scans. Specifically, the noisy ultrasound feature map processed by the current layer of the U-Net is used as the query (Q), and the feature maps extracted from CT images and their structural information are used as the key (K) and value (V). The attention mechanism calculates the dot product of Q and K, obtains the attention weights through Softmax normalization, and applies them to V, realizing the weighting and aggregation of ultrasound features by CT information. This mechanism allows the network to be guided by CT structural information at different resolution levels, thereby establishing a correlation between CT structure and ultrasound physical response at the pixel level.

[0049] The overall loss function is designed as follows:

[0050] +

[0051] Standard diffusion reconstruction loss : Used to drive the diffusion model to learn the correct denoising process, usually the L2 loss between predicted noise and actual noise.

[0052] Structure-guided loss : Using a U-Net segmentation network S(⋅) jointly trained with a generative network, predict the generated ultrasound image G( Anatomical structure segmentation mask S(G() )). Calculate the predicted mask and the original CT segmentation mask S( The cross-entropy loss between the two images forces the generated image to be structurally consistent with the CT scan.

[0053] Ultrasonic physical property simulation loss : Used to constrain the local texture, echo intensity distribution, and statistical properties of generated ultrasound images.

[0054] In this embodiment:

[0055] As shown in Figure 1, the main process of this embodiment includes a pre-training phase and a target-guided sampling phase. Specifically, for unpaired image datasets from two different medical imaging domains (CT and ultrasound), a classifier is first constructed and trained in the pre-training phase. It is used to determine the probability that the input image belongs to the target domain.

[0056] Figure 1(a) illustrates the structure of the classifier. The main body of the classifier adopts the downsampled backbone structure of the U-Net network, and an attention pooling mechanism is introduced in the 8×8 feature layer. Specifically, multi-level features are extracted by combining residual modules with the attention mechanism, and the target domain probability distribution and its gradient are output after attention pooling. This provides a basis for subsequent target-guided sampling.

[0057] Part (b) of Figure 1 illustrates the overall process of target-guided sampling. First, the DPM-Encoder is used to process the input image. Encoding is performed to obtain a unified latent space representation. During sampling, the target domain probability gradient provided by the classifier guides the diffusion model to gradually generate target domain images. Simultaneously with sampling iterations, a structure-preserving constraint based on a U-Net segmentation network is introduced, and the segmentation loss between the generated image and the original CT image segmentation mask is calculated. This allows for effective constraints on the generated structure, ensuring that the anatomical structure of the generated ultrasound image is consistent with the original CT image.

[0058] As shown in Table 1, this invention proposes a target-enhanced DPM-Encoder algorithm based on a diffusion model. This method uses the latent code z generated during the forward diffusion process of the DPM-Encoder as the initial condition for target-guided reverse generation, and combines it with a probabilistic gradient guidance mechanism to achieve end-to-end cross-modal medical image generation without the need for a segmentation network.

[0059] Table 1 shows part of the algorithm flow of this invention.

[0060]

[0061] Specifically, during the forward diffusion phase, the source domain image After a multi-step diffusion process, a multi-scale, multi-level latent code z is obtained, which fully preserves the structure and feature representation of the source image. In the reverse generation stage, the algorithm uses the latent code z as the initial state and introduces a gradient-guided term based on the target modality classifier. The noise prediction of the diffusion model is dynamically adjusted. This guiding term effectively guides the generation process to gradually approximate the feature distribution of the target modality, significantly improving the consistency between the generated image and the target domain attributes while ensuring the inheritance of multi-scale features of the source image.

[0062] References

[0063] [1] Goodfellow, Ian J., et al. "Generative adversarial nets." Advances in neural information processing systems 27 (2014).

[0064] [2] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusionprobabilistic models." Advances in neural information processing systems 33(2020): 6840-6851.

[0065] [3] Wu, Chen Henry, and Fernando De la Torre. "Unifying diffusionmodels' latent space, with applications to cyclediffusion and guidance."arXiv preprint arXiv:2210.05559 (2022).

[0066] [4] Luo, Yimin, et al. "Target-guided diffusion models for unpairedcross-modality medical image translation." IEEE Journal of Biomedical andHealth Informatics (2024).

[0067] [5] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net:Convolutional networks for biomedical image segmentation." Medical imagecomputing and computer-assisted intervention–MICCAI 2015: 18th internationalconference, Munich, Germany, October 5-9, 2015, proceedings, part III 18.Springer international publishing, 2015。

Claims

1. A medical image generation method based on cyclic diffusion and target-guided sampling, used to generate other target modal medical images from existing source modal medical images; characterized by: The medical image generation method uses a CycleDiffusion model to generate a target modality medical image from a source modality medical image. The CycleDiffusion model includes a diffusion model for target domain generation. An additional attention mechanism is introduced into the noise prediction network of the diffusion model to fuse anatomical structure information from the source modality medical image with the current noisy image. At the same time, the anatomical structure segmentation network introduces anatomical structure shape loss. When training the diffusion model, a perceptual-priority weighted P2W training strategy is adopted: by adjusting the weights of the loss function, the diffusion model prioritizes learning the global structure of the image when the noise level is high, and optimizes the image details when the noise level is low. At the same time, a target-guided classifier is trained for the sampling stage. In the sampling stage, a target-guided sampling method is adopted: first, the latent encoding is obtained by backsampling from the source modality image; then, under the guidance of the classifier, the sampling process is guided by the probability gradient information of the target modality to complete the generation of the target modality image.

2. The medical image generation method based on cyclic diffusion and target-guided sampling according to claim 1, characterized in that: The CycleDiffusion model takes as input the processed source modal medical image and the segmentation mask of the corresponding anatomical structure. After preprocessing, the source modal medical image is then segmented into anatomical structure regions to obtain the segmentation mask of the anatomical structure. The U-Net segmentation network serves as the anatomical structure segmentation network.

3. The medical image generation method according to claim 1, characterized in that... When training the diffusion model using P2W, the loss function L is: Where t: diffusion steps, used to control the noise level; Original image, Gaussian noise Noise scheduling parameters; The noise predicted by the diffusion model; where the weight W is: At high noise levels, t is greater than the threshold, the weight W is high, and the diffusion model learns the global structure of the image; at low noise levels, t is not greater than the threshold, the weight W is low, and the diffusion model focuses on the restoration of image details.

4. The medical image generation method based on cyclic diffusion and target-guided sampling according to claim 1, characterized in that... Sampling Phase: First, the source domain image is encoded into a latent space z using a DPM-Encoder. During backsampling, the target domain image is iteratively generated starting from z. In each sampling step: a) For the segmentation mask of the anatomical structure output by the segmentation network, the shape loss of the mask is calculated in real time and compared with the corresponding structure mask of the source domain image, and the sampling parameters or step size is dynamically adjusted; b) During the back-iteration process of the diffusion model, the image x generated in each step is... t The target-guided sampling strategy is as follows: starting from the source modality image x, the latent encoding m is obtained through backsampling using a pre-trained diffusion model. y represents the label of the target modality; based on the latent encoding, a classifier is calculated. Regarding x t gradient c. Noise prediction using gradient-adjusted diffusion models: ， It is noise predicted by the diffusion model. This is the adjusted noise. It is the guiding strength coefficient. This is the structural loss guiding coefficient; d. Use the adjusted noise for the next step of image generation.

5. The medical image generation method based on cyclic diffusion and target-guided sampling according to claim 1, characterized in that: The diffusion model Receive the noise image of the current diffusion step. and diffusion steps t; A cross-modal attention layer is introduced between the encoder and decoder modules of the U-Net structure within the diffusion model network; in the decoder, the resolution of the extracted feature maps gradually increases, with low resolution corresponding to global features and high resolution corresponding to local details. Attention is introduced at different resolution levels to guide sampling using anatomical information from the source modality of medical images at different scales, thereby establishing a correlation between the physical responses of the anatomical structures of the source modality and the target source modality at the pixel level. The overall loss function of the model is designed as follows: Standard diffusion reconstruction loss : Used to drive the diffusion model to learn the correct denoising process, which is the L2 loss between the predicted noise and the actual noise; Structure-guided loss : Using the Unet segmentation network S(⋅) jointly trained with the generator network, predict the anatomical structure segmentation mask for the generated target modality medical image; The cross-entropy loss between the generated image and the segmentation mask of the source modality medical image is calculated; this loss forces the generated image to be anatomically identical to the source modality medical image; the physical property simulation loss of the target source modality medical image is also calculated. Used to constrain the local texture, echo intensity distribution, and statistical properties of the generated image.

6. The medical image generation method based on cyclic diffusion and target-guided sampling according to claim 1, characterized in that: The medical image generation method is used to convert existing CT images into ultrasound images; the anatomical structures in the two modalities of medical images have the same shape.

7. The medical image generation method based on cyclic diffusion and target-guided sampling according to claim 1, characterized in that: The CT image is a CT image of the kidney, and the ultrasound image is an ultrasound image of the kidney.

Images