Image generation method and device based on conditional diffusion model, equipment and product

Abstract

The invention relates to the technical field of image processing, and provides an image generation method and device based on a conditional diffusion model, electronic equipment and a computer program product. The method comprises the following steps: acquiring a to-be-processed target early PET image and a to-be-processed target early CT image; and inputting the target early PET image and the target early CT image into a trained conditional diffusion model for processing, fusing the functional metabolism features of the target early PET image and the anatomical structure features of the target early CT image through the conditional diffusion model, and guiding to generate the target delay PET image by taking the fused multi-modal features as conditions. According to the method, the delayed-period PET image corresponding to the early-period PET image is generated by using the conditional diffusion model, so that secondary imaging can be replaced, a patient does not need to be subjected to secondary scanning, the occupancy of the doctor-seeing time of the patient and PET / CT equipment resources is avoided, the radiation quantity accepted by the patient can be reduced, and the health risk is reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, device, electronic device, and computer program product based on a conditional diffusion model. Background Art

[0002] Prostate-Specific Membrane Antigen (PSMA) PET / CT imaging has become one of the gold standards for prostate cancer imaging diagnosis. PSMA contrast agents have significant advantages in targeting, specificity, and positive detection rate, and are particularly suitable for early diagnosis, staging assessment, and efficacy monitoring of prostate cancer.

[0003] Research has found that early PET images obtained by performing the first scan approximately one hour after PSMA drug injection often do not achieve optimal visualization. In contrast, delayed PET images obtained by scanning again more than two hours after injection can significantly enhance the contrast between tumor and background tissue, improving lesion visualization. Therefore, delayed imaging has gradually become a clinically recommended examination option.

[0004] However, if delayed imaging is used, patients must wait for a second scan, which not only prolongs their appointment time but also uses up valuable PET / CT equipment resources. Furthermore, because PET image reconstruction requires attenuation correction using CT images, a second CT image is often required, requiring the patient to undergo another CT scan with additional radiation exposure, posing potential health risks. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide an image generation method, device, electronic device and computer program product based on the conditional diffusion model, which can use the conditional diffusion model to generate delayed PET images corresponding to early PET images, so that patients do not need to undergo a second scan, which not only avoids occupying patient consultation time and PET / CT equipment resources, but also reduces the amount of radiation received by patients and reduces health risks.

[0006] A first aspect of an embodiment of the present application provides an image generation method based on a conditional diffusion model, comprising:

[0007] Acquiring a target early PET image and a target early CT image to be processed;

[0008] The target early PET image and the target early CT image are input into the trained conditional diffusion model for processing. The functional metabolic features of the target early PET image are fused with the anatomical structural features of the target early CT image through the conditional diffusion model. The multimodal features obtained after fusion are used as conditions to guide the generation of the target delayed phase PET image.

[0009] In the technical solution of the embodiment of the present application, a conditional diffusion model is pre-trained. After acquiring the target early PET image and target early CT image to be processed, they are input into the conditional diffusion model for processing. The conditional diffusion model can fuse the functional metabolic features of the target early PET image with the anatomical structural features of the target early CT image into a multimodal feature. This multimodal feature is then used as a condition to guide the image diffusion process, ultimately generating a target delayed-phase PET image corresponding to the target early PET image. This process uses the early PET / CT image as input and, by introducing the functional metabolic features and anatomical structural features as conditional controls, can deeply fuse functional and structural information, thereby guiding the conditional diffusion model to generate realistic and accurate delayed-phase PET images. This can replace secondary imaging, meaning that patients do not need a second scan, thus avoiding the need for patient consultation time and PET / CT equipment resources, while also reducing the amount of radiation received by patients and lowering health risks.

[0010] In one implementation of the embodiment of the present application, the conditional diffusion model includes a feature extraction network; and the functional metabolic features of the target early PET image and the anatomical structural features of the target early CT image are fused through the conditional diffusion model, including:

[0011] Functional metabolic features and anatomical structure features are extracted respectively through two feature extraction paths of the feature extraction network;

[0012] The extracted functional metabolic features and anatomical structural features are fused to obtain multimodal features.

[0013] In one implementation of the embodiment of the present application, the extracted functional metabolic features and anatomical structural features are fused, including:

[0014] A weighted fusion of functional metabolic features and anatomical structural features is performed based on the following formula:

[0015] F fused =α·F PET +(1-α)·F CT

[0016] Among them, F fused represents multimodal features, F PET represents the functional metabolic characteristics, F CT represents the anatomical structure feature, and α is the adjustable fusion coefficient.

[0017] In one implementation of the embodiment of the present application, the conditional diffusion model is trained in the following manner:

[0018] Acquiring corresponding historical early PET images, historical early CT images, and historical delayed PET images;

[0019] Performing rigid registration, region of interest cropping, and resampling on the historical early PET images, historical early CT images, and historical delayed PET images in sequence to obtain historical early PET images, historical early CT images, and historical delayed PET images of uniform size;

[0020] The conditional diffusion model was trained using historical early PET images, historical early CT images and historical delayed PET images of uniform size as training sets.

[0021] In one implementation of the embodiment of the present application, the conditional diffusion model includes a conditional controlled diffusion network; using the multimodal features obtained after fusion as conditions to guide the generation of a target delayed-phase PET image, including:

[0022] The multimodal features and preset noise are input into the conditional controlled diffusion network, and the multimodal features are used as conditions to guide the image diffusion process, thereby generating the target delayed period PET image.

[0023] In one implementation of an embodiment of the present application, the conditional controlled diffusion network includes a conditional control path and a noise processing path; the conditional control path includes a trainable encoder, a trainable intermediate block and a zero convolution layer, and the noise processing path includes a fixed encoder and a fixed intermediate block.

[0024] In one implementation of the embodiment of the present application, after generating the target delayed period PET image, the method further includes:

[0025] The image consistency between the target delayed-phase PET image and the corresponding real delayed-phase PET image is determined by consistency analysis verification;

[0026] Based on image consistency, the reliability of target delayed phase PET images in clinical quantitative analysis was determined.

[0027] A second aspect of an embodiment of the present application provides an image generation device based on a conditional diffusion model, comprising:

[0028] An image acquisition module, used for acquiring a target early PET image and a target early CT image to be processed;

[0029] The image generation module is used to input the target early PET image and the target early CT image into the trained conditional diffusion model for processing, fuse the functional metabolic features of the target early PET image with the anatomical structural features of the target early CT image through the conditional diffusion model, and use the multimodal features obtained after fusion as conditions to guide the generation of the target delayed phase PET image.

[0030] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the image generation method based on the conditional diffusion model provided in the first aspect of the embodiment of the present application.

[0031] A fourth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the image generation method based on the conditional diffusion model provided in the first aspect of the embodiments of the present application.

[0032] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the image generation method based on the conditional diffusion model provided in the first aspect of the embodiment of the present application.

[0033] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of an image generation method based on a conditional diffusion model provided in an embodiment of the present application;

[0035] Figure 2 This is a schematic diagram of the operation flow of a training data set required for constructing a conditional diffusion model provided by an embodiment of the present application;

[0036] Figure 3 This is a schematic diagram of the network structure of a conditional diffusion model provided in an embodiment of the present application;

[0037] Figure 4 3. This is a schematic diagram of the consistency analysis results of the delayed phase PET image obtained by the image generation method provided in the embodiment of the present application and the corresponding real image in terms of the standard uptake value index;

[0038] Figure 5 1 is a schematic structural diagram of an image generation device based on a conditional diffusion model provided in an embodiment of the present application;

[0039] Figure 6This is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details. In addition, in the description of the present application specification and the appended claims, the terms "first," "second," "third," etc. are only used to distinguish descriptions and are not to be understood as indicating or implying relative importance.

[0041] Currently, clinical delayed PET / CT imaging requires patients to wait approximately two hours after completing their initial scan before undergoing another scan to obtain delayed PET images, which improve tumor contrast and lesion visibility. This process significantly prolongs a single examination, limiting equipment efficiency. Furthermore, users are subject to additional radiation exposure from CT scans, posing a high risk of radiation exposure. Furthermore, traditional image generation methods based on generative adversarial networks (GANs), or U-Nets, often suffer from issues such as insufficient image fidelity when generating delayed PET images.

[0042] In response to the above problems, the embodiments of the present application propose an image generation method, device, electronic device and computer program product based on the conditional diffusion model. By introducing functional metabolic characteristics and anatomical structural characteristics as conditional controls, it is possible to deeply integrate functional information with structural information, thereby guiding the conditional diffusion model to generate realistic and accurate delayed-phase PET images. In this way, the patient does not need to undergo a second scan, which not only avoids taking up the patient's consultation time and PET / CT equipment resources, but also reduces the amount of radiation received by the patient and reduces health risks. For more specific technical implementation details of the embodiments of the present application, please refer to the various method embodiments described below.

[0043] It should be understood that the execution subjects of the various method embodiments proposed in the present application can be various types of electronic devices, such as mobile phones, tablet computers, desktop computers, wearable devices, medical devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of the present application do not impose any restrictions on the specific types of the electronic devices.

[0044] See also Figure 1 , shows an image generation method based on a conditional diffusion model provided by an embodiment of the present application, including:

[0045] 101. Acquire a target early PET image and a target early CT image to be processed;

[0046] First, a target early PET image and a target early CT image to be processed are obtained. The target early PET image refers to a PET image obtained during the initial scan of the current patient, and the target early CT image refers to a CT image obtained during the initial scan of the current patient. The technical solution of the embodiment of the present application requires generating a predicted delayed-phase PET image of the current patient based on the image features of the target early PET image and the target early CT image, referred to as the target delayed-phase PET image. The target early PET image, target early CT image, and target delayed-phase PET image correspond to each other.

[0047] 102. The target early PET image and the target early CT image are input into the trained conditional diffusion model for processing. The functional metabolic features of the target early PET image are fused with the anatomical structural features of the target early CT image through the conditional diffusion model, and the multimodal features obtained after fusion are used as conditions to guide the generation of the target delayed phase PET image.

[0048] The technical solution of the embodiment of the present application requires pre-training a conditional diffusion model to fuse the functional metabolic features of early PET images with the anatomical structural features of early CT images, and use the fused multimodal features as conditions to guide the image diffusion process, ultimately generating delayed-phase PET images with high fidelity and anatomical consistency, thereby improving diagnostic efficiency, reducing radiation risks, and enhancing the clinical usability of image quality. The training method of the conditional diffusion model is first described below.

[0049] In one implementation of the embodiment of the present application, the conditional diffusion model is trained in the following manner:

[0050] (1) Obtaining corresponding historical early PET images, historical early CT images, and historical delayed PET images;

[0051] (2) performing rigid registration, region of interest cropping, and resampling on the historical early PET images, historical early CT images, and historical delayed PET images in sequence to obtain historical early PET images, historical early CT images, and historical delayed PET images of uniform size;

[0052] (3) Using historical early PET images, historical early CT images, and historical delayed PET images of uniform size as training sets, a conditional diffusion model is trained.

[0053] Training the conditional diffusion model requires a certain number of corresponding historical early PET images, historical early CT images, and historical delayed PET images as the model's training set. Specifically, a large number of historical patients' early PET images, early CT images, and corresponding delayed PET images collected by a PET / CT integrated machine can be obtained. Since the initial sizes of PET images and CT images are usually different, in order to ensure image spatial consistency and facilitate the processing of the conditional diffusion model, these historical images can first be preprocessed. The preprocessing operations can include rigid alignment, region of interest cropping, and resampling. Among them, rigid alignment can use an affine alignment method with three-dimensional linear interpolation to achieve spatial alignment of corresponding historical early PET images, historical early CT images, and historical delayed PET images, and find the corresponding spatial mapping relationship. Region of interest cropping is to crop out key areas of interest from historical images. For example, for the PSMA scenario, the pelvic area of ​​the human body can be cropped as the region of interest (ROI). This can remove interference from irrelevant image areas and reduce the computational complexity of image processing. Afterwards, all historical images are resampled to unify the image size (for example, using 256*256 pixels and aligning the number of slices) to adapt to the input of the conditional diffusion model. In addition, in order to eliminate the problem of inconsistent brightness distribution of cross-layer images, patient-level normalization processing, such as Z-score standardization, can be performed on the PET images and CT images of each patient. Through the above image preprocessing operations, historical early PET images, historical early CT images, and historical delayed PET images of uniform size can be obtained, and these historical images serve as the constructed data set. The data set can be divided into a training set and a test set according to a set ratio, and a conditional diffusion model can be trained based on the training set using the training method of a machine learning model.

[0054] As an example, Figure 2 This is a schematic diagram of the operation flow of a training data set required to construct a conditional diffusion model provided by an embodiment of the present application. Figure 2 In the process, a large number of corresponding historical images were first collected, including early PET images, early CT images, and delayed PET images. It can be seen that the size of CT images is different from that of PET images. Then, these historical images were sequentially processed through affine registration, region of interest cropping, and resampling. Ultimately, the uniformly sized (256*256*150) early PET images, early CT images, and delayed PET images were obtained as the training dataset for the conditional diffusion model.

[0055] The conditional diffusion model can fuse the functional metabolic features of early PET images with the anatomical structural features of early CT images. It uses these fused multimodal features as conditions to guide the image diffusion process, thereby generating the corresponding delayed PET images. Next, we describe the network structure and specific working principles of this conditional diffusion model.

[0056] In one implementation of the embodiment of the present application, the conditional diffusion model includes a feature extraction network; and the functional metabolic features of the target early PET image and the anatomical structural features of the target early CT image are fused through the conditional diffusion model, including:

[0057] (1) Through the two feature extraction paths of the feature extraction network, functional metabolic features and anatomical structure features are extracted respectively;

[0058] (2) The extracted functional metabolic features and anatomical structural features are integrated to obtain multimodal features.

[0059] The network front-end of the conditional diffusion model can be equipped with a feature extraction network, which includes two feature extraction paths for processing early PET images and early CT images, respectively, to extract functional metabolic features and anatomical structural features. In practice, the feature extraction network can adopt a convolutional neural network (CNN), and both feature extraction paths can use a backbone network based on ResNet-N (residual network, N-layer structure), which can better capture image features such as regional texture, organ boundaries, and lesion morphology. In addition, to further optimize network performance or adapt to the feature distribution of different types of medical images, the backbone network can be replaced with other deeper or lighter deep neural network models, such as Vision Transformer, EfficientNet, and DenseNet, to improve the expressive power of feature extraction, model training stability, and inference efficiency. In addition, to enhance the model's ability to detect small lesions or areas with blurred edges, multi-scale feature extraction and fusion mechanisms can be introduced, such as Feature Pyramid Network (FPN) or skip connections, to achieve joint perception of information at different spatial scales.

[0060] After processing the target early PET image through one feature extraction path, it outputs a high-dimensional feature map with a certain number of channels (e.g., 512) to represent functional metabolic features. After processing the target early CT image through another feature extraction path, it also outputs a high-dimensional feature map with a certain number of channels (e.g., 512) to represent anatomical features. These two high-dimensional feature maps are then used for fusion. The extracted functional metabolic features and anatomical features are then fused together, resulting in a stable multimodal joint representation that fully integrates metabolic and structural information, also known as a multimodal feature.

[0061] In one implementation of the embodiment of the present application, the extracted functional metabolic features and anatomical structural features are fused, including:

[0062] A weighted fusion of functional metabolic features and anatomical structural features is performed based on the following formula:

[0063] F fused =α·F PET +(1-α)·F CT

[0064] Among them, F fused represents multimodal features, F PET represents the functional metabolic characteristics, F CT represents the anatomical structure feature, and α is the adjustable fusion coefficient.

[0065] In order to improve the collaborative expression ability of functional features and structural features, an adaptive weighted fusion module can be introduced to use the above formula to transform the functional metabolic features F of early PET images into PET Anatomical features of early CT images F CT Weighted fusion is multimodal feature F fused Among them, α is the adjustable fusion coefficient, which is used to adjust the fusion weight. For example, when α = 0.9, the functional metabolic feature F PET The dominance of anatomical structure feature F is stronger, and when α=0.1, CTThe fusion process is more dominant. On the one hand, users can reasonably set the value of the adjustable fusion coefficient α according to the actual scenario requirements. On the other hand, the fusion process also supports adaptive weight adjustment. For example, it can automatically optimize the contribution ratio of PET information and CT information based on image quality. For example, when the quality score of early PET images is high, α is set to a higher value, and vice versa. In addition, to enhance the model's ability to adaptively adjust to changes in the importance of different modalities, the fusion strategy can be replaced with a channel attention mechanism (such as the Squeeze-and-Excitation Block, SE Block) or a spatial attention mechanism (such as the Convolutional Block Attention Module, CBAM) to dynamically adjust the fusion weights of PET and CT information at different spatial locations or channel dimensions. This setting allows for flexible adjustment of the fusion ratio of functional metabolic features to anatomical structural features, thereby obtaining high-precision multimodal features that better meet the needs of actual scenarios.

[0066] The multimodal features obtained after fusion are embedded in a conditional controlled diffusion network, which serves as conditional input to guide the image generation process. During the diffusion network's reverse denoising iterations, a special anatomical guidance path is designed to enhance structural consistency and metabolic expression accuracy, ensuring that the generated delayed-phase PET images approach the true delayed scan results in terms of both visual fidelity and functional feature reproduction. Next, the basic principles of the conditional controlled diffusion network are described.

[0067] In one implementation of the embodiment of the present application, the conditional diffusion model includes a conditional controlled diffusion network; using the multimodal features obtained after fusion as conditions to guide the generation of a target delayed-phase PET image, including:

[0068] The multimodal features and preset noise are input into the conditional controlled diffusion network, and the multimodal features are used as conditions to guide the image diffusion process, thereby generating the target delayed period PET image.

[0069] The conditional control diffusion network can use the pre-trained stable diffusion model (Stable Diffusion) as the main generation architecture, and transform the above multimodal features F fusedand preset noise as input, and multimodal features as conditions to guide the image diffusion process, thereby generating the target delayed-phase PET image. Specifically, the fused feature information of the multimodal features can be injected into the trainable control path within the network encoder, and a zero-initialized convolution control method is adopted to enable the structural feature information and metabolic feature information to work synergistically in the diffusion reconstruction process. The fused features are injected into the control path of the stable diffusion model through the zero-initialized convolution layer, guiding the image generation process to pay more attention to anatomical alignment and boundary consistency, thereby significantly improving the clarity of the generated image in the lesion area, the target-background contrast, and the structural restoration ability, and ultimately generating a delayed-phase PET image that maintains high fidelity in both anatomical structure and metabolic distribution.

[0070] In one implementation of an embodiment of the present application, the conditional controlled diffusion network includes a conditional control path and a noise processing path; the conditional control path includes a trainable encoder, a trainable intermediate block and a zero convolution layer, and the noise processing path includes a fixed encoder and a fixed intermediate block.

[0071] The conditional control diffusion network can include two processing paths, namely the conditional control path and the noise processing path, where the conditional control path is based on the multimodal feature F fused As input, it includes a trainable encoder, a trainable intermediate block and a zero convolution layer. The zero convolution layer can be used to control the injection path of conditional information. By adopting a zero initialization strategy, it can ensure that the model behaves equivalently to an unconditional diffusion model in the early stage of training to enhance training stability. The noise processing path takes preset noise as input, including a fixed encoder and a fixed intermediate block. The noise is gradually added to the image and then removed through the back diffusion process. Under the influence of control conditions, the generated image is guided to develop in a direction that meets the conditions, and finally a high-fidelity delayed-phase PET image is generated.

[0072] As an example, Figure 3 This is a schematic diagram of the network structure of a conditional diffusion model provided in an embodiment of the present application. Figure 3 In

[15] , the conditional diffusion model includes a feature extraction network and a conditional control diffusion network, wherein the feature extraction network includes a functional metabolic feature extraction path and an anatomical structure feature extraction path, and the conditional control diffusion network includes a conditional control path and a noise processing path. The conditional control path inserts an independent control branch in each encoder block to inject conditional features, and the multimodal feature F fused The input is fed into the conditional control path and processed sequentially through multiple trainable encoders, trainable intermediate blocks, and multiple zero-initialized convolutional layers, which can be injected into the image diffusion process as conditional features. The noise processing path includes multiple encoders, intermediate blocks, and convolutional layers with fixed parameters, which perform the process of adding and removing noise under the action of conditional features, and finally generate the corresponding delayed PET image. In general, the multimodal feature Ffused By injecting zero-initialized convolutional layers into the control path of the conditional diffusion model, the image generation process is guided to pay more attention to anatomical alignment and boundary consistency, thereby significantly improving the clarity, target-background contrast and structural restoration capabilities of the generated images in the lesion area.

[0073] Specifically, assume that G(*; Θ) is a trainable neural block with parameters Θ, which transforms the input feature map x(h*w*v, height*width*number of channels) into another feature map y, that is, y=G(x; Θ). Let the main diffusion module (that is, the locked stable diffusion module part) be G(x; Θ) c ), then the output after modulation by the control branch is:

[0074] y c =G(x;Θ c )+Z2(G(x+Z1(x;Θ Z1 );Θ c );Θ Z2 )

[0075] Where c represents the conditional vector of the pre-training input, Z1 and Z2 are two zero convolution layers (1×11×1 convolution, weights and biases initialized to zero), Θ c are the parameters of the trainable replica network (cloned from the original network G, but allowed to be trained), Θ Z1 and Θ Z2 are the parameters of the two zero convolution layers, y c In the initial case, the zero convolution weight is zero, so y c =y, which can avoid noise interference in the deep features of the pre-trained model. During the model training process, the zero convolution parameters are gradually learned from scratch, and conditional control is gradually introduced to convert the multimodal feature F fused As feature map x input, output y c This is the generated delayed PET image.

[0076] The image generation process can use a standard 1000-step diffusion inversion mechanism to gradually refine image noise and guide the generation of the target delayed-phase PET image. In addition, for the optimization of the conditional diffusion model, a joint optimization strategy can be introduced during the training process, combining the constraint structure loss, metabolic difference loss, and fusion consistency indicators to improve the overall stability and diagnostic relevance of image quality. Specifically, the pixel-level L2 loss function can be introduced as follows:

[0077]

[0078] Among them, N represents the number of pixels, It represents the difference in pixel features between the delayed phase PET image predicted by the model and the actual delayed phase PET image.

[0079] At the same time, the reconstruction error loss function combined with the conditional diffusion model is as follows:

[0080]

[0081] Among them, z0 represents the original noise, t is the diffusion time step (number of times noise is added), z t represents the noise diffused after t steps, c t is the text prompt condition, c f is the fusion image condition feature, ∈ is the real noise that obeys the standard normal distribution, ∈ θ is the noise prediction network of the conditional diffusion model, whose parameter is θ.

[0082] The training goal of the conditional diffusion model is to enable the model to gradually remove noise from noisy images by minimizing the L2 distance between the predicted noise and the real noise, while integrating the text condition c t and image condition c f By designing this loss function, we can simultaneously optimize the image's structural restoration and metabolic fidelity.

[0083] In order to evaluate the actual value of the delayed phase PET images predicted by the model in clinical analysis, a consistency analysis technique can be introduced. In one implementation of the embodiment of the present application, after generating the target delayed phase PET images, the following is also included:

[0084] (1) Determine the image consistency between the target delayed-phase PET image and the corresponding real delayed-phase PET image through consistency analysis verification;

[0085] (2) Based on image consistency, determine the reliability of target delayed phase PET images in clinical quantitative analysis.

[0086] Specifically, consistency analysis methods such as Bland-Altman can be used to verify the image consistency between the target delayed phase PET image predicted by the model and the corresponding actual delayed phase PET image. Here, the size of the SUV (Standard Uptake Value) indicator can be verified. Then, based on the verified image consistency, the reliability of the target delayed phase PET image in clinical quantitative analysis can be determined. For example, if the image consistency is higher, it means that the predicted target delayed phase PET image is closer to the actual delayed phase PET image, and therefore the reliability of the target delayed phase PET image in clinical quantitative analysis is higher, and so on.

[0087] As an example, Figure 4It is a schematic diagram of the consistency analysis results of the delayed phase PET image obtained by using the image generation method provided in the embodiment of the present application and the corresponding real image in terms of standard uptake value indicators. Figure 4 A Bland-Altman consistency analysis was performed on the standard uptake values ​​of the lesion areas in typical cases. The results showed that the SUV index of the delayed-phase PET images generated by the embodiment of the present application was highly consistent with the SUV index of the corresponding real delayed-phase PET images, and there was no systematic deviation within the 95% confidence interval. This verifies the reliability of the technical solution of the embodiment of the present application in clinical quantitative analysis, and has better results than traditional image generation solutions such as CycleGAN and SwinIR.

[0088] In the technical solution of the embodiment of the present application, a conditional diffusion model is pre-trained. After acquiring the target early PET image and target early CT image to be processed, they are input into the conditional diffusion model for processing. The conditional diffusion model can fuse the functional metabolic features of the target early PET image with the anatomical structural features of the target early CT image into a multimodal feature. This multimodal feature is then used as a condition to guide the image diffusion process, ultimately generating a target delayed-phase PET image corresponding to the target early PET image. This process uses the early PET / CT image as input and, by introducing the functional metabolic features and anatomical structural features as conditional controls, can deeply fuse functional and structural information, thereby guiding the conditional diffusion model to generate realistic and accurate delayed-phase PET images. This can replace secondary imaging, meaning that patients do not need a second scan, thus avoiding the need for patient consultation time and PET / CT equipment resources, while also reducing the amount of radiation received by patients and lowering health risks.

[0089] The technical solution of the embodiment of the present application has been verified through sufficient experiments, simulations and actual use, and has been proven to have good feasibility and practicality in generating high-quality delayed-phase PET images. The following is a detailed description from three aspects: quantitative indicator evaluation, subjective visual judgment and clinical interpretability.

[0090] For quantitative evaluation, the image quality of the generated delayed PET images can be assessed using three metrics: peak signal-to-noise ratio (PSNR) (which measures image restoration quality), structural similarity index (SSIM) (which assesses anatomical consistency), and root mean square error (RMSE) (which reflects pixel error). Through testing and validation on a large number of patient datasets, the quantitative metric test results are shown in Table 1 below.

[0091] Table 1 Comparison of quantitative indicators of the proposed method and existing methods on the test set

[0092]

[0093] As can be seen from Table 1, the images generated by the method provided by this application (Ours) have the highest peak signal-to-noise ratio (PSNR), the lowest root mean square error (RMSE), and the highest structural similarity index (SSIM). In other words, it outperforms other comparison models in all image quality evaluation indicators, especially in the SSIM and RMSE indicators. This indicates that the generated images have higher fidelity in terms of structural consistency and metabolic reduction.

[0094] Regarding subjective visual judgment, through subjective visual comparison with mainstream image generation methods such as SwinIR, CycleGAN, and U-Net on typical case images, the delayed-phase PET images generated by the embodiment of the present application show higher clarity, structural restoration, and lesion contrast in visual quality. The image details better restore the morphological boundaries and metabolic activity distribution of the lesion area, and have good readability.

[0095] Regarding clinical interpretability, in order to further verify the usability of the generated images in clinical scenarios, a number of medical imaging doctors have been invited to conduct "image quality scoring" and "diagnostic consistency scoring" on the generated delayed phase PET images and the real delayed phase PET images without annotating the source of the images. The results show that the delayed phase PET images generated by the embodiment of the present application are highly consistent with the real delayed phase PET images in the doctor's scoring, which shows that the delayed phase PET images generated by the embodiment of the present application have high substitutability and reference value in actual diagnostic work.

[0096] It can be seen that the technical solution of the embodiment of the present application is superior to the existing representative image generation method in terms of quantitative accuracy, structural restoration and consistency of clinical interpretation, and has good stability, accuracy and clinical feasibility.

[0097] In summary, the embodiments of the present application not only realize the generation of high-quality delayed-phase PET images without scanning conditions, but also solve the problems of missing structural information, image blur and metabolic expression distortion in existing methods from a technical path. It can accurately restore the lesion morphology and metabolic activity distribution while maintaining the integrity of structural details, and take into account the diagnosis and treatment efficiency, patient safety and model stability while improving image quality. It has good clinical application prospects and technological innovation value.

[0098] As an extension, the image generation method based on the multimodal conditional diffusion model proposed in the embodiment of the present application has good versatility and scalability. In addition to being used for delayed PET image generation, it can also be widely used in other types of medical imaging tasks, including but not limited to: medical image enhancement and fMRI time series completion.

[0099] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0100] The above mainly describes an image generation method based on the conditional diffusion model. The following will describe an image generation device based on the conditional diffusion model.

[0101] See also Figure 5 , shows an image generation device based on a conditional diffusion model provided by an embodiment of the present application, comprising:

[0102] An image acquisition module 501 is used to acquire a target early PET image and a target early CT image to be processed;

[0103] Image generation module 502 is used to input the target early PET image and the target early CT image into the trained conditional diffusion model for processing, fuse the functional metabolic features of the target early PET image with the anatomical structural features of the target early CT image through the conditional diffusion model, and use the multimodal features obtained after fusion as conditions to guide the generation of the target delayed phase PET image.

[0104] In one implementation of the embodiment of the present application, the conditional diffusion model includes a feature extraction network; and the image generation module includes:

[0105] A feature extraction unit, configured to extract functional metabolic features and anatomical structure features respectively through two feature extraction paths of the feature extraction network;

[0106] The feature fusion unit is used to fuse the extracted functional metabolic features and anatomical structure features to obtain multimodal features.

[0107] In one implementation of the embodiment of the present application, the feature fusion unit includes:

[0108] The feature fusion subunit is used to perform weighted fusion of functional metabolic features and anatomical structure features based on the following formula:

[0109] F fused =α·F PET +(1-α)·F CT

[0110] Among them, F fused represents multimodal features, F PET represents the functional metabolic characteristics, F CT represents the anatomical structure feature, and α is the adjustable fusion coefficient.

[0111] In one implementation of the embodiment of the present application, the image generation device based on the conditional diffusion model further includes:

[0112] A historical data acquisition module is used to acquire corresponding historical early PET images, historical early CT images, and historical delayed PET images;

[0113] an image processing module, configured to sequentially perform rigid registration, region of interest cropping, and resampling processing on the historical early PET images, the historical early CT images, and the historical delayed PET images to obtain the historical early PET images, the historical early CT images, and the historical delayed PET images of uniform size;

[0114] The model training module is used to train a conditional diffusion model using historical early PET images, historical early CT images, and historical delayed PET images of uniform size as training sets.

[0115] In one implementation of the embodiment of the present application, the conditional diffusion model includes a conditional controlled diffusion network; and the image generation module includes:

[0116] The image generation unit is used to input the multimodal features and preset noise into the conditional controlled diffusion network, and use the multimodal features as conditions to guide the image diffusion process, thereby generating a target delayed period PET image.

[0117] In one implementation of an embodiment of the present application, the conditional controlled diffusion network includes a conditional control path and a noise processing path; the conditional control path includes a trainable encoder, a trainable intermediate block and a zero convolution layer, and the noise processing path includes a fixed encoder and a fixed intermediate block.

[0118] In one implementation of the embodiment of the present application, the image generation device based on the conditional diffusion model further includes:

[0119] a consistency analysis module, configured to determine the image consistency between the target delayed-phase PET image and the corresponding actual delayed-phase PET image by means of consistency analysis verification;

[0120] The reliability determination module is used to determine the reliability of the target delayed period PET image in clinical quantitative analysis based on image consistency.

[0121] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the image generation method based on the conditional diffusion model described in any of the above embodiments is implemented.

[0122] An embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the image generation method based on the conditional diffusion model as described in any of the above embodiments.

[0123] Figure 6 Schematic diagram of an electronic device provided by an embodiment of the present application. Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned embodiments of the image generation method based on the conditional diffusion model are implemented, for example Figure 1 Alternatively, when the processor 60 executes the computer program 62, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 5 Functions of modules 501 - 502 of the illustrated apparatus.

[0124] The computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the electronic device 6.

[0125] The processor 60 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0126] The memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. The memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. Furthermore, the memory 61 may include both an internal storage unit of the electronic device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the electronic device. The memory 61 may also be used to temporarily store data that has been output or is about to be output.

[0127] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0128] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0130] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0133] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0135] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An image generation method based on a conditional diffusion model, characterized in that: include: Acquiring a target early PET image and a target early CT image to be processed; The target early PET image and the target early CT image are input into a trained conditional diffusion model for processing. The functional metabolic features of the target early PET image are fused with the anatomical structural features of the target early CT image through the conditional diffusion model, and the multimodal features obtained after the fusion are used as conditions to guide the generation of the target delayed phase PET image.

2. The method according to claim 1, wherein The conditional diffusion model includes a feature extraction network; fusing the functional metabolic features of the target early PET image with the anatomical structural features of the target early CT image through the conditional diffusion model includes: respectively extracting the functional metabolic features and the anatomical structure features through two feature extraction paths of the feature extraction network; The extracted functional metabolic features and the extracted anatomical structural features are fused to obtain the multimodal features.

3. The method according to claim 2, wherein The fusing of the extracted functional metabolic features and the anatomical structure features comprises: A weighted fusion is performed on the functional metabolic features and the anatomical structure features based on the following formula: F fused =α·F PET +(1-α)·F CT Among them, F fused represents the multimodal features, F PET Indicates the functional metabolic characteristics, F CT represents the anatomical structure feature, and α is an adjustable fusion coefficient.

4. The method according to claim 1, wherein The conditional diffusion model is trained in the following way: Acquiring corresponding historical early PET images, historical early CT images, and historical delayed PET images; performing rigid registration, region of interest cropping, and resampling processing on the historical early PET image, the historical early CT image, and the historical delayed PET image in sequence to obtain the historical early PET image, the historical early CT image, and the historical delayed PET image of uniform size; The conditional diffusion model is obtained by training using the historical early PET images, the historical early CT images, and the historical delayed PET images of uniform size as training sets.

5. The method according to claim 1, wherein The conditional diffusion model includes a conditional controlled diffusion network; the method uses the multimodal features obtained after fusion as a condition to guide the generation of a target delayed phase PET image, including: The multimodal features and preset noise are input into the conditional controlled diffusion network, and the multimodal features are used as conditions to guide the image diffusion process, thereby generating the target delayed period PET image.

6. The method according to claim 5, wherein The conditional controlled diffusion network includes a conditional control path and a noise processing path; the conditional control path includes a trainable encoder, a trainable intermediate block and a zero convolution layer, and the noise processing path includes a fixed encoder and a fixed intermediate block.

7. The method according to any one of claims 1 to 6, wherein: After generating the target delayed period PET image, the method further includes: Determining the image consistency between the target delayed-phase PET image and the corresponding actual delayed-phase PET image by means of consistency analysis verification; The reliability of the target delayed-phase PET image in clinical quantitative analysis is determined based on the image consistency.

8. An image generation device based on a conditional diffusion model, characterized in that: include: An image acquisition module, used for acquiring a target early PET image and a target early CT image to be processed; An image generation module is configured to input the target early PET image and the target early CT image into a trained conditional diffusion model for processing, fuse the functional metabolic features of the target early PET image with the anatomical structural features of the target early CT image through the conditional diffusion model, and use the multimodal features obtained after the fusion as a condition to guide the generation of a target delayed-phase PET image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image generation method based on the conditional diffusion model according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to execute the image generation method based on the conditional diffusion model according to any one of claims 1 to 7.

Images