A method for generating synthetic sCT images from CBCT based on diffusion model

Abstract

The present invention discloses a method for generating a CBCT to synthetic sCT image based on a diffusion model, which belongs to the field of medical imaging technology and includes the following processes: forward process: obtaining the original CT image and gradually adding Gaussian noise to form a noise image; reverse process: performing a denoising operation on the noise image, and restoring the noise image to an sCT image by adding the CBCT image as a conditional input of the denoising process. The present invention not only significantly improves the efficiency and quality of image conversion, but also has better practicality and broad application prospects; the present invention is applicable to the field of medical imaging, especially in radiotherapy planning, to provide patients with more accurate dose calculation and treatment effect evaluation.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a method for generating a CBCT to a synthetic sCT image based on a diffusion model. Background Art

[0002] In radiotherapy planning, CBCT is often used for patient positioning, but its direct use for accurate dose calculation is limited due to its low soft tissue contrast and noise level. Therefore, it is necessary to convert (cone beam computed tomography) CBCT images into higher quality sCT (computed tomography) images. Traditional image conversion methods are often based on complex image processing techniques, which are not only computationally expensive, but also often have difficulty in dealing with noise and detail loss in the images. Summary of the invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide a CBCT to sCT image conversion technology based on a diffusion model, which aims to effectively remove image noise while maintaining image details and texture, and accelerate the inference speed of image conversion.

[0004] In order to achieve the above technical objectives, the present application provides a method for generating a CBCT to a synthetic sCT image based on a diffusion model, comprising the following process:

[0005] Forward process: Get the original CT image and gradually add Gaussian noise to form a noise image;

[0006] Reverse process: denoising operation is performed on the noisy image, and the noisy image is restored to the sCT image by adding the CBCT image as the conditional input of the denoising process.

[0007] Preferably, in the forward process, a noise image is formed based on the original CT image through a forward noise adding process, wherein the forward noise adding process is expressed as:

[0008]

[0009] In the formula, α t represents the parameter controlling the noise level, ∈ t represents the noise drawn from a standard normal distribution, x t+1 It represents the noise pattern formed after noise is gradually added to the CT image x0.

[0010] Preferably, in the reverse process, the denoising process is expressed as:

[0011]

[0012] In the formula, ∈ θ (x t, t) is the noise estimated by the neural network, σ t is a time-dependent noise scaling factor that generally decreases with time.

[0013] Preferably, in the inverse process, a parameterized neural network is used for denoising, the noisy image is input into a main branch of the parameterized neural network, and the CBCT image is input into a conditional branch;

[0014] The encoder is used to encode the main branch image and the conditional branch image, and Fourier transform and wavelet transform are used to optimize the frequency domain features extracted in the encoder. The self-attention mechanism is used to fuse the feature maps of the CBCT branch and the sCT branch. The image is restored through a batch normalization layer and a residual block and a decoder to generate an sCT image.

[0015] Preferably, in the process of generating the sCT image, the encoder of the main branch is connected to the decoder through a skip connection.

[0016] Preferably, in the reverse process, a method based on multi-scale time coding combines linear and nonlinear time scales to capture the noise characteristics of different time steps in the reverse process and is applied to the encoder and decoder of the main branch.

[0017] Preferably, in the inverse process, the loss function of the parameterized neural network includes boundary perception loss EABC, mean square error loss MSE and L1 loss.

[0018] Preferably, in the process of reasoning the diffusion model, a subset is dynamically selected from the complete time series by a DTS algorithm, the steps of which include:

[0019] Initialization: Set the total number of time steps T and the target subset size S;

[0020] Importance scoring model: Use a neural network evaluation model for each time step t to predict the contribution of that step to the final image quality;

[0021] Subset selection: Select the top S time steps as key steps for sampling.

[0022] The present invention also discloses a method for generating a CBCT-to-synthetic sCT image based on a diffusion model, which is used to implement the method for generating a CBCT-to-synthetic sCT image, comprising:

[0023] The forward processing module is used to obtain the original CT image and gradually add Gaussian noise to form a noise image;

[0024] The inverse processing module is used to perform denoising operation on the noisy image and restore the noisy image to the sCT image by adding the CBCT image as the conditional input of the denoising process.

[0025] The present invention discloses the following technical effects:

[0026] The present invention not only significantly improves the efficiency and quality of image conversion, but also has better practicality and broad application prospects.

[0027] The present invention is applicable to the field of medical imaging, especially in radiotherapy planning, and provides more accurate dose calculation and treatment effect evaluation for patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0030] Figure 2 It is a schematic diagram of the test results of the present invention. DETAILED DESCRIPTION

[0031] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0032] like Figure 1-2 As shown, the present invention adopts a forward process and a reverse process in the diffusion model. In the forward process, Gaussian noise is gradually added to the original CT image and finally converted into a noisy image; in the reverse process, the original CT image is gradually restored through the denoising step. In particular, the present invention integrates the CBCT image as a conditional input in the reverse process to guide the restoration process of the current noisy CT image, thereby generating a high-quality sCT image; specifically, it includes the following contents:

[0033] 1. Adaptive High Frequency Optimization Module (AHFO):

[0034] First, the present invention gradually adds noise to the clear CT image x0 to generate a series of increasingly noisy images x1, x2, ..., x T , this process is called the forward noise addition process. The mathematical expression is:

[0035]

[0036] Among them, α t is the parameter controlling the noise level, ∈ t is the noise drawn from a standard normal distribution.

[0037] After obtaining the noisy data, a parameterized neural network is used to perform the reverse denoising process, that is, from x T The original image x0 is gradually restored. The iterative formula for reverse denoising is as follows:

[0038]

[0039] In the formula, ∈ θ (x t , t) is the noise estimated by the neural network, σ t is a time-dependent noise scaling factor that generally decreases with time.

[0040] The parameterized neural network here is designed as a denoising model based on the conditional Unet shape. The main branch is a CT image, and the conditional branch is a CBCT image. The main branch image and the conditional branch image are encoded by two encoders respectively, and then the present invention uses Fourier transform (FFT) and wavelet transform (WT) to optimize the frequency domain features extracted in the encoder. CT images and CBCT images contain rich details and texture information in the high-frequency part. Therefore, accurately processing and optimizing these high-frequency information is crucial to improving image quality.

[0041] First, the input image is passed through the conversion function (can be FFT or WT) is decomposed into multiple sub-bands. These sub-bands each capture a different frequency component of the input image:

[0042]

[0043] Among them, x represents the input image and X represents the subband feature image.

[0044] Next, in order to adjust and optimize the contribution of each subband, the present invention introduces a learned weight matrix A. This weight matrix is ​​multiplied element by element with the subband feature image:

[0045] X′=X⊙A

[0046] Where ⊙ represents the element-by-element product. In this way, the model can adaptively focus on the frequency components that are most critical to improving image quality.

[0047] Finally, use the inverse transform function For example, (inverse FFT or inverse WT) converts the optimized subband feature image back to the spatial domain:

[0048]

[0049] This process not only preserves the high-frequency details of the image, but also enhances the overall quality of the image.

[0050] In order to further optimize this process, the present invention also considers the phase information in the frequency domain and uses a complex number representation method to process the FFT result. f is the complex subband feature map, X f =U+iV, where U and V are the real and imaginary parts respectively. The weight matrix A is expanded into a complex form, so that:

[0051] X′=(U⊙A real -V⊙A imag )+i(U⊙A imag +V⊙A real )

[0052] This complex weight approach can more finely tune the frequency domain features to accommodate different types of image structures and noise patterns.

[0053] 2. Dual-Mode Feature Fusion Module (DMFF):

[0054] The Dual-Mode Feature Fusion Module (DMFF) uses a self-attention mechanism to effectively fuse the feature maps of the CBCT branch and the CT branch. The self-attention mechanism can selectively emphasize important features and suppress unimportant information by calculating the interdependencies between features.

[0055] Specifically, the present invention uses the feature map Fr obtained by FFt ​​and inverse FFT of the CBCT branch in the adaptive high-frequency optimization module and the feature map Hr obtained by CT branch wt and inverse wt. The present invention first generates a query (Q), key (K) and value (V) triple:

[0056] Q=W Q Fr,K=W K Fr,V=W V hr

[0057] Among them, W Q ,W K ,W V is a learnable weight matrix, and then calculate the dot product of Q and K, and obtain the normalized attention weight through the softmax function.

[0058]

[0059] Among them, d k is the dimension of the key vector, and this scaling factor helps the stability of the gradient. In this way, the output of each position is a weighted sum of the values ​​V of all positions, and the weight is determined by the similarity of Q and K. After attention fusion, the model of the present invention is restored through a batch normalization layer and a residual block and finally through a decoder, where the decoder and the main branch encoder are concat through a jump connection.

[0060] 3. Time Embedding Module:

[0061] In order to enable the model to adapt to noise at different time steps during inverse denoising, the present invention introduces a time embedding module based on a multi-scale time coding method that combines linear and nonlinear time scales to more accurately capture and simulate the impact of time changes on noise and image characteristics.

[0062] Specifically, we first define the time encoding function TE(t,i) as follows, which combines the sine and cosine functions of linear and logarithmic time scales to increase the richness and expressiveness of time encoding:

[0063]

[0064] Here, T represents the total number of time steps, t is the current time step, and the log function provides a logarithmic scale of time encoding, which helps the model capture the voice characteristics of rapid changes in the early stage and slow changes in the later stage. At the same time, by processing input signals of different time steps through a multi-layer perceptron, the module can accurately capture the noise characteristics of different time steps in the reverse diffusion process, further improving the model's ability to handle dynamic noise. The present invention adds a time embedding module to the main branch encoder and decoder.

[0065] Loss function:

[0066] In the present invention, three loss functions are used: boundary-aware loss (EABC), mean square error loss (MSE) and L1 loss. These three loss functions are combined to form a composite loss function, which aims to improve the quality of generated images in different aspects, especially in detail fidelity and noise suppression.

[0067] Boundary Aware Loss (EABC):

[0068] The design of boundary-aware loss (EABC) is based on the gradient Laplacian boundary extraction method. The purpose of this loss function is to enhance the model's perception of image edges, so as to more accurately reconstruct image boundaries and details. The Laplacian operator is a second-order derivative operator widely used in the field of image processing, and is particularly suitable for edge detection.

[0069] The Laplace operator is defined as follows:

[0070]

[0071] Operator applied to image I D0 For each pixel point, calculate its second-order derivative to get the boundary image:

[0072]

[0073] In order to convert this boundary map into a binary map, the present invention sets a threshold θ and applies the following binarization operation:

[0074]

[0075] The binarized boundary map B(x, y) can highlight the edge area in the original CT image. In order to form the loss function, the present invention calculates the second-order derivative map of B(x, y) and the original CT image The cross entropy between:

[0076]

[0077] Among them, P(x, y) is the predicted boundary map obtained by transforming the second-order derivative map of the original CT image through the sigmoid function.

[0078] MSE and L1 loss:

[0079] MSE loss is used to quantify the error between the model output and the real noise, which is a standard practice in the training of diffusion models. L1 loss is used to reduce the reconstruction error and enhance the overall visual quality of the image. The definitions of these two loss functions are as follows:

[0080]

[0081]

[0082] in, is the model’s noisy prediction output for the jth sample, is the corresponding real noise, is the image reconstruction output of the model for the jth sample, is the original CT image of the Tth sample.

[0083] The total loss function is the weighted sum of the above three losses:

[0084]

[0085] Among them, α EABC , α MSE and α L1 is the weight coefficient of each loss term, which is adjusted according to the performance of the model during training.

[0086] Accelerate reasoning:

[0087] In the reasoning process of diffusion models, traditional methods usually involve performing complex calculations at each time step, which leads to slow reasoning speed. In order to speed up this process, this paper proposes an innovative scheme based on dynamic time scheduling (DTS) to accelerate reasoning. This scheme aims to significantly improve reasoning efficiency by optimizing time step selection and reducing total computing requirements.

[0088] The DTS algorithm dynamically selects a subset from the complete time series. The steps include:

[0089] Initialization: Set the total number of time steps T and the target subset size S.

[0090] Importance Scoring Model: A neural network evaluation model is used for each time step t to predict the contribution of that step to the final image quality.

[0091] Subset selection: Select the top S time steps as key steps for sampling.

[0092] The core of the DTS strategy is to dynamically select the time step according to the characteristics of the current image. To this end, the present invention adopts a complex decision model based on image features, which uses machine learning technology to predict the importance of each time step. First, for each time step image I t The present invention extracts features including frequency domain features and spatial domain features of the image. The extraction formulas of these features are as follows:

[0093] Features freq (I t )=FFT(I t )·LogGabor(ω)

[0094] Features spatial (I t )=Sobel(I t )☉Canny(I t )

[0095] Among them, FFT stands for fast Fourier transform, LogGabor(ω) is a logarithmic Gabor filter that adjusts the frequency domain response, and ω is a frequency domain variable; Sobel and Canny represent the Sobel operator and the Canny edge detector, respectively, which are used to extract the edge information of the image.

[0096] Importance Scoring Model:

[0097] Based on the above characteristics, a neural network model Nn is constructed importance To calculate the importance score for each time step:

[0098] The model takes as input a time step t and image features and outputs an importance score for that time step.

[0099] Use a dynamic programming algorithm to select an optimal set of time steps that maximizes the overall image quality while minimizing the computational cost:

[0100]

[0101] Among them, T sub is a subset of time steps and λ is a regularization parameter used to balance the trade-off between the number of time steps and image quality.

[0102] After selecting the key time steps, the present invention uses these optimal time steps as subsets to reduce other redundant time steps to speed up reasoning.

[0103] In the CBCT to sCT conversion experiment, a variety of evaluation indicators were used to verify the effect of the model, including PSNR, RMSE, MAE, SSIM, etc.

[0104] Through the above technical solution, the present invention not only significantly improves the efficiency and quality of image conversion, but also has better practicality and broad application prospects. The present invention is applicable to the field of medical imaging, especially in radiotherapy planning, to provide patients with more accurate dose calculation and treatment effect evaluation. At the same time, the present invention can also be used for image conversion in other fields, such as conversion of image modalities such as MR to CT, PET to CT, etc.

[0105] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0107] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for generating CBCT to synthetic sCT images based on a diffusion model, characterized in that: The process includes: Forward process: Get the original CT image and gradually add Gaussian noise to form a noise image; Reverse process: performing a denoising operation on the noisy image, and restoring the noisy image to an sCT image by adding the CBCT image as a conditional input of the denoising process; In the forward process, the noise image is formed based on the original CT image through a forward noise addition process, wherein the forward noise addition process is expressed as: In the formula, ∈ t represents the noise drawn from a standard normal distribution, x t+1 It represents the noise pattern formed after noise is gradually added to the original CT image x0; In the reverse process, the denoising process is expressed as: In the formula, α t represents the parameter controlling the noise level, ∈ θ (x t , t) is the noise estimated by the neural network, σ t is a time-dependent noise scaling factor that usually decreases with time, and z represents the sampled Gaussian noise; In the reverse process, a parameterized neural network is used for denoising, the noise image is input into the main branch of the parameterized neural network, and the CBCT image is input into the conditional branch; The main branch image and the conditional branch image are encoded by an encoder, and the frequency domain features extracted in the encoder are optimized by Fourier transform and wavelet transform, the feature maps of the CBCT branch and the sCT branch are fused by a self-attention mechanism, and the image is restored by a batch normalization layer and a residual block and a decoder to generate an sCT image; In the process of generating the sCT image, the encoder of the main branch is connected to the decoder through a skip connection; In the reverse process, a multi-scale time coding method is used to combine linear and nonlinear time scales to capture the noise characteristics of different time steps in the reverse process and apply them to the encoder and decoder of the main branch. In the reverse process, the loss functions of the parameterized neural network include boundary perception loss EABC, mean square error loss MSE and L1 loss; In the process of inference of the diffusion model, a subset is dynamically selected from the complete time series through the DTS algorithm, and the steps include: Initialization: Set the total number of time steps T and the target subset size S; Importance scoring model: Use a neural network evaluation model for each time step t to predict the contribution to the final image quality; Subset selection: Select the top S time steps as key steps for sampling.

Images