A multimodal image synthesis method and system based on deep learning

Abstract

The present invention relates to a multimodal image synthesis method based on deep learning, which comprises: S1, data collection: collecting CBCT images and corresponding planned CT images of cervical cancer patients; S2, image preprocessing: aligning the planned CT image to the CBCT image and removing non-anatomical structures; neighborhood interpolation and cropping the image to the same size; adjusting the window width and converting the CT value into a linear attenuation coefficient; S3, building a deep learning model framework; S4, hierarchical training: adopting a hierarchical training strategy to train the deep learning model, first training a residual U-Net with a coarse resolution; then taking the previous stage as the starting point, continuing to train the deep learning model with images with higher resolution, so as to help the deep learning model learn more complex image features and capture finer image details; S5, model testing and verification: using the hierarchically trained deep learning model to directly generate a pseudo CT image from the original CBCT image.

Description

Technical Field

[0001] The present invention relates to the technical field of adaptive radiotherapy and image processing, and in particular to a multimodal image synthesis method and system based on deep learning. Background Art

[0002] During radiotherapy, it is necessary to ensure that sufficient dose is deposited in the target area to achieve the therapeutic effect, while at the same time, the irradiation of organs at risk needs to be reduced as much as possible. CBCT (Cone Beam Computed Tomography, CBCT) images are currently commonly used for patient positioning verification. By comparing CBCT with the planning CT, the patient's position is adjusted by bone structure registration to ensure that the radiation dose is deposited in the target area. However, due to problems such as bladder filling, cervical movement, tumor retraction, and gastrointestinal motility, it is challenging to achieve highly conformal dose delivery by relying solely on bone structure registration. As the treatment progresses, it may be necessary to adjust the current fractionated treatment plan in a timely manner according to the current anatomical position to achieve accurate irradiation of the target area while protecting organs at risk, that is, to achieve adaptive radiotherapy. However, compared with the planning CT images, the current CBCT images have lower image contrast, more severe artifacts, and poor soft tissue resolution. In addition, the CT values ​​of tissues in CBCT images often fluctuate, significantly deviating from those obtained in CT images. The above problems limit the clinical application of CBCT in adaptive radiotherapy.

[0003] Several methods have been proposed in the prior art to remove artifacts and improve the image quality of CBCT. These methods can be divided into hardware-based methods and image post-processing methods. Hardware-based methods attempt to use anti-scattering grids to reduce the impact of scattering events when acquiring CBCT images to improve image contrast and clarity. However, anti-scattering grids may also introduce some problems at the same time, such as weakening signal intensity, generating image artifacts, and increasing device cost and complexity. Image post-processing methods mainly include deformation based on planning CT, estimation of scattering kernels, Monte Carlo simulation, histogram matching, and methods based on deep learning. Deformation based on planning CT is one of the commonly used methods. Its basic idea is to deform and register the planning CT and CBCT, and regard the registered planning CT as the processed image corresponding to the CBCT. However, there is currently no optimal deformation registration algorithm, which means that there is still uncertainty after registration, which affects the correctness of the CT value in the CBCT image. Especially in structures such as the intestine, the deformation registration effect is poor. The estimation and Monte Carlo simulation of the scattering kernel are time-consuming and resource-consuming. Histogram matching requires a special linear matching method and also relies on deformation registration. In recent years, deep learning-based methods have developed rapidly. Cycle-GAN-based models are often used in pseudo-CT synthesis tasks and have achieved certain results. Although Cycle-GAN allows the use of unpaired image data to train the model, it may generate structures in the pseudo-CT images that do not actually exist in the original CBCT images. There are also problems such as blurred structural edges and loss of image details. In addition, the Cycle-GAN model takes a long time to train, is difficult to train, and requires large computing resources. Its application is limited for units with insufficient computing resources. In addition, the existing models all use data sets acquired by a single linear accelerator for training, verification, and testing. Since the image acquisition protocols between different accelerators may be different, and each linear accelerator may have its own unique noise distribution, the existing models may have limited generalization ability and poor robustness, especially for linear accelerators produced by different manufacturers.

[0004] In this application, the applicant attempts to use paired image data from the pelvic region of cervical cancer patients treated with linear accelerators of different brands to train the model to complete the task of synthesizing pseudo CT from the original CBCT image. The solution proposed in this application intends to adopt an encoder-decoder structure, with a CBCT image as the input and a generated pseudo CT image as the output. A hybrid loss function and a hierarchical training strategy are used to train the model step by step from low resolution to high resolution, so that the deep learning model has the ability to represent the basic content of the image and the ability to capture the fine details of the image. Summary of the invention

[0005] The present invention aims to provide a multimodal image synthesis method and system based on deep learning. The technical problems to be solved include at least how to synthesize high-quality pseudo CT images with accurate CT values, higher soft tissue resolution and better contrast, how to constrain the training process to well preserve the edges of soft tissue in the original CBCT image in the generated pseudo CT image, and how to use mixed data sets to train a more robust model.

[0006] In order to achieve the above object, the present invention provides a multimodal image synthesis method based on deep learning, comprising the following steps:

[0007] S1. Data collection: Collect CBCT images and corresponding planning CT images of patients with cervical cancer;

[0008] S2, image preprocessing: align the planned CT image to the CBCT image and remove non-anatomical structures; perform neighborhood interpolation and crop the image to the same size; adjust the window width and convert the CT value into a linear attenuation coefficient;

[0009] S3. Construct a deep learning model framework: The deep learning model is based on an encoder-decoder architecture, uses U-Net as a basic model, and integrates residual learning into the basic model; the deep learning model consists of 1 regular convolution block, 4 residual convolution blocks, 4 upsampling residual convolution blocks and 1 output layer;

[0010] S4. Hierarchical training: The deep learning model is trained using a hierarchical training strategy. First, a coarse-resolution residual U-Net is trained to train the deep learning model to learn basic image content representation and common image features. Then, the deep learning model is trained with higher-resolution images as a starting point to help the deep learning model learn more complex image features and capture finer image details.

[0011] S5. Model testing and verification: Use the layered trained deep learning model to generate pseudo CT images directly from the original CBCT images.

[0012] Preferably, the non-anatomical structure comprises a treatment bed, a cotton swab containing contrast agent and / or an intrauterine contraceptive device.

[0013] Preferably, the specific steps of removing non-anatomical structures are: first, using the TotalSegmentator tool to generate a body mask, manually checking the body mask, and using corrosion and dilation operations to ensure that the body mask is as smooth as possible; next, multiplying the planned CT image and the CBCT image by the body mask to remove the treatment bed; then, using a threshold-based method to identify the pixels where the IUD and the cotton swab containing contrast agent are located in the planned CT image, and replacing the identified pixels with the pixel values ​​at the same position in the CBCT image. The specific steps of image preprocessing are: first, performing neighborhood interpolation to ensure that all images are of the same size; then, cropping all images to reduce computing resources for the background; finally, adjusting the window width to [-1000,1500] and converting it into a linear attenuation coefficient.

[0014] Preferably, the conventional convolution block includes two convolution layers, the convolution kernel size is 3×3, the step size is 1, and the padding is 1.

[0015] Preferably, the residual convolution block has an additional branch, which is constructed as an identity mapping to simplify training and alleviate the gradient vanishing problem.

[0016] Preferably, the up-sampled residual block is composed of a deconvolution layer and a residual block, and the deconvolution layer with a convolution kernel size of 2×2 and a step size of 2 gradually restores the size of the output feature map; then, the up-sampled feature map is spliced ​​with the feature map from the encoder through a jump connection to accelerate the training process and retain the structural details of the input image; in order to accelerate and stabilize the training process, batch normalization is used to standardize the input after each convolution operation except the last layer; finally, a convolution layer with a convolution kernel size of 1×1 and a step size of 1 is used to output a pseudo CT image according to the extracted features.

[0017] Preferably, in step S4, the deep learning model is gradually trained in three stages.

[0018] Preferably, in step S4, in order to generate a multi-resolution image dataset for hierarchical training, the original CBCT image and the planning CT image are downsampled, and the number of image pixels is halved; in order to match the input size of the deep learning model, the downsampled image is upsampled to restore to the initial image size; finally, three image datasets with different resolutions are obtained.

[0019] Preferably, in step S4, different loss functions are used to form a complete loss function to train the deep learning model.

[0020] Preferably, the different loss functions include weighted mean absolute error, multi-scale structural similarity loss and gradient difference loss; the calculation formula of weighted mean absolute error is as follows:

[0021]

[0022] Where N represents the total number of pixels in the high CT value area; M represents the total number of pixels in the area other than the high CT value area; y 高值,i is the value of the i-th pixel in the high CT value area of ​​the real image; is the value of the i-th pixel in the high CT value area of ​​the output image; y 其余,i Represents the value of the i-th pixel in the remaining area of ​​the real image except the high CT value area; represents the value of the i-th pixel in the remaining areas of the output image except the high CT value area; the demarcation threshold for distinguishing the high CT value area from other areas is 500HU; ω is a constant, set to 5;

[0023] The multi-scale structural similarity loss calculation formula is as follows:

[0024]

[0025] Where K represents the total number of calculation scales; SSIM K represents the structural similarity index at the i-th scale; α i Represents the weight of the i-th scale, which is used to weighted average the structural similarity index of different scales.

[0026] Gradient difference loss is used to maintain the strong gradient of the original image and enhance the edge sharpness of the tissue during the optimization process. The definition of gradient difference loss is as follows:

[0027]

[0028] Where Y represents the real image, represents the synthesized pseudo CT image; x represents along the x direction, and y represents along the y direction; represents the gradient operator; |||| 2 represents the L2 norm.

[0029] The complete loss function is as follows:

[0030] Total loss = λ1·weighted mean absolute error + λ2·multi-scale structural similarity loss

[0031] +λ3 Gradient difference loss

[0032] Among them, λ1, λ2 and λ3 are set to 1, 3, and 2 respectively.

[0033] The present invention also provides a multimodal image synthesis system based on deep learning, comprising a data collection module, an image preprocessing module, a framework construction module of a deep learning model, a hierarchical training module and a pseudo CT image generation module;

[0034] The data collection module is used to collect CBCT images and corresponding planned CT images of patients with cervical cancer;

[0035] The image preprocessing module is used to register the planned CT image to the CBCT image and remove non-anatomical structures; perform neighborhood interpolation and crop the image to the same size; adjust the window width and convert the CT value into a linear attenuation coefficient;

[0036] The framework construction module of the deep learning model is used to construct a deep learning model framework. The deep learning model is based on an encoder-decoder architecture, uses U-Net as a basic model, and integrates residual learning into the basic model; the deep learning model consists of 1 regular convolution block, 4 residual convolution blocks, 4 upsampling residual convolution blocks and 1 output layer;

[0037] The hierarchical training module is used to train the deep learning model using a hierarchical training strategy. First, the residual U-Net with coarse resolution is trained to train the deep learning model to learn basic image content representation and common image features. Then, the deep learning model is trained with higher resolution images as the starting point, so as to help the deep learning model learn more complex image features and capture finer image details.

[0038] The pseudo CT image generation module is used to generate pseudo CT images directly from original CBCT images using a trained deep learning model.

[0039] Preferably, the multimodal image synthesis system based on deep learning further includes a performance evaluation module for evaluating the performance of the trained deep learning model using three indicators: Mean Absolute Error (MAE), Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR). MAE compares the mean absolute error of pixels between the synthesized pseudo-CT and the real image based on the entire image; MAE is defined as follows:

[0040]

[0041] Where N represents the total pixels of the image; Y i Represents the value of the i-th pixel of the real image; represents the value of the i-th pixel of the synthesized pseudo-CT;

[0042] SSIM evaluates the image similarity between the synthesized pseudo-CT and the real image from the perspective of brightness, contrast, and structural information. The definition of SSIM is as follows:

[0043]

[0044] where μ x represents the average value of x; μ y represents the average value of y; σ x represents the standard deviation of x; σ y represents the standard deviation of y; σ xy is the covariance of x and y; C1 and C2 are constants, usually C1 = 0.01, C2 = 0.03;

[0045] The definition of PSNR is as follows:

[0046]

[0047] Among them I gt (x, y) represents the gray value of the real image at (x, y); I syn (x, y) represents the gray value of the synthetic pseudo-CT image at (x, y); X and Y represent the height and width of the real image respectively; MAX I is the maximum gray value in the real image and the synthesized pseudo CT image.

[0048] Beneficial Effects

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The present invention successfully developed a deep learning-based framework for generating pseudo-CT images from CBCT images of cervical cancer. Compared with previous models, the model has been shown to generate pseudo-CT images with higher image quality and more accurate CT values. In particular, under the constraints of the combined loss function and the hierarchical training strategy, the edges of the tissue are well preserved. To the best of the applicant's knowledge, this is the first model built using a multi-brand linear accelerator dataset, and the results are promising. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the specific implementation methods of the present application, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0052] Figure 1 This is a schematic diagram of the inclusion and exclusion criteria in data collection.

[0053] Figure 2 This is a complete diagram of the image preprocessing process.

[0054] Figure 3 It is a schematic diagram of the deep learning model framework shown in the present invention.

[0055] Figure 4 Schematic diagram of a conventional convolutional block in the deep learning model described in the present invention.

[0056] Figure 5 Schematic diagram of the residual convolution block in the deep learning model described in the present invention.

[0057] Figure 6 It is a schematic diagram of an upsampling residual convolution block in the deep learning model described in the present invention.

[0058] Figure 7 It is a schematic diagram of the hierarchical training strategy.

[0059] Figure 8a is a schematic diagram of MAE between different images.

[0060] Figure 8b It is a schematic diagram of PSNR between different images.

[0061] FIG8c is a schematic diagram of SSIM between different images. DETAILED DESCRIPTION

[0062] The present invention is described in more detail hereinafter to facilitate understanding of the present invention.

[0063] The purpose of this application is to develop a deep learning framework based on multi-vendor mixed dataset training to improve the image quality of CBCT images, reduce image artifacts, and obtain more accurate CT values ​​for application in adaptive radiotherapy.

[0064] This application adopts an encoder-decoder architecture with residual learning and skip-layer connections, and adopts a hierarchical training strategy, which enables the model to capture basic image features at the coarse resolution stage and learn fine structural details at the high-resolution stage.

[0065] The multimodal image synthesis method based on deep learning according to the present invention is described in detail below. The method comprises the following steps:

[0066] S1. Data collection:

[0067] In the study of this application, CBCT images and corresponding planning CT images of 230 patients with cervical cancer were collected. All planning CT images were acquired using Brilliance Big Bore CT (Philips Healthcare, the Netherlands). Among them, 50 patients were treated with the Elekta Axesse linear accelerator, and CBCT images were acquired using the XVI onboard CBCT imaging system. The remaining 180 patients were treated with the Varian Trilogy linear accelerator, and CBCT images were acquired using the OBI imaging system. Finally, we double-checked the collected data set and ultimately included 228 patients (49 patients from the Elekta Axesse linear accelerator and 179 patients from the Varian Trilogy linear accelerator). The inclusion criteria were as follows: Figure 1 shown.

[0068] S2. Image preprocessing

[0069] Given that the planning CT and CBCT images were scanned at different time points, the planning CT images were first registered to the CBCT images using MIM software (v.6.3.4, MIMSoftware Inc., Cleveland, Ohio, USA). Then, in order to mitigate the interference of non-anatomical structures such as the treatment bed, contrast swabs, and intrauterine contraceptive devices, several steps were taken to remove them from the images. First, a body mask was generated using the tool TotalSegmentator developed by Wasserthal et al. After running TotalSegmentator, the body mask was manually checked and erosion and dilation operations were used to ensure that the body mask was as smooth as possible. Next, the planning CT and CBCT images were multiplied by the body mask to remove the treatment bed. When scanning the planning CT, a contrast swab was inserted from the vagina to help the oncologist locate the cervix more clearly. However, during radiotherapy, the swab was removed, resulting in a mismatch between the planning CT and CBCT in some areas. Similarly, due to complex organ motion, the position of the intrauterine contraceptive device may be different in the planning CT and CBCT, which will also cause partial mismatches between the images and affect the model training process. Therefore, a threshold-based method is used to identify these pixels in the planning CT image as much as possible and replace them with the pixel values ​​at the same position in the CBCT image. In order to maintain the same image size, a neighborhood interpolation algorithm is used to interpolate all images to the same size. Subsequently, all images are cropped to 400×400 pixels to reduce unnecessary computing resources. Finally, the window width is adjusted to [-1000,1500] and converted into a linear attenuation coefficient using formula (1):

[0070]

[0071] Where μ iis the linear attenuation coefficient of the i-th pixel, HU i is the CT value of the i-th pixel, μ 水 is the linear attenuation coefficient of water. The complete image preprocessing process, such as Figure 2 shown.

[0072] S3. Build a deep learning model framework:

[0073] The deep learning model can synthesize pseudo CT images from the input CBCT images. The model established in this application is based on the encoder-decoder architecture. The overall structure of the model is as follows Figure 3 As shown. This application uses U-Net as the basic model and integrates residual learning into the model. The model consists of 1 regular convolution block, 4 residual convolution blocks, 4 upsampling residual convolution blocks and 1 output layer.

[0074] The conventional convolutional block consists of two convolutional layers with a kernel size of 3×3, a stride of 1, and a padding of 1. The residual convolutional block differs from the conventional convolutional block in that it has an additional branch that is constructed as an identity mapping to simplify training and alleviate the gradient vanishing problem. The additional branch is Figure 5 The first branch (the top one) is shown in the residual convolution block in Figure 1. The upsampling residual block consists of a deconvolution layer and a residual block. The deconvolution layer with a kernel size of 2×2 and a stride of 2 gradually restores the size of the output feature map. The upsampled feature map is then concatenated with the feature map from the encoder through a skip connection to speed up the training process and preserve the structural details of the input image. To speed up and stabilize the training process, batch normalization is used after each convolution operation to standardize the input except for the last layer. Finally, a convolution layer with a kernel size of 1×1 and a stride of 1 is used to output a pseudo CT image based on the extracted features.

[0075] S4. Layered training:

[0076] The present application adopts a hierarchical training strategy during the training process. First, the applicant trains a coarse-resolution residual U-Net, aiming to train the model to learn basic image content representation and common image features. Then, starting from the previous stage, the model is trained with higher-resolution images to help the model learn more complex image features and capture finer image details. The present application gradually trains the model in three stages.

[0077] In order to generate a multi-resolution image dataset for hierarchical training, this application downsamples the original CBCT images and planned CT images. After the downsampling operation, the number of image pixels is halved. In order to match the input size of the model, the downsampled image is upsampled to restore to the initial image size. The following is the process of generating medium-resolution and low-resolution images used in this application: (1) Downsample the original CBCT image and CT image (image size is 400×400) to obtain CBCT images and CT images with an image size of 200×200, and then upsample them to restore to a size of 400×400. The resulting image is the medium-resolution image used in this application; (2) Downsample the CBCT image and CT image with an image size of 200×200 to obtain CBCT images and CT images with an image size of 100×100, and then upsample them to restore to a size of 400×400. The resulting image is the low-resolution image used in this application. Finally, we obtained three image datasets with different resolutions. Flowchart and dataset examples, such as Figure 7 shown.

[0078] S5. Model testing and verification: Use the layered trained deep learning model to generate pseudo CT images directly from the original CBCT images.

[0079] Preferably, in step S4, different loss functions are used to form a complete loss function to train the deep learning model.

[0080] In order to synthesize pseudo CT images with accurate CT values ​​and clear tissue boundaries, the loss function of the model includes three parts: weighted mean absolute error, multi-scale structural similarity loss, and gradient difference loss. The mean absolute error is used as a loss function to constrain the overall absolute difference between the model output and the true value. However, the model trained using the traditional mean absolute error may focus more on reducing the difference between areas with high CT values ​​(such as bones) while ignoring areas with low CT values ​​(such as soft tissue). This application improves the mean absolute error to a weighted mean absolute error by separately calculating the difference between the high CT value areas and the difference between the remaining areas. The calculation formula of the weighted mean absolute error is as follows:

[0081]

[0082] Where N and M represent the total number of pixels in the high CT value area and the rest area respectively; y 高值,i and Represents the value of the i-th pixel in the area with higher CT value in the real image and the output image respectively; y 其余,i and Represent the value of the i-th pixel in the real image and the rest of the output image, respectively. This application selects 500HU as the threshold to distinguish high CT value areas from the rest of the areas. The specific threshold selection method is: select according to the CT value histogram distribution of the planned CT image and the CT value of the bone area; ω is a constant, set to 5.

[0083] Multi-scale structural similarity is an extension of the Structural Similarity Index (SSIM). By continuously downsampling the image to be evaluated, the similarity between images can be measured from multiple scales. The total evaluation scale of the multi-scale structural similarity loss used in this application is 5. The initial image to be evaluated is continuously downsampled four times, and the structural similarity index of the current scale is calculated after each downsampling. Finally, the structural similarity indices at five scales are weighted averaged to obtain the multi-scale structural similarity. The calculation formula for the multi-scale structural similarity loss is as follows:

[0084]

[0085] Where K represents the total number of calculation scales, which is set to 5 in this application; SSIM i represents the structural similarity index at the i-th scale; α i Represents the weight of the i-th scale, which is used to weighted average the SSIM values ​​of different scales. The weights used in this application are set to 0.0448, 0.2856, 0.3001, 0.2363, and 0.1333.

[0086] The calculation formula of SSIM is:

[0087]

[0088] where μ x and μ y and σ x and σ y Represents the mean and standard deviation of x and y respectively; σ xy is the covariance of x and y; C1 and C2 are constants, usually with the default values ​​of C1=0.01 and C2=0.03.

[0089] The gradient difference loss is used to maintain the strong gradient of the original image and enhance the edge sharpness of the tissue during the optimization process. The gradient difference loss is defined as follows:

[0090]

[0091] Where Y represents the real image, represents the synthesized pseudo CT image; x represents along the x direction, and y represents along the y direction; represents the gradient operator; ||||2 represents the L2 norm.

[0092] The complete loss function is as follows:

[0093] Total loss = λ1·weighted mean absolute error + λ2·multi-scale structural similarity loss

[0094] +λ3 Gradient difference loss

[0095] Among them, λ1, λ2 and λ3 are set to 1, 3, and 2 respectively.

[0096] The present invention also provides a multimodal image synthesis system based on deep learning, comprising a data collection module, an image preprocessing module, a framework construction module of a deep learning model, a hierarchical training module and a pseudo CT image generation module;

[0097] The data collection module is used to collect CBCT images and corresponding planned CT images of patients with cervical cancer;

[0098] The image preprocessing module is used to register the planned CT image to the CBCT image and remove non-anatomical structures; perform neighborhood interpolation and crop the image to the same size; adjust the window width and convert the CT value into a linear attenuation coefficient;

[0099] The framework construction module of the deep learning model is used to construct a deep learning model framework. The deep learning model is based on an encoder-decoder architecture, uses U-Net as a basic model, and integrates residual learning into the basic model; the deep learning model consists of 1 regular convolution block, 4 residual convolution blocks, 4 upsampling residual convolution blocks and 1 output layer;

[0100] The hierarchical training module is used to train the deep learning model using a hierarchical training strategy. First, the residual U-Net with coarse resolution is trained to train the deep learning model to learn basic image content representation and common image features. Then, the deep learning model is trained with higher resolution images as the starting point, so as to help the deep learning model learn more complex image features and capture finer image details.

[0101] The pseudo CT image generation module is used to generate pseudo CT images directly from original CBCT images using a trained deep learning model.

[0102] The multimodal image synthesis system based on deep learning also includes a performance evaluation module for evaluating the performance of the trained deep learning model using three indicators: Mean Absolute Error (MAE), Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR). MAE compares the mean absolute error of pixels between the synthesized pseudo-CT and the real image based on the entire image; MAE is defined as follows:

[0103]

[0104] Where N represents the total pixels of the image; Y i Represents the value of the i-th pixel of the real image; represents the value of the i-th pixel of the synthesized pseudo-CT;

[0105] SSIM evaluates the image similarity between the synthesized pseudo-CT and the real image from the perspective of brightness, contrast, and structural information. The definition of SSIM is as follows:

[0106]

[0107] where μ x and μ y and σ x and σ y Represents the mean and standard deviation of x and y respectively; σ xy is the covariance of x and y; C1 and C2 are constants, usually C1 = 0.01, C2 = 0.03;

[0108] The definition of PSNR is as follows:

[0109]

[0110] Among them I gt (x, y) represents the gray value of the real image at (x, y); I syn (x, y) represents the gray value of the synthesized pseudo-CT image at (x, y); X and Y represent the height and width of the image respectively; MAX I is the maximum gray value in the real image and the synthesized pseudo CT image.

[0111] During the development process, this application divides the paired "Varian" dataset and "Elekta" dataset into three subsets: training set, validation set and test set. The dataset division is random, with a ratio of 6:2:2. This application uses the Adam optimizer with a learning rate of 0.001 and a batch size of 8. Network training, validation and testing are all implemented using the PyTorch library in Python.

[0112] The effectiveness of the network is verified from three perspectives. First, the performance of the model is compared with the previously used deep convolutional neural network (RCNN) architecture with residual blocks. RCNN also uses an encoder-decoder architecture. Then, ablation experiments are performed to verify the role of the combined loss function and the hierarchical training strategy: (1) the network is optimized with different loss functions (only using weighted mean absolute error, gradient difference loss + weighted mean absolute error, and multi-scale structural similarity loss + weighted mean absolute error) and their performance is compared; (2) compared with the model without the hierarchical training strategy. Finally, the performance is compared with a specific model trained on a single dataset.

[0113] Qualitative results

[0114] This application trains the model in three stages, and the total number of cycles is set to 100, including 50 cycles in the first stage, 30 cycles in the second stage, and 20 cycles in the third stage. After optimizing the model weights, we tested its performance on all test cases. During the test, the planned CT, the original CBCT, the pseudo-CT generated by the model of this application, and the pseudo-CT generated by the RCNN model were obtained. By comparing the pseudo-CT generated by the model of this application with the original CBCT, it can be found that the artifacts in the original CBCT are significantly improved and the overall image quality is significantly improved. Using the planned CT as a reference and comparing the absolute difference map, it can be found that compared with the pseudo-CT image generated by the RCNN model and the original CBCT image, the difference in CT values ​​between the pseudo-CT generated by the model of this application and the planned CT is the smallest in other areas such as the bladder area and muscle.

[0115] The CT value histogram and line profile of the pseudo CT generated by the model of the present application show that, compared with RCNN, the model of the present application can better improve the CT value with higher accuracy, and the CT value distribution shown in the histogram is closer to the planned CT.

[0116] Quantitative results

[0117] This application quantitatively evaluates our model on the entire test set. Table 1 and Figure 8aFigure 8c summarizes the MAE, PSNR, and SSIM between the planned CT and different images. The MAE between the pseudo-CT generated by the model of the present application and the planned CT decreased from the initial 50.02HU to 10.93HU, an improvement of 64.21% (p<0.001). In terms of PSNR, the generated pseudo-CT image improved from 27.79dB to 32.74dB (p<0.001), an improvement of 26.95%. SSIM also improved by 19.27%, from 0.77 to 0.90 (p<0.001). Compared with the RCNN model, the model of the present application performed better in all evaluation indicators. Compared with the original CBCT images, the CT images generated by the model of the present application have achieved improvements in all evaluation indicators.

[0118] Table 1 MAE, PSNR and SSIM between planned CT and different images

[0119]

[0120] Ablation Experiment Results

[0121] To demonstrate the effectiveness of the combined loss function used in this application, the applicant trained three other models with different loss functions. All models are built on the same architecture and trained with the same dataset under a hierarchical training strategy. Table 2 summarizes the evaluation metrics of all models based on the same test dataset. The model using only weighted mean absolute error as the loss function achieved the largest MAE of 13.77HU among all models. By introducing gradient difference loss or multi-scale structural similarity loss in the loss function, the models tend to produce smaller MAEs. However, no significant improvement was observed in SSIM, and even a slight decrease was observed. The purpose of introducing gradient difference loss and multi-scale structural similarity loss is to sharpen tissue edges and enhance image clarity. Nevertheless, this performance may be affected by noise and artifacts in the original CBCT images. This loss function may mistakenly identify noise and artifacts as anatomical structures and try to retain them during the optimization process, resulting in no significant improvement in PSNR and SSIM. Overall, the combination of the three produced the best performance.

[0122] Table 2 Quantitative analysis of pseudo CT images generated by models trained with different loss functions

[0123]

[0124]

[0125] The applicant then compared the performance of models trained using different strategies (i.e., single-stage training and layered training). Table 3 lists the comparison of different training strategies on all evaluation indicators. Compared with the model trained in a single stage, the model built using the layered training strategy reduced the MAE from 13.57HU to 10.93HU (p<0.001). The improvement in the PSNR value was also significant (p<0.001).

[0126] Table 3 Quantitative analysis of pseudo CT images generated by models trained with different training strategies

[0127]

[0128] Comparison between different datasets

[0129] To study the impact of different datasets, this application trained two additional specific models using the Varian dataset or the Elekta dataset. Table 4 summarizes the evaluation results between the models trained using different datasets. The specific model trained using the Varian dataset performed better on the Varian test dataset than the model trained using the mixed dataset, with a MAE of 8.77HU, a PSNR of 38.20dB, and a SSIM of 0.93. However, the model trained using the mixed dataset achieved lower MAE, higher SSIM and PSNR on the same Elekta test dataset relative to the specific model. This may be due to the smaller dataset when training the Elekta specific model. The specific model of the Elekta dataset only used data from 29 patients to train the model, which may be difficult to generalize well to other unseen data.

[0130] Table 4 Comparison of pseudo CT images generated by models trained with different datasets

[0131]

[0132]

[0133] discuss

[0134] This application develops a deep learning-based framework to reduce artifacts and improve the quality of CBCT images for further clinical applications. Experimental results show that the model can effectively generate pseudo-CT images with fewer artifacts and more accurate CT values ​​compared to the original CBCT images. As mentioned in the background technology, Cycle-GAN is often used in image synthesis tasks because it can be trained using unpaired image data. However, it is challenging to strike a balance between the generator and the discriminator, so the model is difficult to train. In addition, it requires more computing resources and takes a long time to train. Therefore, the model of this application is based on U-Net instead of Cycle-GAN. U-Net has also been widely used in medical image processing tasks. It is based on an encoder-decoder architecture, and jump connections can reduce information loss during encoding and promote information propagation in the network. This application integrates residual learning into the model to enhance the expressiveness of the model and alleviate the gradient vanishing problem.

[0135] This application uses a hybrid loss function consisting of weighted mean absolute error, gradient difference loss, and multi-scale structural similarity loss to constrain the optimization process. Unlike the traditional mean absolute error, the weighted mean absolute error is designed to avoid the model focusing on learning the mapping between high pixel value areas. This application also attempts to alleviate the image blur problem by utilizing multi-scale structural similarity loss and sharpen the edges of soft tissue by introducing gradient difference loss. According to the results of the ablation experiment, it can be concluded that training the model using a hybrid loss function is effective, reducing MAE from 13.77HU to 10.93HU, and also slightly improving PSNR.

[0136] The present application adopts a hierarchical training strategy to train the model. The present application does not train the model only in a single stage, but trains the model step by step from low resolution to high resolution. In the first stage, low-resolution images are used to train the model to learn basic image representations. Then higher-resolution images are continuously used to fine-tune the previous model to help the model capture more details. In order to demonstrate the effectiveness of this training strategy, the present application conducted a comparative experiment. The present application trained another model with the same architecture, but only used high-resolution images for training. As shown in Table 3, the model trained using the hierarchical training strategy outperforms the model trained in a single stage in all evaluation indicators.

[0137] The deep learning-based framework can improve the quality of CBCT images. Studies have shown that the use of RCNN can improve the accuracy of CT values ​​in CBCT of prostate cancer patients, and the best improvement in MAE was achieved, with a reduction of 85.20%. Given that the RCNN model in the prior art is also based on an encoder-decoder architecture, the applicant attempted to compare the performance of the model of this application with the RCNN model in the prior art. Since the code for RCNN in the prior art is private, the applicant reproduced the RCNN model in the prior art according to the description in the paper and trained it using the data set of this application. The comparison results are shown in Table 1, Figure 8a , Figure 8b , as shown in Figure 8c. The model of the present application has achieved excellent performance both qualitatively and quantitatively. In addition, clear tissue edges can be observed in the images generated by the model of the present application. The model of the prior art is proposed for prostate cancer patients. Considering that the anatomical structure of cervical cancer patients is more complex, it may not be applicable to the dataset of the present application. The prior art also proposed 3D Cycle-GAN to generate pseudo CT images of cervical cancer patients, with a reported MAE of 51.62HU, an SSIM of 0.86, and a PSNR increased from 27.15dB to 30.70dB. As far as the research of the present application is concerned, the PSNR increased from 27.79dB to 33.91dB. In terms of MAE, it decreased from 50.02HU to 10.93HU. The present application also achieved higher values ​​in SSIM, with an average value of 0.90.

[0138] The quality of the data set will have a great impact on the performance of the model. In the study of this application, the applicant used a mixed data set to train the model to improve the versatility and robustness of the model. The mixed data set consists of images collected by linear accelerators of different brands. In order to study the effects of different data sets, this application trained specific models for the Varian data set and the Elekta data set respectively. According to the evaluation indicators, when using the same test data set, the model of this application performs worse than the specific model of the Varian data set. This may be due to the different distribution of artifacts and noise in the Varian data set and the Elekta data set. When using a mixed data set, the model aims to maintain a balance between different data sets, so the performance is worse than that of the specific model. Overall, the differences in MAE and SSIM are not particularly obvious, which to some extent proves the robustness and universality of the model of this application. But for the specific model trained only with the Elekta data set, the performance becomes worse. This may be due to the small size of the training data set, which only contains 29 patients. Better performance may be obtained when training a specific model with a larger Elekta data set.

[0139] Based on the existing results, the model developed in this application has the potential to generate pseudo CT images from CBCT images for application in adaptive radiotherapy.

[0140] This application successfully developed a deep learning-based framework for generating pseudo-CT images from CBCT images of cervical cancer. The model has been shown to generate pseudo-CT images with higher image quality and more accurate CT values. In particular, the edges of the tissue are well preserved under the constraints of the combined loss function. To the best of the applicant's knowledge, this is the first model built using a multi-brand linear accelerator dataset, and the results are promising.

[0141] The above describes the preferred embodiments of the present invention, but it is not intended to limit the present invention. Those skilled in the art may make improvements and changes to the embodiments disclosed herein without departing from the scope and spirit of the present invention.

Claims

1. A multimodal image synthesis method based on deep learning, characterized in that: The multimodal image synthesis method based on deep learning comprises the following steps: S1. Data collection: Collect CBCT images and corresponding planning CT images of patients with cervical cancer; S2, image preprocessing: align the planned CT image to the CBCT image and remove non-anatomical structures; perform neighborhood interpolation and crop the image to the same size; adjust the window width and convert the CT value into a linear attenuation coefficient; S3. Construct a deep learning model framework: The deep learning model is based on an encoder-decoder architecture, uses U-Net as a basic model, and integrates residual learning into the basic model; the deep learning model consists of 1 regular convolution block, 4 residual convolution blocks, 4 upsampling residual convolution blocks and 1 output layer; S4, hierarchical training: a hierarchical training strategy is adopted to train the deep learning model. First, a coarse-resolution residual U-Net is trained to train the deep learning model to learn basic image content representation and common image features. Then, the deep learning model is trained with higher-resolution images as the starting point to help the deep learning model learn more complex image features and capture finer image details. The total training level is set to 3. S5. Model testing and verification: Use the layered trained deep learning model to directly generate pseudo CT images from the original CBCT images; In step S4, different loss functions are used to form a complete loss function to train the deep learning model; The different loss functions include weighted mean absolute error, multi-scale structural similarity loss and gradient difference loss; the calculation formula of weighted mean absolute error is as follows: ； Where N represents the total number of pixels in the high CT value area; M represents the total number of pixels in the area other than the high CT value area; y 高值,i is the value of the i-th pixel in the high CT value area of ​​the real image; 高值,i is the value of the i-th pixel in the high CT value area of ​​the output image; y 其余,i Represents the value of the i-th pixel in the remaining area of ​​the real image except the high CT value area; 其余,i represents the value of the i-th pixel in the remaining areas of the output image except the high CT value area; the demarcation threshold for distinguishing the high CT value area from the remaining areas is 500HU; ω is a constant, set to 5; The multi-scale structural similarity loss calculation formula is as follows: ； Where K represents the total number of calculation scales; SSIM i represents the structural similarity index at the i-th scale; Represents the weight of the i-th scale, which is used to weighted average the structural similarity index of different scales; Gradient difference loss is used to maintain the strong gradient of the original image and enhance the edge sharpness of the tissue during the optimization process. The definition of gradient difference loss is as follows: ； Where Y represents the real image, Ỹ represents the synthesized pseudo CT image; x represents along the x direction, and y represents along the y direction; Represents the gradient operator represents the L2 norm; The complete loss function is as follows: in, Set to 1, 3 and 2 respectively.

2. The multimodal image synthesis method based on deep learning according to claim 1, characterized in that: The specific steps of removing non-anatomical structures are as follows: first, using the TotalSegmentator tool to generate a body mask, manually checking the body mask, and using corrosion and dilation operations to ensure that the body mask is smooth; next, multiplying the planned CT image and the CBCT image by the body mask to remove the treatment bed; then, using a threshold-based method to identify the pixels where the IUD and the cotton swab with contrast agent are located in the planned CT image, and replacing the identified pixels with the pixel values ​​at the same position in the CBCT image; the specific steps of image preprocessing are as follows: first, performing neighborhood interpolation to ensure that all images are of the same size; then, cropping all images to reduce computing resources for the background; finally, adjusting the window width to [-1000,1500] and converting it into a linear attenuation coefficient.

3. The multimodal image synthesis method based on deep learning according to claim 1, characterized in that: The conventional convolution block includes two convolution layers, the convolution kernel size is 3×3, the stride is 1, and the padding is 1.

4. The multimodal image synthesis method based on deep learning according to claim 1, characterized in that: The residual convolution block has an additional branch that is constructed as an identity mapping to simplify training and alleviate the gradient vanishing problem.

5. The multimodal image synthesis method based on deep learning according to claim 1, characterized in that: The up-sampled residual convolution block is composed of a deconvolution layer and a residual block. The deconvolution layer with a convolution kernel size of 2×2 and a stride of 2 gradually restores the size of the output feature map. Then, the up-sampled feature map is concatenated with the feature map from the encoder through a jump connection to accelerate the training process and retain the structural details of the input image. In order to accelerate and stabilize the training process, batch normalization is used to standardize the input after each convolution operation except the last layer. Finally, a convolution layer with a convolution kernel size of 1×1 and a stride of 1 is used to output a pseudo CT image based on the extracted features.

6. The multimodal image synthesis method based on deep learning according to claim 1, characterized in that: In step S4, the deep learning model is gradually trained in three stages.

7. The multimodal image synthesis method based on deep learning according to claim 1, characterized in that: In step S4, in order to generate a multi-resolution image dataset for hierarchical training, the original CBCT images and planning CT images are downsampled, and the number of image pixels is halved; in order to match the input size of the deep learning model, the downsampled images are upsampled to restore to the initial image size; finally, three image datasets with different resolutions are obtained.

Images