CBCT image reconstruction method and system based on deep learning

Abstract

The invention discloses a CBCT image reconstruction method and system based on deep learning, the system comprises an information input module, a CBCT reconstruction network module and an information output module, the reconstruction method is combined with a freezing-unfreezing mechanism to train a CBCT reconstruction network, a reconstructed CBCT image is generated, and in each training process, the CBCT image is subjected to CBCT image reconstruction. Randomly selecting projection data at a projection angle, and performing projection data filtering through the unfrozen projection domain filtering network; projection data at other projection angles are subjected to projection data filtering through the frozen projection domain filtering network, in view of the fact that the projection domain filtering network is frozen, in the forward transmission stage of training, feature maps of all network layers do not need to be temporarily stored so as to be used for updating parameters in follow-up back propagation, and therefore the training efficiency is improved. And a plurality of continuous cross-sectional images are randomly selected to carry out image domain post-processing, so that the input data volume of an image domain filtering network is reduced, and through the two measures, the demand on a video memory during CBCT reconstruction network training is remarkably reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of CT image reconstruction, and particularly relates to a CBCT image reconstruction method and system based on deep learning. Background Art

[0002] Cone-beam Computerized Tomography (CBCT) technology uses a two-dimensional array to emit a cone beam of X-rays. After the rays pass through the object to be reconstructed, they are received by a flat-panel detector, and the information obtained by the detector is recombined in a computer to obtain a final three-dimensional reconstructed image, which intuitively shows the three-dimensional structure. CBCT technology has the advantages of short scanning time, high ray utilization rate, high spatial resolution, and low scanning cost.

[0003] Due to its strong feature extraction ability, feature mapping, and combination ability, the convolutional neural network algorithm has been concerned and studied by many scholars. Using the convolutional neural network to solve the problems in CT imaging technology in the case of fan-beam scanning and improving the CT imaging effect has become a current research hotspot. The main idea of applying the neural network to CT reconstruction is to integrate the neural network into the reconstruction process. For example, in the dual-domain reconstruction network, the calculation steps such as filtering, back-projection, and post-processing in the analytical reconstruction algorithm are implemented by the neural network; in the deep unfolding neural network, the iterative calculation steps in the iterative reconstruction algorithm are replaced by the neural network.

[0004] Applying deep learning technology to CBCT reconstruction can also improve the imaging quality of CBCT images. Compared with CT reconstruction in the case of fan-beam scanning, CBCT uses cone-beam scanning, and the amount of projection data required for acquisition and the number of tomographic images obtained after reconstruction will both increase significantly. If the network designed for CT reconstruction is directly applied to the CBCT image reconstruction task, it will cause a sharp increase in the video memory required during the network training process, far exceeding the upper limit of the video memory capacity of the current available devices. To solve this thorny problem, scholars have adopted diverse research strategies. Some scholars focus on the research of sparse-angle CBCT reconstruction networks, trying to reduce the video memory burden by reducing the number of projection angles. However, the limitation of this method is that as the input of valid data is reduced, the quality of the reconstructed image will inevitably decline. On the other hand, some other scholars are committed to reducing the network scale in order to achieve more efficient video memory utilization. Unfortunately, this approach often comes with a significant weakening of the network's feature extraction ability. Summary of the Invention

[0005] The purpose of the present invention is to provide a CBCT image reconstruction method and system based on deep learning, to solve the problem of high dependence on a large amount of video memory resources in the training of the CBCT reconstruction network, and to ensure the successful completion of the training of the CBCT network under the conditions that the scale of the reconstruction network remains unchanged, the number of projection angles and the number of tomographic images remain the same.

[0006] To achieve the above object, the solution of the present invention is: a CBCT image reconstruction method based on deep learning, including the following steps:

[0007] S1. Input CBCT projection data and system parameters;

[0008] S2. Train the CBCT reconstruction network in combination with the freeze-thaw mechanism to generate a reconstructed CBCT image. The CBCT reconstruction network includes a projection domain filtering network, a backprojection network, and an image domain filtering network. The specific steps for generating the reconstructed CBCT image are as follows:

[0009] S2.1. Prepare two sets of projection domain filtering networks, freeze one set of projection domain filtering networks, then the other set of projection domain filtering networks is in the thawed state. Filter the projection data at one projection angle through the thawed projection domain filtering network, and filter the projection data at the remaining projection angles through the frozen projection domain filtering network;

[0010] S2.2. Perform domain conversion on all filtered projection data through the backprojection network to generate tomographic images;

[0011] S2.3. Use the image domain filtering network to perform refined post-processing in the image domain on randomly selected and multiple consecutive tomographic images;

[0012] S2.4. Thaw the frozen projection domain filtering network in S2.1, freeze the thawed projection domain filtering network, then perform projection data filtering, and then repeat steps S2.2 - S2.4 to train the CBCT reconstruction network to generate the final CBCT reconstruction image;

[0013] S3. Output the information of the final CBCT reconstruction image.

[0014] Further, in step S2.4, in each training, after the forward propagation of the network, calculate the value of the entire network loss function according to the tomographic images and real images processed in step S2.3, backpropagate the value of the loss function, and update the parameter values in the thawed projection domain filtering network, while the parameter values in the frozen projection domain filtering network remain unchanged.

[0015] Further, in step S2.4, calculate the value of the entire network loss function through the L1 norm loss function, L2 norm loss function, or SSIM loss function.

[0016] Further, in step S2.4, after each training session, the parameters of the thawed projection domain filtering network are copied to the frozen projection domain filtering network.

[0017] Further, in step S2.4, after training is completed, testing is performed. The projection data at all projection angles is filtered through the projection domain filtering network; the filtered projection data is transformed through the backprojection network to generate tomographic images; all tomographic images are post-processed through the image domain filtering network to generate the final CBCT reconstruction result.

[0018] Further, in step S2.1, after preparing two sets of projection domain filtering networks, the parameters of the two sets of projection domain filtering networks are initialized using the same initialization method to ensure that the initialization parameters of the two network models are the same.

[0019] Further, in step S1, the projection data is the projection data collected by the CBCT hardware system or the cone beam projection data obtained by computer simulation. The system parameters are the corresponding parameters used when collecting or simulating the cone beam projection data. The system parameters include the detector size and sampling interval, the number of projection data and projection angles, the size and sampling interval of the reconstruction phantom, the distance from the radiation source to the rotation center, and the distance from the radiation source to the detector.

[0020] Further, in step S2, the CBCT reconstruction network is composed of a combination of a CBCT reconstruction algorithm and a neural network.

[0021] The present invention also provides a CBCT image reconstruction system based on deep learning, including:

[0022] An information input module for inputting CBCT projection data and system parameters;

[0023] A CBCT reconstruction network module including a CBCT reconstruction network composed of a projection domain filtering network, a backprojection network, and an image domain filtering network. The projection domain filtering network is used to filter the projection data. The CBCT reconstruction network is trained using two sets of projection domain filtering network models in combination with a freeze-thaw mechanism; the backprojection network is used to perform domain conversion on the filtered projection data to generate tomographic images; the image domain filtering network is used to randomly select multiple consecutive tomographic images for post-processing in the image domain;

[0024] An information output module for outputting the final CBCT reconstruction image information.

[0025] After adopting the above solution, the gain effect of the present invention is as follows:

[0026] The present invention combines a freeze-thaw mechanism for training a CBCT reconstruction network. During each training process, projection data at a randomly selected projection angle is filtered through an unfrozen projection-domain filtering network; projection data at the remaining projection angles is filtered through a frozen projection-domain filtering network. Given that the projection-domain filtering network is frozen, during the forward pass of training, there is no need to temporarily store the feature maps of each network layer for use in updating parameters during subsequent backpropagation. This measure greatly reduces the demand for video memory during the training of the projection-domain filtering network. Moreover, multiple consecutive tomographic images are randomly selected for post-processing in the image domain, rather than selecting all tomographic images for post-processing in the image domain, significantly reducing the amount of input data for the image-domain filtering network. This measure significantly reduces the demand for video memory during the training of the image-domain filtering network. By adopting the above two measures, it is possible to successfully complete the training of the CBCT network under the condition that the scale of the reconstruction network does not decrease, and the number of projection angles and the number of tomographic images remain the same. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flowchart of the calculation method for the CBCT image reconstruction method of the present invention;

[0028] Figure 2 is a structural diagram of the CBCT reconstruction network of the present invention;

[0029] Figure 3 is a structural diagram of the projection-domain filtering network and the image-domain filtering network of the present invention;

[0030] Figure 4 is a structural diagram of a specific implementation case of the present invention combining the freeze-thaw mechanism. DETAILED DESCRIPTION OF THE INVENTION

[0031] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The present invention provides a CBCT image reconstruction method based on deep learning, including the following steps:

[0033] S1. Input CBCT projection data and system parameters. Among them, the CBCT projection data can be the projection data collected by the CBCT hardware system or the cone-beam projection data obtained by computer simulation, and the system parameters are the corresponding parameters used when collecting or simulating the cone-beam projection data. The system parameters include the detector size and sampling interval, the number of projection data and projection angles, the size and sampling interval of the reconstruction phantom, the distance from the radiation source to the rotation center, and the distance from the radiation source to the detector.

[0034] In this application, a cone-beam X-ray is used to irradiate a phantom, and a flat-panel detector is used for imaging to collect two-dimensional projection data on the flat-panel detector at different irradiation angles. In this embodiment, only the tomographic slice of the three-dimensional object is reconstructed. The number of projections is 720, the length of the projection data is 768 pixels, the height of the projection data is 768 pixels, and the spatial resolution of the projection image is 1×1 mm 2 ; the height of the reconstructed image is 512 voxels, the length of the tomographic image is 512 voxels, the width of the tomographic image is 512 voxels, and the spatial resolution of the reconstructed image is 0.7×0.7×0.7 mm 3 ; the distance from the ray source to the detector is 1000 mm, and the distance from the ray source to the rotation center is 500 mm.

[0035] S2. Train the CBCT reconstruction network in combination with the freeze-thaw mechanism to generate the reconstructed CBCT image. As Figure 1 shown, the CBCT reconstruction network includes a projection-domain filtering network, a back-projection network, and an image-domain filtering network. The specific steps for generating the reconstructed CBCT image are as follows:

[0036] S2.1. Prepare two sets of projection-domain filtering networks, initialize the model parameters of the two sets of projection-domain filtering networks using the same initialization method to ensure that the initialization parameters of the two network models are consistent. Then freeze one set of projection-domain filtering networks, so that the other set of projection-domain filtering networks is in the thawed state. Filter the projection data at one projection angle through the thawed projection-domain filtering network, and filter the projection data at the remaining projection angles through the frozen projection-domain filtering network;

[0037] S2.2. Perform domain conversion on all filtered projection data through the back-projection network to generate a tomographic image;

[0038] S2.3. Use the image-domain filtering network to perform refined post-processing in the image domain on randomly selected, multiple consecutive tomographic images; select a finite number of consecutive tomographic images for post-processing in the image domain instead of selecting all tomographic images for post-processing in the image domain, which significantly reduces the input data volume of the image-domain filtering network. This measure significantly reduces the demand for video memory during the training of the image-domain filtering network, and can effectively exploit the correlation between tomographic images with the help of deep learning technology, further improving the quality of the CBCT reconstructed image;

[0039] S2.4. Thaw the frozen projection domain filtering network in S2.1, freeze the thawed projection domain filtering network, then perform projection data filtering, and then repeat steps S2.2 - S2.4 to train the CBCT reconstruction network. After the training process is completed, the final CBCT reconstruction image can be generated. During each training process, after the network propagates forward, the constructed loss function can be used to calculate the value of the entire network loss function based on the tomographic images and real images processed in step S2.3. The value of the loss function is propagated backward to update the parameter values in the thawed projection domain filtering network, while the parameter values in the frozen projection domain filtering network remain unchanged to guide the training process of the neural network model, and then the next training is carried out. In addition, the specific number of training times can be selected through a large number of experiments and experimental observations.

[0040] S3. Output the information of the final CBCT reconstruction image.

[0041] Furthermore, in step S2.4, after each training is completed, the parameters of the thawed projection domain filtering network need to be copied to the frozen projection domain filtering network to ensure the consistency of network parameters during the training process, and then the next training is carried out. After the training is completed, the trained model can be applied to the images to be tested, that is, the projection data at all projection angles are filtered through the projection domain filtering network; the filtered projection data are transformed through the backprojection network to generate tomographic images; all tomographic images are post - processed through the image domain filtering network to generate the final CBCT reconstruction result. During the testing process, since there is no need to save the feature maps of each network layer, the testing process does not require a large amount of video memory.

[0042] Furthermore, in step S2.4, the required loss function is constructed using point - to - point pixel difference (L1 norm loss function, L2 norm loss function) or structural similarity of images (SSIM loss function).

[0043] The following is further illustrated by a specific embodiment:

[0044] In step S2, the CBCT reconstruction network is composed of the combination of the CBCT reconstruction algorithm and the neural network. Specifically, in the dual - domain reconstruction network, a U - Net network with an "encoding - decoding" structure is adopted, specifically as Figure 3 shown. The depth of the encoding - decoding network is four, that is, there are four downsampling and upsampling operations in total. Each level of the encoder is composed of two consecutive convolutional layers. Specifically, the downsampling layer uses a max - pooling operation, and the convolutional kernel size is 3×3 and the stride is 2×2. The output end of the network uses a convolutional layer with a convolutional kernel size of 1×1 and a stride of 1 to fuse the features. The overall structure of the CBCT reconstruction network is as Figure 4As shown, where Net A and Net B are projection domain filtering networks, Net C is a back-projection network without adjustable parameters, and Net D is an image domain filtering network. The forward propagation of network Net C is implemented using a voxel-based back-projection algorithm, and the reverse propagation is implemented using a ray-based forward projection algorithm. Then step S2 specifically includes the following steps:

[0045] S2.1. Before training the CBCT reconstruction network, initialize networks Net A and Net B to ensure that the initial values of the two network model parameters are the same. Freeze network Net B, and network Net A is in an unfrozen state. Randomly select the projection data at one projection angle and perform projection domain preprocessing through network Net A, and perform projection domain preprocessing on the projection data at the remaining angles through network Net B. Given that network Net B has been frozen, during the forward pass of training, there is no need to temporarily store the feature maps of each network layer for use in updating parameters during subsequent backpropagation. This measure greatly reduces the demand for video memory during the training of the projection domain filtering network.

[0046] S2.2. The filtered projection data is converted through network Net C to generate tomographic images;

[0047] S2.3. Randomly select 10 consecutive tomographic images and perform post-processing on the tomographic images through network Net D. During training, by only selecting a limited number of tomographic images (such as 10) and inputting them into network Net D, not only can the training goal of network Net D be achieved, but also the required video memory occupancy can be effectively reduced;

[0048] S3.3. Unfreeze network Net B, freeze network Net A, and then perform projection data filtering. Repeat steps S2.2 - S2.3 to complete one training. Before the end of training, copy the parameters of network Net A to network Net B to ensure the consistency of network parameters during training. Then unfreeze network Net A and freeze network Net B again to perform training again, repeat steps S2.2 - S2.3, and so on, until all training is completed to generate the final CBCT reconstruction image. In this embodiment, the norm loss function and the multi-scale structural similarity loss function (MS-SSIM) can be used as the loss function of the entire network model.

[0049] Similarly, during the testing process, the projection data at all projection angles is preprocessed in the projection domain through network Net A; the filtered projection data is converted through network Net C to generate tomographic images; all tomographic images are post-processed in the image domain through network Net D to generate the final CBCT reconstruction result.

[0050] The present invention also provides a CBCT image reconstruction system based on deep learning, including:

[0051] An information input module for inputting CBCT projection data and system parameters. The CBCT projection data can be the projection data collected by a CBCT hardware system or the cone-beam projection data obtained by computer simulation. The system parameters are the corresponding parameters used when collecting or simulating the cone-beam projection data. The system parameters include the detector size and sampling interval, the number of projection data and projection angles, the size and sampling interval of the reconstruction phantom, the distance from the radiation source to the rotation center, and the distance from the radiation source to the detector;

[0052] A CBCT reconstruction network module, including a CBCT reconstruction network composed of a projection domain filtering network, a back-projection network, and an image domain filtering network. The projection domain filtering network is used to filter the projection data, and two sets of projection domain filtering network models are used for the training of the CBCT reconstruction network in combination with the freeze-thaw mechanism; the back-projection network is used to perform domain conversion on the filtered projection data to generate tomographic images; the image domain filtering network is used to randomly select multiple consecutive tomographic images for post-processing in the image domain;

[0053] An information output module for outputting the final CBCT reconstruction image information.

[0054] The above is only the preferred embodiment of the present invention, and it does not limit the design of this case. All equivalent changes made according to the key design of this case fall within the protection scope of this case.

Claims

1. A CBCT image reconstruction method based on deep learning, characterized in that: The following steps are involved: S1. Input CBCT projection data and system parameters; S2. The CBCT reconstruction network is trained in combination with the freeze-thaw mechanism to generate a reconstructed CBCT image. The CBCT reconstruction network includes a projection domain filtering network, a back-projection network, and an image domain filtering network. The specific steps of generating the reconstructed CBCT image are as follows: S2.1, prepare two sets of projection domain filter networks, freeze one set of projection domain filter networks, and keep the other set of projection domain filter networks in an unfrozen state, filter the projection data at one projection angle through the unfrozen projection domain filter network, and filter the projection data at the remaining projection angles through the frozen projection domain filter network; S2.2, performing domain conversion on all filtered projection data through a back-projection network to generate a tomographic image; S2.3, using an image domain filtering network to perform refined image domain post-processing on a plurality of randomly selected continuous tomographic images; S2.4, unfreeze the frozen projection domain filter network in S2.1, freeze the unfrozen projection domain filter network, perform projection data filtering, and then repeat steps S2.2-S2.4 to perform CBCT reconstruction network training to generate a final CBCT reconstructed image; S3. Output the final CBCT reconstructed image information.

2. The CBCT image reconstruction method based on deep learning according to claim 1, characterized in that: In step S2.4, in each training, after the network is forward propagated, the value of the loss function of the entire network is calculated based on the tomographic image and the real image processed in step S2.3, the value of the loss function is back-propagated, and the parameter values ​​in the unfrozen projection domain filter network are updated, while the parameter values ​​in the frozen projection domain filter network remain unchanged.

3. The CBCT image reconstruction method based on deep learning according to claim 2, characterized in that: In step S2.4, the value of the entire network loss function is calculated by using the L1 norm loss function, the L2 norm loss function or the SSIM loss function.

4. The CBCT image reconstruction method based on deep learning according to claim 1, characterized in that: In step S2.4, after each training, the parameters of the unfrozen projection domain filter network are copied to the frozen projection domain filter network.

5. The CBCT image reconstruction method based on deep learning according to claim 1, characterized in that: In step S2.4, after the training is completed, a test is performed, and the projection data at all projection angles are filtered through the projection domain filtering network; the filtered projection data are domain transformed through the back-projection network to generate tomographic images; all tomographic images are post-processed in the image domain through the image domain filtering network to generate the final CBCT reconstruction results.

6. The CBCT image reconstruction method based on deep learning according to claim 1, characterized in that: In step S2.1, after preparing two sets of projection domain filter networks, the parameters of the two sets of projection domain filter networks are initialized using the same initialization method to ensure that the initialization parameters of the two sets of network models are consistent.

7. The CBCT image reconstruction method based on deep learning according to claim 1, characterized in that: In step S1, the projection data is projection data collected by a CBCT hardware system, or cone beam projection data obtained by computer simulation, and the system parameters are corresponding parameters used when collecting or simulating cone beam projection data, and the system parameters include detector size and sampling interval, number of projection data and projection angle, size and sampling interval of the reconstructed model, distance from the ray source to the rotation center, and distance from the ray source to the detector.

8. The CBCT image reconstruction method based on deep learning according to claim 1, characterized in that: In step S2, the CBCT reconstruction network is formed by combining a CBCT reconstruction algorithm with a neural network.

9. A CBCT image reconstruction system based on deep learning, characterized in that: include: An information input module, used for inputting CBCT projection data and system parameters; The CBCT reconstruction network module includes a CBCT reconstruction network composed of a projection domain filtering network, a back-projection network and an image domain filtering network. The projection domain filtering network is used to filter the projection data, and two sets of projection domain filtering network models are used to train the CBCT reconstruction network in combination with a freeze-thaw mechanism; the back-projection network is used to perform domain conversion on the filtered projection data to generate a tomographic image; and the image domain filtering network is used to randomly select multiple continuous tomographic images for image domain post-processing; The information output module is used to output the final CBCT reconstructed image information.

Images