Cone beam CT full respiratory cycle image reconstruction method based on four-dimensional neural network

Abstract

The invention discloses a cone beam CT full respiratory cycle image reconstruction method based on a four-dimensional neural network. The method comprises the following steps: step 1, acquiring cone beam projection data and a respiratory signal in a respiratory state; 2, a respiratory cycle is divided into N phases, multi-phase cone beam projection data Qi are obtained, i = 1, 2,..., N, the Qi is reconstructed through a filtered back projection algorithm, and a multi-phase under-sampling CT reconstructed image # imgabs0 # is obtained; 3, # imgabs1 # is input into a four-dimensional neural network, and a three-dimensional deformation field Di is obtained; 4, performing weighted summation after deformation on the # imgabs2 # by using the Di to obtain a reference phase reconstruction image Xref after motion correction; and step 5, performing inversion on the Di to obtain a three-dimensional reverse deformation field # imgabs3 #, for any breathing phase, performing image deformation on the reference phase reconstruction image Xref by using the corresponding # imgabs4 #, and performing reconstruction to obtain a motion correction image Xi. According to the method, the breathing movement correction can be quickly carried out on the cone beam CT image.

Description

Technical Field

[0001] The present invention relates to the field of cone beam CT, and in particular to a cone beam CT full respiratory cycle image reconstruction method based on a four-dimensional neural network. Background Art

[0002] In recent years, cone beam computed tomography (Cone beam computed tomography) technology has developed rapidly. Compared with traditional spiral CT systems, cone beam CT systems are often smaller and more mobile. Therefore, they can be used in operating rooms in orthopedics, dentistry, radiotherapy, respiratory intervention, etc. in hospitals, and have a wide range of application scenarios. However, the frame rotation speed of the cone beam CT system is slow. During the scanning process, the patient's respiratory movement will cause respiratory motion artifacts in the reconstructed image, affecting the accuracy of lesion localization.

[0003] By synchronously recording the patient's respiratory signal during cone-beam CT imaging, the cone-beam CT image reconstruction process can distinguish data from different respiratory phases, and reconstruct independent cone-beam CT images phase by phase, further obtaining a four-dimensional cone-beam CT image that can describe the respiratory movement process. However, this scheme splits the original complete cone-beam CT projection data into multiple subsets. In each respiratory phase, the projection data does not meet the integrity requirements of CT image reconstruction. Therefore, there are often serious undersampling artifacts on the four-dimensional cone-beam CT images, affecting the image quality. Existing research work mainly enhances the quality of four-dimensional cone-beam CT images through a motion estimation-motion compensation framework or a CT iterative reconstruction algorithm based on compressed sensing theory. However, these two methods are computationally complex and take a long time to reconstruct, and cannot meet the needs of quickly obtaining high-quality reconstructed images during surgery.

[0004] The technology of using deep neural network models to enhance the quality of CT images is receiving widespread attention. In the field of 4D cone-beam CT image reconstruction, researchers have proposed a variety of targeted deep neural network models. However, due to the limitations of computing resources and the number of training samples, most studies focus on using 2D deep neural networks to enhance images, and thus fail to deeply explore the special structural prior information in 4D cone-beam CT images. In fact, motion artifacts in 4D cone-beam CT images show a special spatiotemporal distribution pattern. Its stripe-like structure is mainly distributed along the cross-section of the cone-beam CT volume data, and presents a special rotational motion pattern during the respiratory cycle, which is different from the respiratory motion pattern of normal tissue structures. Therefore, the spatiotemporal features in 4D cone-beam CT images are of great significance for decoupling artifacts from images. In addition, the respiratory motion of the human body is mainly driven by the diaphragm, and the respiratory motion of various organs and tissues is clearly reflected in the coronal and sagittal planes of cone-beam CT images. Therefore, complete cone-beam CT volume data is very important for the restoration of respiratory motion, and existing 2D neural network methods cannot effectively utilize this information. In addition, existing methods often lack explicit modeling of the physical process of respiratory motion and lack interpretability. In summary, the respiratory motion correction method for intraoperative four-dimensional cone-beam CT is still immature.

[0005] Therefore, technicians in this field are committed to developing a cone-beam CT full respiratory cycle image reconstruction method based on a four-dimensional neural network. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is:

[0007] 1. The traditional model-driven 4D cone-beam CT image enhancement algorithm has limited performance and slow calculation speed; the data-driven 4D cone-beam CT image enhancement algorithm based on deep neural network has limited performance and poor interpretability;

[0008] 2. Existing deep neural network-based cone-beam CT image enhancement algorithms use two-dimensional or three-dimensional deep neural networks and fail to deeply explore the special structural prior information in four-dimensional cone-beam CT images.

[0009] To achieve the above objectives,

[0010] In a first aspect, the present invention provides a method for reconstructing cone-beam CT full respiratory cycle images based on a four-dimensional neural network, comprising steps 1-5:

[0011] Step 1: Perform cone-beam CT scanning on the patient in the respiratory state and collect cone-beam projection data Q, which is a series of sub-data q collected at different angles during the rotation of the gantry. j , j = 1, 2, ..., M, synchronously recording the patient's respiratory signal at each sub-data collection moment;

[0012] Step 2: Divide a respiratory cycle into N phases, and determine each sub-data q in the cone beam projection data according to the respiratory signal j Corresponding respiratory phase i, i = 1, 2, ..., N, merge the sub-data of the same phase to obtain multi-phase cone beam projection data Q i ,i=1,2,…,N,Q i Contains multiple projection sub-data at different angles, and multi-phase cone beam projection data Q i The multi-phase undersampled CT reconstructed images are reconstructed using the filtered back projection algorithm independently for each phase

[0013] Step 3: Reconstruct N undersampled CT images Combined into a four-dimensional image, input into a four-dimensional neural network, take a certain phase as the reference phase, and obtain N three-dimensional deformation fields D i ,i=1,2,…,N;

[0014] Step 4: The three-dimensional deformation field D obtained by the neural network output i Undersampled CT images of multiple phases with corresponding phases After deformation, weighted summation is performed to obtain the reference phase reconstructed image X after motion correction ref ;

[0015] Step 5: Three-dimensional deformation field D i Inverse, get N three-dimensional inverse deformation fields i=1,2,…,N, for any respiratory phase, use the corresponding inverse deformation field to reconstruct the image X of the reference phase ref Perform image deformation and reconstruct the motion correction image X corresponding to any respiratory phase in the whole respiratory cycle i ,i=1,2,…,N。

[0016] Furthermore, the specific method for obtaining cone beam projection data and respiratory signals under respiratory state is: using cone beam CT equipment to collect cone beam projection data of patients under free breathing or ventilator-assisted breathing, using respiratory gating equipment to synchronously record the patient's respiratory signal, or calculating the patient's respiratory signal from projection data.

[0017] Further, a respiratory cycle is divided into N phases, specifically according to the amplitude of the respiratory signal.

[0018] Furthermore, the four-dimensional neural network can be specifically set as a four-dimensional separable convolutional neural network.

[0019] Furthermore, the macro structure of the four-dimensional separable convolutional neural network can be specifically set to UNet.

[0020] Furthermore, the four-dimensional separable convolutional neural network can be specifically configured to consist of a series of convolutional layers, each of which contains two consecutive four-dimensional separable convolutional modules and a nonlinear module.

[0021] Furthermore, the nonlinear module can be specifically set to a Leaky ReLu module.

[0022] Furthermore, the four-dimensional separable convolution module can be specifically configured to be composed of multiple parallel, orthogonal low-dimensional convolution modules.

[0023] Furthermore, the four-dimensional separable convolution module can be specifically configured to consist of a spatial convolution in the x-direction, a spatial convolution in the y-direction, a spatial convolution in the z-direction, and a temporal convolution in the t-direction, wherein the temporal convolution in the t-direction uses circular padding to maintain the size of the feature map unchanged, and the design of the circular padding is selected according to the periodicity of the respiratory motion.

[0024] Furthermore, the four-dimensional separable convolutional neural network can be specifically set to be trained by regression training.

[0025] Furthermore, the training data set can be specifically set to be acquired through computer-simulated four-dimensional cone-beam CT scanning, including dynamic projection data and corresponding respiratory signals, and reconstructing multi-phase under-sampled CT reconstructed images by simulating clinical scanning scenes. The training data set includes multiple training samples, and each training sample includes a set of multi-phase under-sampled CT reconstructed images. and a corresponding end-inspiratory reference image X, the training sample does not need to contain the true value of the respiratory deformation field.

[0026] During training, the image is minimized after motion compensation. The L1 norm of the deviation from the end-inspiratory reference image X is used as the self-supervised optimization target, where The ADAM optimizer is used to optimize the parameters of the neural network.

[0027] Furthermore, since the training process requires the use of four-dimensional data, which is relatively scarce in clinical practice, and a small-scale training data set is difficult to effectively train a four-dimensional neural network, the present invention makes full use of the characteristics of separable convolution and trains the model progressively in two stages.

[0028] Specifically, during the first stage of pre-training, the z-direction spatial convolution operator of each convolution module in the 4D separable convolutional neural network is frozen, and this branch does not participate in the calculation.

[0029] At this point, the four-dimensional separable convolutional neural network degenerates into a three-dimensional neural network that processes two-dimensional xy plane spatial information and t direction time information. Accordingly, the four-dimensional training samples are also cut in cross-section, rearranged into multiple three-dimensional data, and used to train the network. This operation expands the number of training samples. The training mode of the first stage is designed based on a full analysis of the inherent characteristics of undersampled four-dimensional cone-beam CT images. The main training goal of the neural network in this stage is to achieve the decoupling of the characteristic stripe artifacts and original anatomical structures in the xy plane, while the t direction time dimension information helps the network to pay attention to the complementarity and correlation between the phase images, so as to better identify the image. The first stage of training can be carried out with a very high data throughput, thereby accelerating the convergence of model parameters.

[0030] After completing the pre-training in the first stage, the z-direction spatial convolution in the 4D separable convolution module is reactivated in the second stage.

[0031] At this time, the 4D separable convolutional neural network can simultaneously process image information from four dimensions. Accordingly, the second stage of training is completed using 4D data and amplified by random cropping. The training goal of this stage is to enable the network to better learn the inherent movement patterns of respiratory movement. Considering that respiratory movement is mainly driven by the diaphragm, the 4D cone-beam CT image will have a larger movement amplitude in the z direction, and the second stage of training can enable the neural network to quickly focus on the information in the z direction.

[0032] The model training strategy adopted by the present invention fully considers the characteristics of respiratory movement, and combines it with the characteristics of the four-dimensional separable convolutional neural network included in the present invention, so as to complete the effective training of the four-dimensional neural network using only limited four-dimensional training samples.

[0033] The input of the four-dimensional separable convolutional neural network is a set of multi-phase undersampled cone-beam CT images, which are affected by undersampling artifacts and have poor image quality, making it impossible to estimate the respiratory motion field through traditional image registration algorithms. The trained four-dimensional separable convolutional neural network can distinguish the real structural information and image artifacts in undersampled cone-beam CT images in a data-driven manner, thereby achieving reliable estimation of the respiratory motion deformation field.

[0034] In a second aspect, an embodiment of the present application provides a computer device, including:

[0035] Memory;

[0036] one or more processors coupled to the memory;

[0037] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and one or more applications are configured to execute the cone-beam CT full respiratory cycle image reconstruction method based on a four-dimensional neural network as provided in the first aspect above.

[0038] Technical Effects

[0039] 1. The present invention proposes a method for reconstructing the image of the whole respiratory cycle of cone beam CT based on a four-dimensional neural network, which fully combines the advantages of the data-driven method and the model-driven method, trains a deep neural network with big data, and uses the deep neural network to directly estimate the respiratory motion deformation field from the highly undersampled cone beam CT image, and further reconstructs a high-quality four-dimensional cone beam CT image under the motion compensation framework. By training a deep neural network on a large-scale data set, the neural network model can automatically capture the structure and motion information in the highly undersampled cone beam CT image, and rely on group prior information to generate a high-quality respiratory motion deformation field. Compared with the traditional motion estimation algorithm, the reasoning speed of the deep neural network is fast, only one forward calculation is required, and no repeated iteration is required, so the calculation time is shorter. The respiratory deformation field will be further used for motion compensation reconstruction, that is, the motion field correction is introduced in the CT filter back projection reconstruction process, so as to integrate all the projection data information and achieve high-quality image reconstruction. Compared with the deep neural network model that directly performs image enhancement, this method explicitly models the respiratory motion process and has better interpretability. By combining the advantages of model-driven and data-driven algorithms, the cone-beam CT full respiratory cycle image reconstruction method proposed in the present invention has both high reconstruction accuracy and high computational efficiency. The invention can alleviate the impact of respiratory motion during the imaging process of a cone-beam CT system with a slower rotation speed, suppress respiratory motion artifacts, and improve imaging accuracy. At the same time, the method can also achieve dynamic imaging, restore the movement of the scanned object during the complete respiratory cycle, and thus provide image references for dynamic surgical navigation. The high computational efficiency of the method ensures that it is suitable for intraoperative imaging scenarios, which is also an important application scenario of cone-beam CT.

[0040] 2. The present invention proposes a four-dimensional neural network based on a separable convolution operator, which fully exploits the image features of four-dimensional cone-beam CT images in spatial and temporal dimensions to achieve higher performance motion estimation. The four-dimensional convolutional neural network can directly process four-dimensional cone-beam CT images, thereby estimating the respiratory deformation field from the complete image data during the inference process. The separable four-dimensional convolution operator decomposes the four-dimensional isotropic convolution into low-dimensional convolution operators along different dimensions, which can greatly reduce the computational complexity and realize the processing of four-dimensional images under limited computing resources. In addition, the separable convolution operator can be fully trained in a staged manner, such as freezing the Z-direction convolution operator in the first stage, and only using a single layer of data to train the remaining network parameters in exchange for greater data throughput and data richness, and accelerating convergence; in the second stage, the frozen parameters are reactivated, and then all parameters are fine-tuned with four-dimensional data to complete the training of the four-dimensional model, and finally the effective training of the four-dimensional neural network is realized with limited four-dimensional training data. The four-dimensional neural network based on the separable convolution operator can directly process the complete four-dimensional cone-beam CT image in a forward process, thereby better estimating the four-dimensional respiratory deformation field and achieving more accurate respiratory motion compensated reconstruction.

[0041] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of a cone-beam CT full respiratory cycle image reconstruction method based on a four-dimensional neural network according to a preferred embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the convolutional layer structure of a preferred embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a breathing signal of a preferred embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of projection data distribution of a preferred embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of an undersampled cone-beam CT reconstructed image according to a preferred embodiment of the present invention;

[0047] Figure 6 It is a comparison diagram before and after motion correction of a preferred embodiment of the present invention;

[0048] Figure 7 is a schematic diagram of a breathing signal of a preferred embodiment of the present invention;

[0049] Figure 8is a schematic diagram of projection data distribution of a preferred embodiment of the present invention;

[0050] Fig. 9 is a schematic diagram of an undersampled cone-beam CT reconstructed image according to a preferred embodiment of the present invention;

[0051] Fig.10 It is a comparison diagram before and after motion correction of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0053] In the drawings, components with the same structure are denoted by the same numerical numerals, and components with similar structure or function are denoted by similar numerical numerals.

[0054] Embodiment 1

[0055] The invention provides a cone beam CT full respiratory cycle image reconstruction method based on a four-dimensional neural network.

[0056] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0057] Step 1: construct a four-dimensional neural network for respiratory motion correction. The neural network can process four-dimensional images, and the output of the model is a three-dimensional deformation field between a set of multi-phase cone-beam CT images.

[0058] Step 2: construct a training data set and train and optimize the parameters of the four-dimensional neural network.

[0059] Step 3: Use cone beam CT equipment to collect cone beam projection data P of patients with free breathing or ventilator-assisted breathing, use respiratory gating equipment to synchronously record the patient's respiratory signal, or estimate the patient's respiratory signal from the projection data. According to the amplitude of the respiratory signal, a respiratory cycle is divided into N phases, and the corresponding respiratory phase is determined for each projection angle.

[0060] Step 4: Use the filtered back projection algorithm to reconstruct the multi-phase undersampled CT reconstructed image Each respiratory phase corresponds to an independent reconstruction process, and only the projection data corresponding to the respiratory phase determined in step three is used to reconstruct the image.

[0061] Step 5: Combine N undersampled CT reconstructed images into a four-dimensional image, input it into a four-dimensional neural network, and use a certain phase (such as the end of inspiration) as the reference phase to obtain N three-dimensional deformation fields D i,i=1,2,…,N。

[0062] Step 6: The three-dimensional deformation field D obtained by the neural network output i Undersampled CT images of multiple phases with corresponding phases After deformation, weighted summation is performed to obtain the reference phase reconstructed image X after motion correction ref .

[0063] Step 7: Invert the deformation field to obtain N three-dimensional inverse deformation fields For any respiratory phase, use the corresponding inverse deformation field to reconstruct the image X from the reference phase ref By performing image deformation, the motion correction image X corresponding to any respiratory phase in the whole respiratory cycle can be reconstructed. i , i=1,2,…,N, thereby realizing cone-beam CT image reconstruction of the entire respiratory cycle.

[0064] In step one, if Figure 1 As shown, the neural network included in the present invention is a four-dimensional separable convolutional neural network, and its macro structure can be selected from but not limited to UNet. The input of the network is a four-dimensional image, and the output is a set of four-dimensional deformation fields. The network is composed of a series of convolutional layers cascaded, each convolutional layer contains two consecutive four-dimensional separable convolution modules and a nonlinear module, and the nonlinear module can be selected from but not limited to the Leaky ReLu module. Each four-dimensional separable convolution module is composed of multiple parallel, orthogonal low-dimensional convolution modules. Figure 2 An optional combination method is given, where each four-dimensional separable convolution consists of a spatial convolution in the x direction, a spatial convolution in the y direction, a spatial convolution in the z direction, and a temporal convolution in the t direction. Among them, the temporal dimension convolution in the t direction uses circular padding to maintain the size of the feature map unchanged. The design of the circular padding is selected based on the periodicity of respiratory motion. The four-dimensional separable convolution enables the neural network to have a four-dimensional receptive field and can directly process four-dimensional cone-beam CT images. In addition, the separable convolution kernel has fewer parameters and lower computational complexity than the conventional four-dimensional convolution kernel, which requires less computing resources to run the model.

[0065] In step 2, the four-dimensional separable convolutional neural network included in the present invention uses regression training to optimize parameters, and a training data set needs to be constructed according to a specific application scenario. The training data set is obtained through computer-simulated four-dimensional cone-beam CT scanning, including dynamic projection data and corresponding respiratory signals, and simulates clinical scanning scenes to reconstruct multi-phase undersampled CT reconstruction images. The training data set includes multiple training samples, each of which includes a set of multi-phase undersampled CT reconstruction images. And a corresponding end-inspiratory reference image X, the training sample does not need to contain the true value of the respiratory deformation field. During the training process, the image after motion compensation is minimized. The L1 norm of the deviation from the end-inspiratory reference image X is used as the self-supervised optimization target, where The image deformation operator is used, and the parameters of the neural network are optimized using the ADAM optimizer. Since the training process requires the use of four-dimensional data, which is relatively scarce in clinical practice, it is difficult to effectively train a four-dimensional neural network with a small-scale training data set. Therefore, the present invention makes full use of the characteristics of separable convolution and trains the model progressively in two stages.

[0066] Specifically, during the pre-training process of the first stage, the z-direction spatial convolution operator of each convolution module in the four-dimensional separable convolutional neural network is frozen, and the branch does not participate in the calculation. At this time, the four-dimensional neural network included in the present invention degenerates into a three-dimensional neural network that processes two-dimensional xy plane spatial information and t-direction time information. Correspondingly, the four-dimensional training samples are also cut in cross-section, rearranged into multiple three-dimensional data, and used to train the network. This operation expands the number of training samples. The training mode of the first stage is designed on the basis of fully analyzing the inherent characteristics of undersampled four-dimensional cone-beam CT images. The main training goal of the neural network in this stage is to achieve the decoupling of the characteristic stripe-like artifacts and the original anatomical structure in the xy plane, and the t-direction time dimension information helps the network to pay attention to the complementarity and correlation between the phase images, so as to better identify the image. The training of the first stage can be carried out with a very high data throughput, thereby accelerating the convergence of the model parameters.

[0067] After completing the pre-training of the first stage, in the second stage, the z-direction spatial convolution in the four-dimensional separable convolution module is reactivated, and the four-dimensional neural network included in the present invention can synchronously process image information from four dimensions. Accordingly, the training of the second stage is completed using four-dimensional data, and the data is amplified by random cutting. The training goal of this stage is to enable the network to better learn the inherent movement pattern of respiratory movement. Considering that respiratory movement is mainly driven by the diaphragm, the movement amplitude of the four-dimensional cone-beam CT image in the z direction will be larger, and the training of the second stage can enable the neural network to quickly pay attention to the information in the z direction. The model training strategy adopted by the present invention fully considers the characteristics of respiratory movement, and combines it with the characteristics of the four-dimensional separable convolutional neural network included in the present invention, and can complete the effective training of the four-dimensional neural network using only limited four-dimensional training samples.

[0068] In step 5, the input of the neural network is a set of multi-phase undersampled cone-beam CT images, which are affected by undersampling artifacts and have poor image quality, making it impossible to estimate the respiratory motion field through traditional image registration algorithms. The trained four-dimensional neural network model can distinguish the real structural information and image artifacts in undersampled cone-beam CT images in a data-driven manner, thereby achieving reliable estimation of the respiratory motion deformation field.

[0069] Compared with the prior art, the cone beam CT full respiratory cycle image reconstruction method based on four-dimensional neural network proposed in the present invention fully combines the advantages of data-driven methods and model-driven methods. The method proposed in the present invention explicitly models the respiratory motion process, transforms the respiratory motion correction problem into the respiratory motion deformation field estimation problem, conforms to the physical model of dynamic cone beam CT imaging, and has better interpretability and reconstruction accuracy than general deep learning methods; the estimation of the respiratory motion deformation field is realized by a four-dimensional neural network model trained with a large-scale simulation data set. The model fully learns the image prior contained in the four-dimensional cone beam CT image, and does not need to enhance the image of the highly undersampled cone beam CT image, nor does it need to estimate the respiratory motion deformation field through repeated iterative optimization. Instead, it can restore the respiratory motion pattern from the highly undersampled cone beam CT image in one step, and further reconstruct a high-quality cone beam CT image under the motion compensation framework, which has faster calculation speed and higher motion field accuracy than the general motion estimation-motion compensation method. The method proposed in the present invention has reliable performance, simple steps, and fast calculation speed, and is expected to be applied in clinical scenarios such as intraoperative cone beam CT imaging that require higher real-time performance of the algorithm.

[0070] Embodiment 2

[0071] The present invention can be applied to respiratory interventional surgery scenarios. In respiratory interventional surgery, doctors need to use surgical instruments to perform puncture sampling or ablation treatment on lesions in the lungs percutaneously or through natural cavities. During the operation, doctors need to repeatedly use cone beam CT to image the patient's lungs, confirm the relative position of the surgical instruments and the lesions, and adjust the operation based on this. The operation can be performed under local anesthesia. Therefore, the patient is in a natural breathing state during cone beam CT imaging, and respiratory motion correction needs to be introduced during the image reconstruction process.

[0072] Step 3: Use cone beam CT equipment to collect cone beam projection data P of the patient under free breathing, and use respiratory gating equipment to record the patient's respiratory signal. For cone beam CT systems that are not equipped with a respiratory gating system, the Amsterdam Shround method can be used to estimate the respiratory signal from the projection data P, and a respiratory cycle is divided into N phases. A schematic diagram of the respiratory signal can be seen. Figure 3 .

[0073] Step 4: Based on the collected respiratory signal, the projection data is independently reconstructed phase by phase to obtain a multi-phase undersampled cone-beam CT reconstructed image. In the respiratory interventional surgery scenario, the typical data collection time is 6 to 30 seconds, and the data collection angle range is 180°, such as Figure 4 As shown in , the projection data distribution has the characteristics of both "limited angle sampling" and "sparse angle sampling". Therefore, Figure 5 As shown in FIG. 1 , the undersampled cone-beam CT reconstructed image is covered by strong stripe artifacts in the cross section and is difficult to be directly used for estimating the respiratory motion deformation field.

[0074] Step 5: Combine N undersampled cone-beam CT reconstructed images into a four-dimensional image, input it into a four-dimensional neural network, and use the end of inspiration as the reference phase to obtain N three-dimensional deformation fields D i ,i=1,2,…,N. The N input undersampled cone-beam CT reconstructed images do not overlap in the data acquisition angle, so their image spatial information is complementary; each undersampled image corresponds to a different respiratory phase, and the change of its image morphology follows the continuous and periodic pattern of respiratory movement, so its time dimension information is correlated. During the calculation process, the four-dimensional neural network included in the present invention can synchronously mine the spatiotemporal information in the four-dimensional undersampled image, so as to better restore the respiratory movement. In addition, benefiting from the design of four-dimensional separable convolution, the model can directly process the complete four-dimensional cone-beam CT image during the reasoning process without the need to block the image, thereby making full use of the complete image information.

[0075] Step 6: The deformation field obtained by the output of the neural network is used to deform the undersampled CT images of each phase and then sum them to obtain the motion-corrected end-inspiratory reconstructed image X. This step reuses the multi-phase undersampled cone-beam CT reconstructed image calculated in step 4, and the motion-corrected image can be obtained after a simple image transformation. The corrected image can show the three-dimensional relative position relationship between the surgical instrument and the lesion in the patient's end-inspiratory phase, assisting the doctor in adjusting the surgical process.

[0076] Step 7: Invert the deformation field to obtain N three-dimensional inverse deformation fields By combining the specific phase image and the inverse deformation field, the motion correction image X corresponding to any respiratory phase in the whole respiratory cycle is reconstructed. i , i = 1, 2, ..., N, thus realizing dynamic cone beam CT image reconstruction. The purpose of this step is to obtain the position changes of surgical instruments and patient lesions in a complete breathing cycle, so as to provide image reference for possible dynamic surgical navigation systems and assist doctors in planning surgical paths under free breathing of patients. The correction results are shown in Figure 6 As shown, the tissue boundaries are clear and the relative positions of the surgical instruments and the lesions are clear.

[0077] Since respiratory interventional surgery often requires repeated cone beam CT scans, and the respiratory motion pattern of the same patient generally does not change significantly in a short period of time, the respiratory motion model obtained through the above steps can be used for subsequent scans to further accelerate the imaging process. Specifically, for subsequent scans, after repeating steps three and four above, there is no need to call the neural network again to estimate the respiratory motion model. The respiratory motion deformation field obtained in the first scan can be directly used to deform and sum the multi-phase undersampled cone beam CT reconstructed image to obtain the motion-corrected reconstructed image of the new scan.

[0078] Embodiment three:

[0079] The present invention can be applied to the radiotherapy scenario of thoracic and abdominal tumors. During the tumor radiotherapy process, the patient must first undergo a positioning cone-beam CT scan to align the spatial coordinate system of the patient when receiving treatment with the spatial coordinate system used by the patient when formulating the radiotherapy plan. However, patients with thoracic and abdominal tumors are in a free breathing state during the positioning cone-beam CT scan and radiotherapy. Respiratory movement will cause displacement and deformation of lesion tissue and normal tissue, so it is necessary to correct the respiratory movement, so as to achieve more accurate radiotherapy for the lesion tissue and protect the external tissue.

[0080] Step 3: Use cone beam CT equipment to collect cone beam projection data P of the patient under free breathing, and use respiratory gating equipment to record the patient's respiratory signal. For cone beam CT systems that are not equipped with respiratory gating equipment, the Amsterdam Shround method can be used to estimate the respiratory signal from the projection data P, and a respiratory cycle is divided into N phases. A schematic diagram of the respiratory signal can be seen. Figure 7 .

[0081] Step 4: Based on the collected respiratory signal, the projection data is independently reconstructed phase by phase to obtain a multi-phase undersampled cone-beam CT reconstructed image. In the tumor radiotherapy scenario, the typical data acquisition time is 60 seconds, and the data acquisition angle range is 360°. Figure 8 As shown in , the distribution of projection data is similar to the characteristics of “sparse angle sampling”. Therefore, Fig. 9 As shown in FIG. 1 , the undersampled cone-beam CT reconstructed image is covered by fine stripe-like artifacts in the cross section, and is difficult to be directly used for estimating the respiratory motion deformation field.

[0082] Step 5: Combine N undersampled cone-beam CT reconstructed images into a four-dimensional image, input it into a four-dimensional neural network, and use the end of inspiration as the reference phase to obtain N three-dimensional deformation fields D i,i=1,2,…,N. The N input undersampled cone-beam CT reconstructed images do not overlap in the data acquisition angle, so their image spatial information is complementary; each undersampled image corresponds to a different respiratory phase, and the change of its image morphology follows the continuous and periodic pattern of respiratory movement, so its time dimension information is correlated. During the calculation process, the four-dimensional neural network included in the present invention can synchronously mine the spatiotemporal information in the four-dimensional undersampled image, so as to better restore the respiratory movement. In addition, benefiting from the design of four-dimensional separable convolution, the model can directly process the complete four-dimensional cone-beam CT image during the reasoning process without the need to block the image, thereby making full use of the complete image information.

[0083] Step 6: The deformation field obtained by the neural network output is used to deform the undersampled CT images of each phase and then sum them to obtain the motion-corrected end-inspiratory reconstructed image X. This step reuses the multi-phase undersampled cone-beam CT reconstructed image calculated in step 4, and the motion-corrected image can be obtained after a simple image transformation. The corrected image can show the specific three-dimensional position of the lesion and normal tissue in the patient's end-inspiratory phase.

[0084] Step 7: Invert the deformation field to obtain N three-dimensional inverse deformation fields By combining the specific phase image and the inverse deformation field, the motion correction image X corresponding to any respiratory phase in the whole respiratory cycle is reconstructed. i , i = 1, 2, ..., N, thus realizing dynamic cone beam CT image reconstruction. The purpose of this step is to obtain the position change of the lesion and normal tissue in a complete respiratory cycle. The correction result is as follows Fig.10 As shown in the figure, the boundary between the tumor and normal tissue is clear, the relative position is clear, and the dynamic image can well depict the respiratory movement process. According to the dynamic cone beam CT image, the tumor radiotherapy plan can be corrected online. In order to achieve better treatment effects, the irradiation range can be expanded according to the dynamic image so that the lesion is always under a higher radiation intensity during the treatment process; in order to better protect the surrounding tissues, the execution time of radiotherapy can be adjusted according to the dynamic image, such as only executing it in part of the respiratory phase of the patient's respiratory cycle to avoid excessive radiation to external tissues.

[0085] Compared with the prior art, the cone beam CT full respiratory cycle image reconstruction method based on four-dimensional neural network proposed in the present invention fully combines the advantages of data-driven methods and model-driven methods. The method proposed in the present invention explicitly models the respiratory motion process, transforms the respiratory motion correction problem into the respiratory motion deformation field estimation problem, conforms to the physical model of dynamic cone beam CT imaging, and has better interpretability and reconstruction accuracy than general deep learning methods; the estimation of the respiratory motion deformation field is realized by a four-dimensional neural network model trained with a large-scale simulation data set. The model fully learns the image prior contained in the four-dimensional cone beam CT image, and does not need to enhance the image of the highly undersampled cone beam CT image, nor does it need to estimate the respiratory motion deformation field through repeated iterative optimization. Instead, it can restore the respiratory motion pattern from the highly undersampled cone beam CT image in one step, and further reconstruct a high-quality cone beam CT image under the motion compensation framework, which has faster calculation speed and higher motion field accuracy than the general motion estimation-motion compensation method. The method proposed in the present invention has reliable performance, simple steps, and fast calculation speed, and is expected to be applied in clinical scenarios such as intraoperative cone beam CT imaging that require higher real-time performance of the algorithm.

[0086] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for reconstructing the full respiratory cycle image of cone beam CT based on a four-dimensional neural network, characterized in that: Includes steps 1-5: Step 1: Acquire cone beam projection data and respiratory signals under respiratory state; Step 2: Divide a respiratory cycle into N phases, determine the respiratory phase i corresponding to each sub-data in the cone-beam projection data, i = 1, 2, ..., N, merge the sub-data with the same phase, and obtain multi-phase cone-beam projection data Q i , i = 1, 2, ..., N, for multi-phase cone beam projection data Q i Multi-phase undersampled CT reconstructed images were obtained using the filtered back projection algorithm. i=1,2,…,N; Step 3: Reconstruct N undersampled CT images Combined into a four-dimensional image, input into a four-dimensional neural network, take a certain phase as the reference phase, and obtain N three-dimensional deformation fields D i ,i=1,2,…,N; Step 4: The three-dimensional deformation field D obtained by the output of the four-dimensional neural network i Undersampled CT images of multiple phases with corresponding phases After deformation, weighted summation is performed to obtain the reference phase reconstructed image X after motion correction ref ; Step 5: Three-dimensional deformation field D i Inverse, get N three-dimensional inverse deformation fields i=1,2,…,N, for any respiratory phase, use the corresponding inverse deformation field to reconstruct the image X of the reference phase ref Perform image deformation and reconstruct the motion correction image X corresponding to any respiratory phase in the whole respiratory cycle i ,i=1,2,…,N。 2. The method for reconstructing the full respiratory cycle image of cone beam CT based on a four-dimensional neural network as claimed in claim 1, characterized in that: The respiratory signal is synchronously recorded using a respiratory gating device or calculated from projection data.

3. The method for reconstructing the full respiratory cycle image of cone beam CT based on a four-dimensional neural network as claimed in claim 1, characterized in that: The four-dimensional neural network is a four-dimensional separable convolutional neural network.

4. The method for reconstructing the full respiratory cycle image of cone beam CT based on a four-dimensional neural network as claimed in claim 3, characterized in that: The macro structure of the four-dimensional separable convolutional neural network is UNet.

5. The method for reconstructing the full respiratory cycle image of cone beam CT based on a four-dimensional neural network as claimed in claim 3, characterized in that: The four-dimensional separable convolutional neural network is composed of a plurality of cascaded convolutional layers, each convolutional layer comprises two consecutive four-dimensional separable convolutional modules and a nonlinear module, and the four-dimensional separable convolutional module is composed of a plurality of parallel and orthogonal low-dimensional convolutional modules.

6. The method for reconstructing the image of the cone beam CT full respiratory cycle based on a four-dimensional neural network as claimed in claim 5, characterized in that: The nonlinear module is a Leaky ReLu module.

7. The method for reconstructing the full respiratory cycle image of cone beam CT based on a four-dimensional neural network as claimed in claim 5, characterized in that: The four-dimensional separable convolution module consists of a spatial convolution in the x direction, a spatial convolution in the y direction, a spatial convolution in the z direction, and a temporal convolution in the t direction, wherein the temporal convolution in the t direction adopts a circular filling method, and the design of the circular filling is selected according to the periodicity of the respiratory motion.

8. The method for reconstructing the full respiratory cycle image of cone beam CT based on a four-dimensional neural network as claimed in claim 7, characterized in that: The four-dimensional separable convolutional neural network is trained by regression training. The training data set is obtained by computer-simulated four-dimensional cone-beam CT scanning, including dynamic projection data and corresponding respiratory signals, and multi-phase under-sampled CT reconstruction images are reconstructed by simulating clinical scanning scenes. The training data set contains multiple training samples, and each training sample contains a set of multi-phase under-sampled CT reconstruction images. i=1,2,…,N and a corresponding end-inspiratory reference image X. The training sample does not need to contain the true value of the respiratory deformation field. During the training process, the image after motion compensation is minimized. The L1 norm of the deviation from the end-inspiratory reference image X is used as the self-supervised optimization target, where ° is the image deformation operator, and the ADAM optimizer is used to optimize the parameters of the neural network.

9. The method for reconstructing the image of the cone beam CT full respiratory cycle based on a four-dimensional neural network as claimed in claim 8, characterized in that: The training of the four-dimensional separable convolutional neural network is divided into two stages. During the pre-training process of the first stage, the z-direction spatial convolution operator of each convolution module in the four-dimensional separable convolutional neural network is frozen, and this branch does not participate in the calculation. After completing the pre-training of the first stage, in the second stage, the z-direction spatial convolution in the four-dimensional separable convolution module is reactivated.

10. A computer device, characterized in that: include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by one or more processors, and the one or more applications are configured to execute the cone-beam CT full respiratory cycle image reconstruction method based on a four-dimensional neural network as described in any one of claims 1 to 9.

Images