Cone beam CT image reconstruction method, system, device and storage medium

Abstract

The invention discloses a cone beam CT image reconstruction method, system, device and storage medium, and relates to the field of CT reconstruction technology. The method comprises: collecting two-dimensional projection images to be reconstructed at multiple angles in cone beam CT; constructing a cone beam CT image reconstruction model; inputting the two-dimensional projection image into the cone beam CT image reconstruction model, performing block feature extraction on the two-dimensional projection image through a projection feature extraction module; fusing the projection features extracted at various angles through a projection feature cross-angle fusion module; learning the mapping relationship between projection domain features and reconstructed image domain through an image generation network to obtain a three-dimensional volume image; obtaining a spatial feature map of the image through an image domain reconstruction network, representing the CBCT volume image as a continuous intensity field, and sending the feature value of the spatial feature map into the continuous intensity field to obtain a reconstructed cone beam CT image, thereby accurately capturing all important information and improving the reconstruction capability.

Description

Technical Field

[0001] The present invention relates to the field of CT reconstruction technology, and in particular to a cone-beam CT image reconstruction method, system, device and storage medium. Background Art

[0002] X-ray computed tomography (CT) is an endoscopy technique commonly used in modern medicine. CT technology mainly utilizes an X-ray beam and a detector to rotate around the human body and continuously perform cross-sectional scanning. During each scan, the detector receives the attenuated X-ray information after passing through the human body and inputs it into a computer. The electronic computer reconstructs the image based on the received attenuated X-ray information to obtain an image of the detected part of the human body.

[0003] A NeRF-based two-stage sparse angle CBCT self-supervised reconstruction method is currently proposed for CBCT image reconstruction tasks in radiotherapy scenarios. By proposing a ray precise sampling module based on geometric priors, the precise path of the ray in the intensity field is obtained, and the attenuation of the X-ray after passing through the imaging object is simulated by the attenuation field. Then, the initial reconstructed image is optimized by the intensity field in a two-stage reconstruction framework, which solves a series of problems in the sparse angle CBCT reconstruction task. Although this method can reconstruct a high-quality target image, the NeRF-based reconstruction method relies on intensive network calculations to render the image, which makes its training and reasoning process very time-consuming and resource-consuming. It takes 15 to 40 minutes to reconstruct an image. Although there have been some recent works on speed optimization, computational efficiency is still a challenge for CBCT reconstruction tasks that require fast response. In addition, individual object generalization is an important problem faced by NeRF-based reconstruction methods. Although such methods can achieve good results in specific patient scenarios, the trained model is difficult to directly apply to other imaging objects. Each new scenario requires re-collecting data and optimizing the model, which affects its efficiency and flexibility in practical applications.

[0004] In summary, although the NeRF-based reconstruction method can in principle restore the complete scene from sparse data, it usually requires a large amount of sparse angular CBCT scan data for training. In the case of extremely sparse data, it is difficult for the model to accurately capture all important information, resulting in poor reconstruction quality. Summary of the invention

[0005] In view of the shortcomings of the prior art that, in the case of extremely sparse data, the model is difficult to accurately capture all important information, resulting in poor reconstruction quality, the present invention proposes a cone beam CT image reconstruction method, system, device and storage medium, which uses a projection domain reconstruction network and a spatial feature preservation module to each generate a preliminary CBCT image spatial feature block, and regresses the preliminary generated spatial feature values ​​to voxel intensity values ​​to obtain a CBCT reconstructed image, thereby solving the problems existing in the prior art.

[0006] A cone beam CT image reconstruction method comprises the following steps:

[0007] Collect two-dimensional projection images to be reconstructed at multiple angles in cone beam CT;

[0008] Constructing a cone beam CT image reconstruction model; the cone beam CT image reconstruction model includes a projection domain reconstruction network and an image domain reconstruction network;

[0009] The two-dimensional projection images to be reconstructed at multiple angles are respectively input into the cone beam CT image reconstruction model, and the multi-angle feature extraction module in the projection domain reconstruction network is used to extract block features of the two-dimensional projection images to be reconstructed at multiple angles. The projection features extracted at each angle are fused by the projection feature cross-angle fusion module with a self-attention mechanism. The mapping relationship between the fused projection features and the image domain reconstruction network is learned by the image generation module to obtain a three-dimensional volume image. The two-dimensional projection image is converted from the two-dimensional space to the three-dimensional space by the spatial feature preservation module based on geometric information in the image domain reconstruction network to obtain a three-dimensional spatial feature block.

[0010] The cone beam CT three-dimensional volume image is represented as a continuous intensity field through an implicit neural representation method, and the spatial coordinates and eigenvalues ​​of the three-dimensional points in the three-dimensional spatial feature block are sent into the continuous intensity field to obtain the reconstructed cone beam CT image.

[0011] Furthermore, the two-dimensional projection images to be reconstructed at multiple angles in cone beam CT are collected. The specific process is that during the CBCT imaging process, the X-ray source and the corresponding detector rotate around the scanned object, and the three-dimensional object is projected onto the two-dimensional detector through X-rays, so that the two-dimensional projection images are obtained from multiple angles.

[0012] Furthermore, the image domain reconstruction network extracts features from the two-dimensional projection image, and sends the extracted high-level features of multiple different scales to the image generation module in the projection domain reconstruction network to guide the learning process for image domain feature fusion.

[0013] Furthermore, the projection feature cross-angle fusion module that introduces the self-attention mechanism fuses the projection features extracted at each angle, specifically including the following steps:

[0014] Obtaining scanning angle values ​​of the projection image at multiple angles and projection feature maps at each angle generated by a projection feature extraction module;

[0015] Map each angle value into a continuous high-dimensional space through the embedding layer to obtain the angle information vector;

[0016] Each angle information vector is fused with the corresponding projection feature block respectively; the fusion process specifically includes: inputting the angle information vector into the fully connected layer to generate attention weights; applying the generated attention weights to the corresponding projection feature map through a dot multiplication operation, where each point on the feature map is multiplied by the corresponding attention weight to complete the fusion of projection features at each angle.

[0017] Furthermore, the two-dimensional projection image is converted from two-dimensional space to three-dimensional space through a spatial feature preservation module based on geometric information in the image domain reconstruction network to obtain a three-dimensional spatial feature block; specifically, the two-dimensional projection image to be reconstructed at multiple angles is converted from two-dimensional space to three-dimensional space by adopting a back-projection algorithm to obtain a spatial feature map of the image; wherein, the back-projection algorithm obtains the spatial feature map of the projection image by placing the intensity value of the pixel of the two-dimensional projection image back to the corresponding voxel of the 3D imaging subject located on the ray path connecting the pixel point and the X-ray source.

[0018] Furthermore, it also includes, after the two-dimensional projection image is converted from the two-dimensional space to the three-dimensional space through the spatial feature preservation module based on geometric information in the image domain reconstruction network, refining the spatial feature map in the three-dimensional space through the 3D U-Net network to generate a refined image spatial feature block.

[0019] The present invention also includes a cone beam CT image reconstruction system, comprising:

[0020] An acquisition module, used for acquiring two-dimensional projection images to be reconstructed at multiple angles in cone beam CT;

[0021] A model building module, used to build a cone beam CT image reconstruction model; the cone beam CT image reconstruction model includes a projection domain reconstruction network and an image domain reconstruction network;

[0022] An input module is used to input the two-dimensional projection images to be reconstructed at multiple angles into the cone beam CT image reconstruction model respectively, and extract block features of the two-dimensional projection images to be reconstructed at multiple angles through the multi-angle feature extraction module in the projection domain reconstruction network, and introduce the projection feature cross-angle fusion module of the self-attention mechanism to fuse the projection features extracted at each angle, and learn the mapping relationship between the fused projection features and the image domain reconstruction network through the image generation module to obtain a three-dimensional volume image; the two-dimensional projection image is converted from the two-dimensional space to the three-dimensional space through the spatial feature preservation module based on geometric information in the image domain reconstruction network to obtain a three-dimensional spatial feature block;

[0023] The reconstruction module is used to represent the cone beam CT three-dimensional volume image as a continuous intensity field through an implicit neural representation method, and to send the spatial coordinates and eigenvalues ​​of the three-dimensional points in the three-dimensional spatial feature block into the continuous intensity field to obtain a reconstructed cone beam CT image.

[0024] The present invention also includes a cone beam CT image reconstruction computer device, comprising: a memory, a processor and a computer program stored in the memory, and the processor implements the steps of the cone beam CT image reconstruction method when executing the computer program.

[0025] Furthermore, a readable storage medium is provided, wherein the readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, they are used to execute the steps of the cone beam CT image reconstruction method.

[0026] The present invention provides a cone beam CT image reconstruction method, system, device and storage medium, which have the following beneficial effects:

[0027] The present invention aims at the scenario of ultra-sparse angle CBCT image reconstruction task, in which a small number of projection images cannot provide enough information to accurately reconstruct the image of an unknown subject, and further designs a cone beam CT image reconstruction method; in the projection domain network, in view of the problem that the traditional CNNS network is difficult to capture the global information of the projection image, a multi-angle feature extraction module is designed, and feature representations at different angles are extracted through multiple branch networks; in view of the different sampling distributions of projections at different angles, a projection feature cross-angle fusion module is designed, and a self-attention mechanism is introduced to allow the model to learn the attention distribution between different projection images, thereby improving the interpretability and performance of the model; in view of the fact that a small number of projection images cannot provide enough information to accurately reconstruct the image of an unknown subject, more prior information needs to be introduced, and an image domain reconstruction network is constructed through a spatial feature preservation module based on geometric information, and the image domain reconstruction network is used to convert the projection domain data to the image domain, so as to realize the embedding of geometric prior information, and an implicit neural representation method is introduced to combine the cone beam CT three-dimensional volume image and the three-dimensional spatial feature block to obtain a reconstructed cone beam CT image, and the method can accurately capture all important information in the case of extremely sparse data, thereby improving the image reconstruction capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the structure of a cone beam CT image reconstruction method in an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the structure of a projection feature extraction module in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of a multi-angle projection feature extraction module in an embodiment of the present invention;

[0031] Figure 4 are projection images at different angles in the embodiments of the present invention;

[0032] Figure 5 Schematic diagram of the structure of the projection feature cross-angle fusion module in an embodiment of the present invention;

[0033] Figure 6 A schematic diagram of the geometric structure of a CBCT system in an embodiment of the present invention;

[0034] Figure 7 A schematic diagram showing the comparison between the experimental reconstruction result and the real image in an embodiment of the present invention;

[0035] Figure 8 Schematic diagram showing the difference between the reconstruction results of the experiment and the baseline in the embodiment of the present invention;

[0036] Fig. 9 Schematic diagram of reconstruction results of different methods in embodiments of the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0038] At present, the fully supervised methods in the field of sparse angle CBCT deep reconstruction are mainly divided into the following types: 1) image domain learning methods; 2) projection domain learning methods; 3) model-based end-to-end domain transformation methods; 4) geometric integration deep reconstruction methods. The image domain learning method uses the FDK algorithm to roughly reconstruct the sparse angle CBCT image, and then uses a two-dimensional CNNs model to denoise each slice image. Since FDK introduces a large number of streak artifacts in the sparse sampling scenario, the performance of this method is still limited. The projection domain learning method uses the information extraction ability of CNNs or transformers to repair the sparse angle sinusoidal information, and reconstructs the CT image with the enhanced sinusoidal and original sinusoidal images. It has achieved satisfactory results in the two-dimensional medical image reconstruction task, but is not suitable for three-dimensional CBCT image reconstruction scenarios. The model-based end-to-end domain transformation method uses large-scale data to train convolutional neural networks to directly learn the mapping from the original measured projection domain to the CBCT image domain. When this method is trained on large-scale data, there is a risk of overfitting, that is, the model may perform well on the training set, but poorly on unseen data. The geometric integration deep learning framework method embeds the geometric prior information of the CBCT system into the network through the back-projection operation, bridging the dimensionality gap between the 2D projection domain and the 3D image domain and improving the imaging performance of the network model. However, the reconstructed image details are severely lost at ultra-sparse angles, and there is room for further improvement.

[0039] A CBCT depth reconstruction method, the overall process is as follows Figure 1 As shown in the figure, it includes a multi-angle projection feature extraction module, a projection feature cross-angle fusion module and a spatial feature preservation module based on geometric information. The projection feature extraction module, the projection feature cross-angle fusion module and the image generation network constitute the projection domain reconstruction network. In the projection domain network, the projection feature extraction module based on Swin Transformer is first used to extract block features of the projection image to improve the expression and generalization capabilities of the features. Then, considering the geometric angle information of the imaging system, the self-attention mechanism is introduced through the projection feature cross-angle fusion module to learn the attention distribution between different projection images, so that the projection features at each angle are effectively fused. Finally, the mapping relationship between the projection domain features and the reconstructed image domain is learned through the generation network to obtain the intermediate representation of the three-dimensional volume image.

[0040] (1) Multi-angle projection feature extraction module: In the sparse angle (angle number ≤ 10) CBCT deep reconstruction method, feature extraction of projection images is a key step to improve reconstruction accuracy and quality. The main purpose is to extract sufficient information from sparse and limited projection data and strengthen the representation of key structural features so that the reconstruction algorithm can effectively reconstruct high-quality images.

[0041] At present, most sparse angle CBCT deep reconstruction methods use CNNs to extract projection features. Traditional CNNs methods have strong local feature extraction capabilities, but it is difficult to capture the global information of the projection map. Compared with CNNs, the Swin Transformer model is good at capturing global contextual information and long-distance dependencies, which is conducive to maintaining richer information. At the same time, through its hierarchical design, it can extract features at different scales. This is particularly valuable for processing medical images with various structures and sizes, and can help deep reconstruction algorithms better restore everything from small details to overall structures. In addition, compared to CNNs that can only capture local features through fixed convolution kernels, the Swin Transformer model can dynamically focus on the most relevant parts of the image with its inherent self-attention mechanism. In sparse angle CBCT reconstruction, this means that the model can focus more on the projection features that have the greatest impact on the reconstruction quality, with higher visual interpretability.

[0042] In the CBCT reconstruction task, the global features in the projection and the relationship between the features affect the quality of the reconstructed image. Therefore, based on the Swin Transformer model, the present invention designs a projection feature extraction module to model the relationship between all pixels in the projection image and learn the global features. The module consists of multiple residual blocks, each of which has two common Swin Transformer modules and convolutional layers. The structure is as follows Figure 2 This module has the advantages of both Swin Transformer and CNNS, and has the ability to extract global and local features.

[0043] In the ultra-sparse angle CBCT (cone beam computed tomography) reconstruction task, the method proposed in the present invention aims to achieve high-quality image reconstruction from only 10 scanning angles. Existing deep learning reconstruction technology usually relies on a single feature extraction network to process multiple projection images at the same time. This approach often fails to fully tap the unique information and subtle differences in each projection image, resulting in the loss of important feature information, thus affecting the quality of the reconstructed image. In addition, since the information contained in multiple projection images is highly multidimensional and complex, the processing method of a single network is also difficult to fully integrate the features between different perspectives to reflect the spatial correlation between projection images, which limits the model's ability to integrate information from different perspectives.

[0044] The present invention designs a multi-angle projection feature extraction module, the architecture of which is as follows: Figure 3 . For the input projection data of 10 angles, this method adopts the design of 5 sub-network branches, each branch specializes in processing the projection data of two specific angles. Through this design, each sub-network branch can focus on extracting unique feature information from its corresponding angle. The input received by the entire multi-angle projection feature extraction module is 10 2D projection images, and the output is the comprehensive feature representation corresponding to these images.

[0045] This multi-angle feature extraction method can effectively solve the problems of information loss and insufficient feature fusion when a single network processes multi-view data. By assigning projection data to different network branches for processing, this module aims to enhance the model's sensitivity and feature extraction capabilities for each angle projection and promote the effective integration of features between different perspectives, thereby improving the model's ability to recognize spatial correlations. This is crucial for high-quality reconstruction of data acquired from extremely limited angles.

[0046] (2) Projection feature cross-angle fusion module: In the CBCT imaging system, projection data is obtained by projecting the object under test at different angles. The projections of objects at different angles have different sampling distributions. The angle information reflects the sampling conditions of the projection data and helps to improve the accuracy of reconstruction of the internal structure of the object. Using the deep learning-based reconstruction algorithm, when performing image reconstruction, the angle information can be used to guide the reconstruction process and help the algorithm understand the relationship between each projection and other projections. For example, information obtained from the front (face-to-face) direction is usually more important than information from the side or oblique direction (such as Figure 4 The frontal view can be given a higher weight (for example, the projection image at 0° and the projection image at 90°). Therefore, the richer the angular information, the more accurately the algorithm can restore high-quality image details and structural information, thereby improving the resolution and contrast of the image. However, the current end-to-end methods for sparse angle CBCT depth reconstruction do not take this into account.

[0047] Considering that projections at different angles contribute differently to the reconstruction results, projections at certain angles may contain more useful information about the scanned object, while other angles may be less useful due to problems such as overlap, artifacts or noise, the present invention designs a projection feature cross-angle fusion module based on the self-attention mechanism. The module self-learns the weight of each projection image feature based on the geometric parameters of the CBCT system, namely the angle information of the X-ray source and the detector, and then performs cross-angle fusion on multi-angle projection features, emphasizing projections that contribute greatly to the reconstruction quality, while reducing the impact of projections with less information or poor quality, so that the model can learn the attention distribution between different projection images, thereby allocating different weights at different angles according to the dynamic characteristics of the input data, optimizing the overall reconstruction process, and improving the interpretability and performance of the model.

[0048] The structure of the projection feature cross-angle fusion module is as follows: Figure 5 As shown in the figure, the input of this module is the scanning angle values ​​of 10 projection images and the 5 projection feature maps under 10 projection images generated by the multi-angle feature projection feature extraction module. First, the 10 discrete angle values ​​are mapped to a continuous high-dimensional space through the embedding layer to obtain the angle information vector, so that the network can better learn the complex relationship between angle information and image features. Then the angle information vector is fused with the corresponding projection feature block. The specific process is as follows: the angle information vector is sent to a fully connected layer with a depth of 2 and a width of 64 to generate attention weights; then the generated attention weights are applied to the corresponding projection feature map through a dot multiplication operation, where each point on the feature map is multiplied by the corresponding attention weight, so that the model can pay more attention to certain features according to different angles; finally, the projection feature map with fused angle information is convolved, and the 5 projection feature maps after the convolution operation are spliced ​​and sent to the image generation network for the next stage of operation.

[0049] (3) In the spatial feature preservation module based on geometric information, a differentiable back-projection operator is first used to realize the conversion of projection domain data to image domain, and then a 3D U-Net network is used for image enhancement. Multiple high-level features of different scales extracted from the network are sent to the generative network in the projection domain for image domain feature fusion.

[0050] At present, the end-to-end domain transformation reconstruction method uses large-scale data to train convolutional neural networks to directly learn the mapping from the original measured projection domain to the CBCT image domain. However, this type of reconstruction model lacks clear geometric guidance and cannot accurately learn three-dimensional spatial information from two-dimensional projections. In addition, the reconstruction performance depends on various factors such as the distribution of training data, network structure, and hyperparameters, resulting in almost no transparency and interpretability of the generated model. This type of model can achieve good performance on training data, but when faced with new data that is different from the training set (such as data obtained by different devices, or new tasks), the generalization ability will be affected, resulting in performance degradation.

[0051] In addition, during the CBCT imaging process, the X-ray source and the corresponding detector rotate around the scanned object, and the three-dimensional object is projected onto the two-dimensional detector through the X-ray, so that projection images can be obtained from multiple angles. The projection at each angle contains information about the scanned object in different directions. This process is essentially a compression of information dimensions, in which multiple points in three-dimensional space are mapped to one point on the two-dimensional projection image. Due to the complexity of the scanned object and the diversity of its internal structure, the information captured by the projection images from different angles is not independent, but contains overlapping and intertwined feature information. These features are highly coupled, making it difficult to extract their respective spatial features separately.

[0052] To address the above problems, the present invention proposes a spatial feature preservation module based on geometric information, which combines the prior geometric relationship between the input projection and the output data domain to enhance the imaging capability of the end-to-end domain transformation reconstruction method.

[0053] In this module, a differentiable back-projection operator without any learning parameters is first used to deterministically expand the internal structural information of the imaging object in the sparse projection into the three-dimensional image space to obtain a spatial feature map. This process greatly simplifies the overall learning task and facilitates the flow of information from the two-dimensional projection domain to the three-dimensional image domain. After obtaining the spatial feature map of the sparse angular data, the 3D U-Net network is then used to perform the spatial feature enhancement task. At the same time, the learned image spatial features are also sent to the projection domain reconstruction network to give the projection domain network clear geometric guidance.

[0054] The 3D U-Net network is an encoder-decoder structure. The encoder part extracts the features of the 3D volume image generated by the back-projection operator, and the decoder part increases the spatial dimension and reduces the number of channels of the extracted features to enhance the details and structure of the image. The decoder part of the 3D U-Net consists of 5 upsampling layers, and the image generation network in the projection domain network also consists of 5 upsampling layers. The image features of multiple scales learned by the encoder of the 3D U-Net are sent to the image generation network in the projection domain network to guide the learning process of the projection domain network.

[0055] like Figure 6 As shown in the figure, during the CBCT imaging data acquisition process, the penetrating rays emitted by the X-ray source pass through the scanned object and are projected onto the detection plane by integrating the intensities of intersecting voxels in the three-dimensional object. This process is called forward projection. In this way, multiple projection images of the internal anatomical structure of the scanned object are obtained.

[0056] In forward projection, the geometric relationship between the incident X-ray beam and the scanned object is determined by the geometric parameters of the imaging system, including the distance between the X-ray source and the rotation center, the distance between the rotation center and the detector plane, the physical size of the object voxel and the detector pixel, the cone beam geometry, and the projection angle. All of this information is relevant geometric prior information provided by the CBCT imaging system.

[0057] The goal of CBCT image reconstruction is to solve the inverse problem and reconstruct a three-dimensional subject from two-dimensional projections at different viewing angles. In order to link the two-dimensional projection domain with the three-dimensional image domain, a back-projection operation is performed based on the previous imaging system geometric parameters to convert the two-dimensional projection back into three-dimensional space. The back-projection operation is based on the same set of geometric relationships as the forward projection, and the specific process is as follows:

[0058] The intensity value of the pixel on the 2D projection is placed back on the corresponding voxel of the 3D imaging subject located on the ray line path connecting the pixel point and the X-ray source. Through the above operation, the relationship between the pixel-by-pixel intensity on the two-dimensional projection and the three-dimensional voxel intensity is embedded in the three-dimensional image reconstruction process. Therefore, the present invention constructs a differentiable back-projection operator, integrates the geometric information of the imaging system into the deep learning framework, and uses the back-projection operation to realize the conversion of the projection domain data to the image domain.

[0059] Through the back-projection operator, a sparse angle-projected spatial feature map is obtained, which provides a rough image spatial structure for the reconstruction network. However, due to the sparsity of the projection data, the spatial feature map has blurred edges and needs to be further enhanced and refined. In view of the fact that the U-Net network has good detail feature capture capabilities and is widely used in medical image tasks, the present invention uses a 3D U-Net network to generate further refined image spatial feature blocks from the fine structure and edge information in the sparsely projected spatial feature map. In addition, multiple high-level features of different scales learned in the 3D U-Net network will be sent to the generative network in the projection domain for image domain feature fusion to enhance the spatial feature retention capability of the reconstruction model.

[0060] The projection domain reconstruction network and the spatial feature preservation module each generate a preliminary CBCT image spatial feature block. Finally, an implicit neural representation method is introduced to represent the CBCT volume image as a continuous intensity field. A fully connected layer with a depth of 4 and a width of 64 is used as an intensity field inference module to regress the preliminary generated spatial feature values ​​to voxel intensity values ​​to obtain the CBCT reconstructed image.

[0061] The present invention proposes an ultra-sparse angular CBCT depth reconstruction method, which completes the CBCT image reconstruction task under 10 scanning angles.

[0062] This method represents the CBCT volume image as a continuous intensity field, and the reconstructed image is regarded as a continuous function g(·) of a point in three-dimensional space. The property of point p in space in this field represents its intensity value v, that is, v = g(p). Therefore, the reconstruction problem is restated as regressing the intensity value of any three-dimensional space point from 10 two-dimensional projections I, that is, v = g(I,p).

[0063] First, in the projection domain network, projection features are extracted from 2D projection images at given 10 angles. Since the information richness contained in projection images at different angles is different, a self-attention mechanism is introduced to allow the model to learn the attention distribution between different projection images, further integrating the angle information of the imaging system. In addition, for ultra-sparse angle imaging tasks, a small number of projection images cannot provide enough information to accurately reconstruct the image of an unknown subject, and more prior information needs to be introduced: CBCT imaging system geometric information.

[0064] In view of the characteristics of the CBCT imaging mechanism, one pixel pair in the projection image corresponds to multiple voxel points in the CBCT volume image, and the information of multiple voxel points in the target image is stacked on one eigenvalue of the projection feature, which is severely coupled.

[0065] To address these two problems, a spatial feature preservation module based on geometric information is proposed. The two-dimensional projection images at 10 sparse angles are converted into three-dimensional spatial feature maps through a differentiable back-projection operator based on the geometric information of the imaging system. Then, a 3D U-Net network is used for image enhancement. Under the parallel processing of the projection domain network and the image domain network, a three-dimensional spatial feature block is obtained respectively. Finally, given a three-dimensional point in the three-dimensional spatial feature block, the spatial coordinates and feature values ​​of the point are sent into the continuous intensity field to obtain the intensity value of the point in the field, that is, the voxel value of the CBCT volume image.

[0066] Experimental proof: First, the effectiveness of the projection feature extraction module is verified using a mouse dataset collected in the laboratory. The dataset contains two sets of CBCT projection data of the same mouse acquired at two different time points. Each set of projection data has 720 CBCT projection images, which are recorded as set1 and set2 respectively. The postures of the mice in the two groups are quite different. Due to the small number of samples in the mouse dataset, the LIDC-IDRI dataset used in this invention is used to verify the performance of the overall network. The dataset contains a total of 1018 patient CT scan images.

[0067] For the mouse dataset, the FDK algorithm was first used to reconstruct the 3D volume images of set1 and set2 as reference images. 1050 and 260 CBCT projection images were randomly selected from the 1440 CBCT projection images in the mouse dataset for model training and validation. The remaining 130 CT projection images were used for model testing. The original projection image size was 1944×1536, and the reconstructed image size was 500×512×512. In order to improve the training rate, the size of the projection images was uniformly adjusted to 256×256, and the size of the 3D volume images was adjusted to 138×256×256.

[0068] In the LIDC-IDRI dataset, due to the different imaging parameters, the resolution of each scanned image is different. First, each scanned image is resampled to a resolution of 1×1×1mm3. Then the resampled scanned image is cropped to a size of 138×256×256 as a reference image. In the experiment, 916 patient images were randomly selected for training and validation, and 109 patient images were used for testing. For the projection image of each patient's CT image, the tomography tool ODL was still used to generate 10 uniform projections in the range of 0° to 180° to simulate ultra-sparse angle cone beam projection data. The specific CBCT scanning parameters are shown in Table 1.

[0069] Table 1 Ultra-sparse CBCT projection generation parameters

[0070]

[0071] For training, the proposed method is implemented in PyTorch version 1.12 and Python version 3.9, and two NVIDIA-A100 graphics cards are used as GPU. The loss function is defined as the sum of the mean square error (MSE) and structural similarity (SSIM) between the network-predicted CBCT image and the reference image, and the calculation method is shown in formula (1), where and β are used to adjust the balance between different losses. In the present invention, α=β=0.5. The Adam optimizer is used to minimize the loss function. The initial learning rate is 0.00002, and the learning rate decays according to the validation loss value. The specific process is that if the validation loss remains unchanged within 5 cycles, the learning rate is reduced by 2 times. In addition, the batch-size is set to 4, and 200 epochs are trained.

[0072] L=αL MSE +βL SSIM (1)

[0073] Experimental results:

[0074] (1) Comparison results of projection feature extraction modules: The effectiveness of the projection feature extraction module in the method of the present invention was verified on a mouse dataset. As shown in Table 2, the effects of two different feature extraction modules on the performance of the sparse angle CBCT reconstruction method were compared.

[0075] Table 2 Reconstruction results of different methods

[0076]

[0077] There are two groups of comparative experiments in the table. The first group is the baseline model. The baseline model uses the Shen-Net[8] method, which is a standard end-to-end reconstruction model based on codec and uses conventional CNNs to extract projection features. The second group of experiments uses the projection feature extraction module in the method of the present invention to process the projection image. The decoders of the reconstruction methods in the two groups of experiments use the same network structure.

[0078] The experimental results show that, compared with the baseline, the model reconstruction performance is improved after using the projection feature extraction module of the present invention, especially the MSE and PSNR are greatly improved. Figure 7 It can also be seen that the image reconstructed by the method of the present invention has a clear outline and is slightly different from the real image. At the same time, the results of the method of the present invention are compared with the results of the baseline model. The two have the same structural similarity, but the contour features reconstructed by the method of the present invention are clearer and have fewer artifacts. The difference in the reconstruction results of the two methods can also be seen from the comparison Figure 8 It can be seen in.

[0079] (2) Comparative test of projection image input method: The multi-angle projection feature extraction module of the method of the present invention uses 5 sub-network branches to extract projection image features under 10 scanning angles, and each branch network processes projection data under two scanning angles at the same time. In order to explore how to choose the projection angle, a verification test was carried out on the LIDC-IDRI dataset. The projection images with mutually orthogonal scanning angles (orthogonal projection) and the projection images with adjacent scanning angles (adjacent projection) were selected as the input of each branch network in the projection feature extraction module. The experimental results are shown in Table 3.

[0080] Table 3 Experimental results of different projection image input methods

[0081]

[0082] From the experimental results in the table, it can be seen that selecting two adjacent projection images as the input of the branch network in the multi-angle projection feature extraction module is the best choice. The numerical indicators of the reconstruction results under this method are better than those of the orthogonal projection input. The structural features of the two adjacent projection images are similar, and the purpose of each branch network is to extract similar features of multi-angle projection to provide more spatial correlation for the feature fusion of different views and enhance the model's ability to fuse information from different perspectives. The experimental results verify the effectiveness of inputting two adjacent projection images into each branch network.

[0083] (3) Comparative experiment with other end-to-end reconstruction methods: In order to fully demonstrate the superiority of the sparse angle CBCT deep reconstruction method proposed in this invention, this invention selected traditional CBCT reconstruction methods and ultra-sparse angle reconstruction methods proposed in recent years for comparison, namely FDK, SART, GIIR, SNAF, and DIF. In addition, in order to verify the effectiveness of the method of this invention in the case of ultra-sparse angles, it was also compared with the method proposed in this invention and the NeRF-based reconstruction method. Each method completed the CBCT reconstruction task on the projection image data at 10 scanning angles. The experimental results are shown in Table 4.

[0084] Table 4 Comparative experimental results with other reconstruction methods

[0085]

[0086] The experimental results in the table show that compared with the traditional CBCT reconstruction method and the end-to-end reconstruction method proposed in recent years, the end-to-end sparse angle CBCT deep reconstruction method proposed in the present invention has better performance in image quality and structural recovery in the reconstruction results of ultra-sparse angle task scenarios. Due to the lack of a large amount of projection data, traditional CBCT reconstruction methods such as FDK and SART have achieved very poor reconstruction results. Compared with the same end-to-end method, the method proposed in the present invention has obvious improvement. In particular, compared with DIF, the method of the present invention has improved by 4.3 points in the structural similarity index (SSIM) and by 3.34 in the peak signal-to-noise ratio (PS NR), and the performance has been significantly improved. Compared with the two-stage sparse angle CBCT self-supervised reconstruction method based on NeRF, the improvement is even greater.

[0087] The reconstruction results of representative methods are visualized. Fig. 9 As shown, (a) reference image; (b) the method of the present invention; (c) DIF; (d) two-stage sparse angle CBCT self-supervised reconstruction method based on NeRF; (e) SART; (f) FDK. It can also be seen from the figure that the method of the present invention can effectively restore the contours and various features of the CBCT volume image. The reconstruction results of the FDK method have a large number of artifacts, and no image features can be observed. The reconstructed image of the SART method has obvious noise and low resolution. The reconstructed image of DIF has roughly similar contour features to the method of the present invention, but the reconstructed image of the method of the present invention is clearer and has a higher contrast. The reconstruction result of the method of the present invention also has obvious noise, and the structural information is seriously lost, which further illustrates that the reconstruction method based on the neural radiation field requires more scanning degree data for optimization and is not suitable for ultra-sparse angle CBCT reconstruction tasks.

[0088] (4) Comparison of ablation experiments on each module

[0089] In order to verify the effectiveness of each module, an ablation experiment was conducted on the method of the present invention. Different combination experiments were conducted on the projection feature cross-angle fusion module, the projection domain reconstruction network, and the image domain network to explore the effectiveness of each module and the impact on the performance of the reconstruction method. The experimental results are shown in Table 5.

[0090] Table 5 Ablation experiments of each module

[0091]

[0092] The experimental results show that after adding the projection feature cross-angle fusion module to the projection domain network, the network can learn the attention distribution between different projection images, and the reconstruction performance of the method is significantly improved. At the same time, with the addition of the spatial feature preservation module, the projection domain network obtains accurate image spatial features, which further improves the performance of the reconstruction method.

[0093] In view of the ultra-sparse angle CBCT image reconstruction task scenario, a small number of projection images cannot provide enough information to accurately reconstruct the image of an unknown subject. The present invention designs an ultra-sparse angle CBCT deep reconstruction method based on deep fusion of geometric information. In the projection domain network, in view of the problem that the traditional CNNS network is difficult to capture the global information of the projection image, a multi-angle feature extraction module based on SwinTransformer is designed. Through multiple branch networks, feature representations at different angles are extracted. In view of the problem that projections at different angles have different sampling distributions, a projection feature cross-angle fusion module is designed, and a self-attention mechanism is introduced to allow the model to learn the attention distribution between different projection images, thereby improving the interpretability and performance of the model. In order to solve the problem that a small number of projection images cannot provide enough information to accurately reconstruct the image of an unknown subject, more prior information needs to be introduced. An image domain reconstruction network is constructed through a spatial feature preservation module based on geometric information. The projection domain data is converted to the image domain using the back-projection operator to embed geometric prior information. Then, the 3DU-Net network is used to enhance the spatial feature map, and multiple high-level features of different scales extracted from the image domain reconstruction network are sent to the projection domain network to guide the learning process of the projection domain network and improve the image reconstruction capability. Finally, experiments were conducted on the mouse dataset and LIDC-IDRI dataset collected in the laboratory to verify the effectiveness of each module in the method of the present invention.

[0094] Based on the same inventive concept, the present invention also proposes a cone beam CT image reconstruction system, comprising:

[0095] The acquisition module is used to acquire two-dimensional projection images to be reconstructed at multiple angles in cone beam CT.

[0096] The model building module is used to build a cone beam CT image reconstruction model; the cone beam CT image reconstruction model includes a projection domain reconstruction network and an image domain reconstruction network.

[0097] The reconstruction module is used to input the two-dimensional projection images to be reconstructed at multiple angles into the cone beam CT image reconstruction model to obtain the reconstructed cone beam CT image; the reconstruction module includes:

[0098] The projection domain unit is used to extract block features from the two-dimensional projection image through a multi-angle feature extraction module based on Swin Transformer in the projection domain reconstruction network, to fuse the projection features extracted from each angle through a projection feature cross-angle fusion module with a self-attention mechanism, and to learn the mapping relationship between the fused projection features and the image domain reconstruction network through an image generation module to obtain a three-dimensional volume image.

[0099] The image domain unit is used to convert the two-dimensional projection image from the two-dimensional space to the three-dimensional space through the spatial feature preservation module based on geometric information in the image domain reconstruction network to obtain a three-dimensional spatial feature block.

[0100] The reconstruction unit is used to introduce an implicit neural representation method to represent the cone beam CT three-dimensional volume image as a continuous intensity field, and to send the spatial coordinates and eigenvalues ​​of the three-dimensional points in the three-dimensional spatial feature block into the continuous intensity field to obtain a reconstructed cone beam CT image.

[0101] Based on the same inventive concept, the present invention also proposes a cone-beam CT image reconstruction computer device, comprising: a memory, a processor, and a computer program stored in the memory, and the processor implements the steps of the cone-beam CT image reconstruction method when executing the computer program.

[0102] Based on the same inventive concept, the present invention also proposes a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the cone beam CT image reconstruction method.

[0103] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A cone beam CT image reconstruction method, characterized in that: The following steps are involved: Collect two-dimensional projection images to be reconstructed at multiple angles in cone beam CT; Constructing a cone beam CT image reconstruction model; the cone beam CT image reconstruction model includes a projection domain reconstruction network and an image domain reconstruction network; The two-dimensional projection images to be reconstructed at multiple angles are respectively input into the cone beam CT image reconstruction model, and the multi-angle feature extraction module in the projection domain reconstruction network is used to extract block features of the two-dimensional projection images to be reconstructed at multiple angles. The projection features extracted at each angle are fused by the projection feature cross-angle fusion module with a self-attention mechanism. The mapping relationship between the fused projection features and the image domain reconstruction network is learned by the image generation module to obtain a three-dimensional volume image. The two-dimensional projection image is converted from the two-dimensional space to the three-dimensional space by the spatial feature preservation module based on geometric information in the image domain reconstruction network to obtain a three-dimensional spatial feature block. The cone beam CT three-dimensional volume image is represented as a continuous intensity field through an implicit neural representation method, and the spatial coordinates and eigenvalues ​​of the three-dimensional points in the three-dimensional spatial feature block are sent into the continuous intensity field to obtain the reconstructed cone beam CT image.

2. A cone beam CT image reconstruction method according to claim 1, characterized in that: The two-dimensional projection images to be reconstructed at multiple angles in cone beam CT are collected. Specifically, during CBCT imaging, an X-ray source and a corresponding detector are rotated around the scanned object, and the three-dimensional object is projected onto the two-dimensional detector through X-rays to obtain two-dimensional projection images at multiple angles.

3. The cone beam CT image reconstruction method according to claim 1, characterized in that: It also includes extracting features from the two-dimensional projection image through the image domain reconstruction network, and sending the extracted multiple high-level features of different scales into the image generation module in the projection domain reconstruction network to guide the learning process for image domain feature fusion.

4. The cone beam CT image reconstruction method according to claim 1, characterized in that: The projection feature cross-angle fusion module that introduces the self-attention mechanism fuses the projection features extracted at each angle, specifically including the following steps: Obtaining scanning angle values ​​of the projection image at multiple angles and projection feature maps at each angle generated by a projection feature extraction module; Map each angle value into a continuous high-dimensional space through the embedding layer to obtain the angle information vector; Each angle information vector is fused with the corresponding projection feature block respectively; the fusion process specifically includes: inputting the angle information vector into the fully connected layer to generate attention weights; applying the generated attention weights to the corresponding projection feature map through a dot multiplication operation, where each point on the feature map is multiplied by the corresponding attention weight to complete the fusion of projection features at each angle.

5. The cone beam CT image reconstruction method according to claim 1, characterized in that: The two-dimensional projection image is converted from the two-dimensional space to the three-dimensional space through the spatial feature preservation module based on geometric information in the image domain reconstruction network to obtain the three-dimensional spatial feature block; Specifically, a back-projection algorithm is used to convert a two-dimensional projection image to be reconstructed at multiple angles from a two-dimensional space to a three-dimensional space to obtain a spatial feature map of the image; wherein the back-projection algorithm obtains the spatial feature map of the projection image by placing the intensity value of the pixel of the two-dimensional projection image back to the corresponding voxel of the 3D imaging subject located on the ray path connecting the pixel point and the X-ray source.

6. The cone beam CT image reconstruction method according to claim 5, characterized in that: It also includes, after the two-dimensional projection image is converted from the two-dimensional space to the three-dimensional space by the spatial feature preservation module based on geometric information in the image domain reconstruction network, refining the spatial feature map in the three-dimensional space through the 3D U-Net network to generate a refined image space feature block.

7. A cone beam CT image reconstruction system, characterized in that: include: An acquisition module, used for acquiring two-dimensional projection images to be reconstructed at multiple angles in cone beam CT; A model building module, used to build a cone beam CT image reconstruction model; the cone beam CT image reconstruction model includes a projection domain reconstruction network and an image domain reconstruction network; An input module is used to input the two-dimensional projection images to be reconstructed at multiple angles into the cone beam CT image reconstruction model respectively, and extract block features of the two-dimensional projection images to be reconstructed at multiple angles through the multi-angle feature extraction module in the projection domain reconstruction network, and introduce the projection feature cross-angle fusion module of the self-attention mechanism to fuse the projection features extracted at each angle, and learn the mapping relationship between the fused projection features and the image domain reconstruction network through the image generation module to obtain a three-dimensional volume image; the two-dimensional projection image is converted from the two-dimensional space to the three-dimensional space through the spatial feature preservation module based on geometric information in the image domain reconstruction network to obtain a three-dimensional spatial feature block; The reconstruction module is used to represent the cone beam CT three-dimensional volume image as a continuous intensity field through an implicit neural representation method, and to send the spatial coordinates and eigenvalues ​​of the three-dimensional points in the three-dimensional spatial feature block into the continuous intensity field to obtain a reconstructed cone beam CT image.

8. A cone beam CT image reconstruction computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the processor implements the steps of the cone beam CT image reconstruction method according to any one of claims 1 to 6 when executing the computer program.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the cone-beam CT image reconstruction method according to any one of claims 1 to 6.

Images