CT implicit reconstruction method based on single X-ray image

Abstract

The invention discloses a CT implicit reconstruction method based on a single X-ray image. The method comprises the following steps: training each sample in a CT data set into an implicit expression; constructing a weight generation network, recombining the implicitly expressed network weights into a data set, and training the weight generation network; predicting a weight generation network parameter and a camera parameter corresponding to the input X-ray image through an image encoder; according to the predicted weight generation network parameters and camera parameters, performing end-to-end optimization on the encoder in combination with the generated implicitly expressed miniaturized body rendering result; in the reasoning stage, an X-ray image is input for implicit expression prediction of the corresponding CT, and the task of CT reconstruction is achieved. According to the CT implicit reconstruction method based on the single X-ray image, through learning in an implicit space, implicit expression prediction can be conducted on input of the single X-ray image at low cost, and then high-resolution CT reconstruction is achieved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and medicine, and in particular to a CT implicit reconstruction method based on a single X-ray image. Background Technology

[0002] Three-dimensional reconstruction technology has significant applications in the medical field, especially in computed tomography (CT) scans. Due to the penetrating power of X-rays, CT data can be reconstructed from multiple X-ray fluoroscopic images surrounding the human body, greatly aiding in accurate clinical diagnosis and treatment planning. Traditional CT reconstruction requires numerous fluoroscopic projection views and complex reconstruction algorithms, such as filtered backprojection or iterative reconstruction algorithms. Multiple X-ray imaging sessions expose patients to high radiation doses in a short period. For example, the average annual natural background radiation dose for humans is approximately 2.4 mSv, while a typical chest CT scan delivers a radiation dose of approximately 5-7 mSv, which can have negative effects on the human body. Therefore, researching how to predict CT structures using a single X-ray image is of broad significance.

[0003] Compared to the radiation dose, time, and cost of a CT scan, obtaining a single frontal (AP) X-ray view is significantly easier. If a patient's three-dimensional internal structure could be accurately inferred from a single X-ray view in clinical practice, it would greatly reduce the cost of medical diagnosis. However, predicting three-dimensional structure from a single X-ray view is inherently flawed; any scaling of the structure along any projection direction can lead to very similar results. Therefore, prior knowledge of the three-dimensional structure is essential to aid the prediction process.

[0004] Among the most similar methods retrieved in the field of CT reconstruction are analytical or iterative CT reconstruction methods, sparse-view CT reconstruction methods, and data-driven CT reconstruction methods. However, early analytical or iterative CT reconstruction methods often required a large number of X-ray images as a basis to obtain CT structural data. The reconstruction results were poor for sparse-view scenarios, and they were even less applicable to scenarios with a single X-ray image.

[0005] Sparse-view CT reconstruction methods are often based on NeRF or similar 3D reconstruction methods, enabling universal 3D reconstruction applications in the medical field. However, these methods often yield meaningless reconstruction results for single X-ray images because they lack prior knowledge and cannot infer the complete 3D structure from minimal viewpoint information. Data-driven CT reconstruction methods, on the other hand, train neural networks on existing datasets to perform CT reconstruction, thus requiring less stringent input during the inference phase. They can achieve CT reconstruction results with dual-plane input or even single-image input. However, these methods often perform calculations in explicit voxel space. Due to hardware limitations (mainly computing power and video memory limitations), the reconstruction accuracy of these methods is often lower because they require learning at lower resolutions. Furthermore, these methods lack universality for image input and require strictly fixed input viewpoints and camera parameters.

[0006] Therefore, existing CT reconstruction technologies either require a large number of X-ray images (more than 100) for CT reconstruction, which is very accurate but exposes people to a lot of radiation; or they sacrifice some accuracy requirements and use methods such as NeRF for sparse view reconstruction; or they are based on data-driven neural network learning for CT reconstruction, but this often requires high computing power and is difficult to meet high resolution requirements.

[0007] Therefore, in view of the limitations of the existing technology, the present invention proposes a CT implicit reconstruction method based on a single X-ray image. Summary of the Invention

[0008] The purpose of this invention is to provide a CT implicit reconstruction method based on a single X-ray image. By learning the implicit space, it is possible to predict the implicit representation (IR) of a single X-ray image input at a low cost, thereby achieving high-resolution CT reconstruction.

[0009] To achieve the above objectives, the present invention provides a CT implicit reconstruction method based on a single X-ray image, comprising the following steps: Step S1: During the training phase, each CT sample in the CT dataset is trained as an implicit representation. Step S2: Construct a weight generation network WG, combine the implicitly expressed network weights into a new dataset, and train the weight generation network. Step S3: Predict the weight generation network parameters and camera parameters corresponding to the input X-ray image using the image encoder; Step S4: Generate network parameters and camera parameters based on the predicted weights, and perform end-to-end optimization of the encoder by combining the generated implicit expression of differentiable volume rendering results. Step S5: In the inference stage, an X-ray image is input, and its corresponding CT implicit representation prediction is performed to realize the subsequent tasks of new perspective synthesis and CT reconstruction.

[0010] Preferably, in step S1, each CT sample in the CT dataset is trained as an implicit representation, and the specific process is as follows: Each CT sample in the CT dataset is trained as an implicit representation (IR). The IR is a multilayer perceptron (MLP), and the network is trained by directly using the three-dimensional spatial values ​​in the CT sample as supervision. For a CT dataset, each CT sample consists of several slices. Each slice reconstructs a matrix of HU values ​​according to its internal definition, representing the absorption capacity of X-rays at the corresponding spatial location, which is called the attenuation coefficient. Based on the physical meaning of the HU value, when the value is -1000, i.e., air, the attenuation of X-rays is close to 0. Therefore, all CT data are linearly mapped to [0, 1] according to the interval [-1000, 3000] to facilitate the learning of implicit representations. For a CT sample, a 4-layer MLP with residual connections is used as an implicit representation to learn the mapping from spatial 3D coordinates to attenuation coefficients. The implicit representation is trained directly using three-dimensional data as labels. In each iteration, vertices and corresponding interpolation decay coefficients are randomly selected from [-1, 1]^3 for optimization until convergence.

[0011] Preferably, in step S2, a weight generation network WG is constructed, the implicitly expressed network weights are combined into a new dataset, and the weight generation network is trained. The specific process is as follows: After training each CT sample in the dataset to be in the form of an implicit representation, a weight generation network WG is constructed. The network weights of the implicit representation are combined into a new dataset, and the weight generation network is trained. The network accepts a parameter vector that conforms to a normal distribution as input and samples from a Gaussian distribution of fixed dimensions to generate an implicit representation of the CT data.

[0012] Preferably, in step S3, the image encoder predicts the weight generation network parameters and camera parameters corresponding to the input X-ray image. The specific process is as follows: After training the weight generator, the image encoder is used to predict the WG parameters and camera parameters of the X-ray image. The WG parameters are latent space vectors that follow a normal distribution, and the camera parameters are a six-dimensional vector, as shown below: ; in, This represents the camera's position coordinates in three-dimensional space; , and This represents the camera's spherical coordinates; This represents the distribution of camera parameters when generating the training set, where the camera's focal length is a fixed value. During training, the objective optimization function As shown below: ; in, For reconstruction loss; Loss of identity; The weights for the configurable identity loss; This is the regularization loss; The weights for the configurable regularization loss.

[0013] Preferably, in step S4, network parameters and camera parameters are generated based on the predicted weights. Combined with the generated implicit expression differentiable volume rendering results, the encoder is optimized end-to-end. The specific process is as follows: According to Beer-Lambert's law, the intensity of X-rays decreases exponentially with the attenuation coefficient of the penetrating medium and the path length. That is, for a ray that has passed through N sampling points, its final intensity is as follows: ; In the formula, The initial light intensity; Indicates the distance between adjacent sampling points; Indicates the attenuation coefficient; For sampling points; All X-ray volume renderings invert the final light intensity and take... =1, as shown below: .

[0014] Preferably, in step S5, during the inference stage, an X-ray image is input, and its corresponding implicit CT representation is predicted to achieve the subsequent tasks of new perspective synthesis and CT reconstruction. The specific process is as follows: During the inference phase, for a given single X-ray image, the latent space parameter z and the unwanted camera parameter c are predicted, and then the implicit 3D reconstruction result IR is obtained through WG. For CT reconstruction tasks, rasterized vertex coordinates of arbitrary resolution are uniformly selected within [-1, 1]^3 and fed into the MLP of IR to obtain CT reconstruction results. The model is fitted to the X-ray image input using an optimization method, that is, an optimal solution is obtained through optimization, as shown below: ; in, Input X-ray image; Rendering for volume.

[0015] Therefore, this invention employs the aforementioned implicit CT reconstruction method based on a single X-ray image. Through learning in the implicit space, it can predict the implicit representation (IR) of a single X-ray image input at a low cost, thereby achieving high-resolution CT reconstruction. This invention decouples training set preparation, generative model training, and parameter prediction, enabling better application. The parameterized generative model can perform CT generation tasks in the implicit representation weight space, greatly reducing the complexity of 3D generation in medical images.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is a flowchart of an implicit CT reconstruction method based on a single X-ray image according to the present invention; wherein, (i), (ii), and (iii) are training phases; (i) means training each CT sample in the dataset as an IR; (ii) means training a WG; (iii) means training the network structure of the image encoder; (iv) is the inference phase, which means how to predict the IR and the corresponding CT based on the X-ray input when applying it; Figure 2 These are the experimental results of the method proposed in this invention on the thoracic dataset; where (a) represents the input; (b) represents the visualization effect of the predicted CT; (c) represents the slice of the predicted CT; and (d) represents the result of synthesizing the new perspective X-ray image. Detailed Implementation

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] This invention discloses a CT implicit reconstruction method based on a single X-ray image, comprising the following steps: Step S1: During the training phase, each CT sample in the CT dataset is trained as an implicit representation. Step S2: Construct a weight generation network (WG), combine the implicitly expressed network weights into a new dataset, and train the weight generation network. Step S3: Predict the weight generation network parameters and camera parameters corresponding to the input X-ray image using the image encoder; Step S4: Generate network parameters and camera parameters based on the predicted weights, and perform end-to-end optimization of the encoder by combining the generated implicit expression of differentiable volume rendering results. Step S5: In the inference stage, an X-ray image is input, and its corresponding CT implicit representation prediction is performed to realize the subsequent tasks of new perspective synthesis and CT reconstruction.

[0020] Example 1 like Figure 1 As shown, the training and inference phases of this invention are intuitively illustrated, along with the detailed components of each phase. The specific content of these two phases will be explained in detail below.

[0021] Step S1: Train each CT sample in the CT dataset into an implicit representation.

[0022] The proposed method framework focuses on the implicit representation of objects, without explicitly processing data such as point clouds or voxels. Each CT sample in the CT dataset is trained as an implicit representation (IR). This IR is a simple multi-layer perceptron (MLP), and the network is trained directly using the 3D spatial values ​​in the CT samples as supervision.

[0023] For a CT dataset, each CT sample is generally composed of a variable number of slices. Each slice is reconstructed according to its internal definition to obtain a specific matrix of HU values, which represents the absorption capacity of X-rays at the corresponding spatial location and is called the attenuation coefficient.

[0024] Considering the physical meaning of the HU value, when the value is -1000 (air), the attenuation of X-rays is close to 0. All CT data are linearly mapped to [0, 1] according to the interval [-1000, 3000] to facilitate the learning of implicit representations.

[0025] For a CT sample, a 4-layer MLP with residual connections is used as the implicit representation, such as... Figure 1 As shown in (i), the mapping from the three-dimensional coordinates of the learning space to the attenuation coefficient is shown.

[0026] The implicit representation is trained using three-dimensional data as labels, with each iteration starting from [-1, 1]. 3 In the process, a large number of vertices and their corresponding interpolation decay coefficients are randomly selected for optimization, and the training continues until convergence.

[0027] Under the specified parameter settings, each CT sample, after implicit expression compression, is expressed using only approximately 130K parameters, achieving efficient and high-fidelity compression. For example, compared to 256... 3 The compression ratio is approximately 0.78%.

[0028] It should be noted that there are no restrictions on the training data. This invention is a universal method. This embodiment is implemented on a chest CT dataset, but it does not mean that it can only be applied to chest CT. Other CT data such as brain, pelvis or knee are also applicable.

[0029] Step S2: Construct a weight generation network (WG), combine the implicitly expressed network weights into a new dataset, and train the weight generation network.

[0030] After training each CT sample in the dataset to be in the form of an implicit representation, a weight generation network (WG) is constructed. The network weights of the implicit representation are combined into a new dataset, and the weight generation network is trained. This network can accept a parameter vector that conforms to a normal distribution as input to generate an implicit representation of CT data.

[0031] This invention uses a variational autoencoder (VAE) to generate IR weights. Once the VAE is trained, it can sample from a Gaussian distribution with a fixed dimension (256 in the experiments of this invention), and the decoder can generate an IR based on the latent parameter vector.

[0032] It should be noted that the weight generation network mentioned in this invention is not limited to the VAE implementation method; any model and method that can map fixed-dimensional random parameters to IR parameters is applicable.

[0033] Step S3: Predict the weighted generation network parameters and camera parameters corresponding to the input X-ray image using an image encoder.

[0034] After training the weight generator, the image encoder is used to predict the WG parameters and camera parameters of the X-ray image. The WG parameters are denoted as the normally distributed latent space vectors mentioned in step S2, and the camera parameters are a six-dimensional vector, as shown below: ; in, This represents the camera's position coordinates in three-dimensional space; , and This represents the camera's spherical coordinates; This represents the distribution of camera parameters when generating the training set, where the camera's focal length is a fixed value. Furthermore, to avoid ambiguity, the camera referred to here is an X-ray emission source.

[0035] In fact, once each CT is encoded as an implicit representation, all subsequent training is unsupervised. This part is also unsupervised; for each IR sample, multiple random selections from the distribution... Camera parameters are selected and rendered into X-ray composite images to form the training and validation sets for this part of the network.

[0036] like Figure 1 As shown in (iii), the parameters of WG are frozen, and only the image encoder E is trained. The image encoder uses ResNet50 for image feature extraction, followed by a fully connected layer to regress the WG parameters and camera parameters.

[0037] During training, the objective optimization function As shown below: ; in, For reconstruction loss; Loss of identity; The weights for the configurable identity loss; This is the regularization loss; The weights for the configurable regularization loss.

[0038] Step S4: Generate network parameters and camera parameters based on the predicted weights, and perform end-to-end optimization of the encoder by combining the generated implicit expression of differentiable volume rendering results.

[0039] According to Beer-Lambert's law, the intensity of X-rays decreases exponentially with the attenuation coefficient of the penetrating medium and the path length. That is, for a ray that has passed through N sampling points, its final intensity is as follows: ; In the formula, The initial light intensity; Indicates the distance between adjacent sampling points; Indicates the attenuation coefficient; These are the sampling points.

[0040] Based on this formula, it can be found that the larger the attenuation coefficient on a light path, the smaller the final light intensity. This does not conform to the intuitive understanding of medical images (areas with greater attenuation, such as bone regions, are brighter). Therefore, all X-ray volume renderings in this invention invert the final light intensity and take... =1, as shown below: .

[0041] Step S5: In the inference stage, an X-ray image is input, and its corresponding CT implicit representation prediction is performed to realize the subsequent tasks of new perspective synthesis and CT reconstruction.

[0042] In the reasoning stage, such as Figure 1As shown in (iv), for a given single X-ray image, the present invention can predict the latent space parameter z and the unwanted camera parameter c, and then obtain the implicit 3D reconstruction result IR through WG.

[0043] For CT reconstruction tasks, rasterized vertex coordinates of arbitrary resolution can be uniformly selected within [-1, 1]^3 and fed into the MLP of IR to obtain CT reconstruction results.

[0044] This invention also provides an optional optimization method to make the model fit the X-ray image input as closely as possible, that is, to obtain an optimal solution through optimization, as shown below: ; in, Input X-ray image; Rendering for volume.

[0045] Example 2 This embodiment was implemented and validated on the LIDC-IDRI thoracic CT dataset, and the experimental results are as follows: Figure 2 As shown.

[0046] First, a computer hardware and software configuration for neural network training and inference verified by this invention is provided, as shown in Table 1.

[0047] Table 1 Computer Hardware and Software Configuration

[0048] The implementation process then includes pre-training and application after training. The actual implementation is the application on the thoracic cavity dataset. This requires additional code to read the dataset content and convert it into a standardized format, which is not part of the core method of the invention and will not be elaborated further.

[0049] Finally, in the application phase, a single X-ray image close to the front is selected as input from the test set and other datasets. The method proposed in this invention can obtain the corresponding CT reconstruction results based on the input. The prediction process for each input takes about 1-2 minutes and has practical application value.

[0050] Therefore, this invention employs the aforementioned implicit CT reconstruction method based on a single X-ray image. Through learning in the implicit space, it can predict the implicit representation (IR) of a single X-ray image input at a low cost, thereby achieving high-resolution CT reconstruction. This invention decouples training set preparation, generative model training, and parameter prediction, enabling better application. The parameterized generative model can perform CT generation tasks in the implicit representation weight space, greatly reducing the complexity of 3D generation in medical images.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A CT implicit reconstruction method based on a single X-ray image, characterized in that, Includes the following steps: Step S1: During the training phase, each CT sample in the CT dataset is trained as an implicit representation. Step S2: Construct a weight generation network WG, combine the implicitly expressed network weights into a new dataset, and train the weight generation network. Step S3: Predict the weight generation network parameters and camera parameters corresponding to the input X-ray image using the image encoder; Step S4: Generate network parameters and camera parameters based on the predicted weights, and perform end-to-end optimization of the encoder by combining the generated implicit expression of differentiable volume rendering results. Step S5: In the inference stage, an X-ray image is input, and its corresponding CT implicit representation prediction is performed to realize the subsequent tasks of new perspective synthesis and CT reconstruction.

2. The CT implicit reconstruction method based on a single X-ray image according to claim 1, characterized in that, In step S1, each CT sample in the CT dataset is trained as an implicit representation, and the specific process is as follows: Each CT sample in the CT dataset is trained as an implicit representation (IR). The IR is a multilayer perceptron (MLP), and the network is trained by directly using the three-dimensional spatial values ​​in the CT sample as supervision. For a CT dataset, each CT sample consists of several slices. Each slice reconstructs a matrix of HU values ​​according to its internal definition, representing the absorption capacity of X-rays at the corresponding spatial location, which is called the attenuation coefficient. Based on the physical meaning of the HU value, when the value is -1000, i.e., air, the attenuation of X-rays is close to 0. Therefore, all CT data are linearly mapped to [0, 1] according to the interval [-1000, 3000] to facilitate the learning of implicit representations. For a CT sample, a 4-layer MLP with residual connections is used as an implicit representation to learn the mapping from spatial 3D coordinates to attenuation coefficients. The implicit representation is trained directly using three-dimensional data as labels. In each iteration, vertices and corresponding interpolation decay coefficients are randomly selected from [-1, 1]^3 for optimization until convergence.

3. The CT implicit reconstruction method based on a single X-ray image according to claim 1, characterized in that, In step S2, a weight generation network WG is constructed, the implicitly expressed network weights are combined into a new dataset, and the weight generation network is trained. The specific process is as follows: After training each CT sample in the dataset to be in the form of an implicit representation, a weight generation network WG is constructed. The network weights of the implicit representation are combined into a new dataset, and the weight generation network is trained. The network accepts a parameter vector that conforms to a normal distribution as input and samples from a Gaussian distribution of fixed dimensions to generate an implicit representation of the CT data.

4. The CT implicit reconstruction method based on a single X-ray image according to claim 1, characterized in that, In step S3, the image encoder predicts the weighted generation network parameters and camera parameters corresponding to the input X-ray image. The specific process is as follows: After training the weight generator, the image encoder is used to predict the WG parameters and camera parameters of the X-ray image. The WG parameters are latent space vectors that follow a normal distribution, and the camera parameters are a six-dimensional vector, as shown below: ； in, This represents the camera's position coordinates in three-dimensional space; , and This represents the camera's spherical coordinates; This represents the distribution of camera parameters when generating the training set, where the camera's focal length is a fixed value. During training, the objective optimization function As shown below: ； in, For reconstruction loss; Loss of identity; The weights for the configurable identity loss; This is the regularization loss; The weights for the configurable regularization loss.

5. The CT implicit reconstruction method based on a single X-ray image according to claim 1, characterized in that, In step S4, network parameters and camera parameters are generated based on the predicted weights. Combined with the generated implicit expression differentiable volume rendering results, the encoder is optimized end-to-end. The specific process is as follows: According to Beer-Lambert's law, the intensity of X-rays decreases exponentially with the attenuation coefficient of the penetrating medium and the path length. That is, for a ray that has passed through N sampling points, its final intensity is as follows: ； In the formula, The initial light intensity; Indicates the distance between adjacent sampling points; Indicates the attenuation coefficient; For sampling points; All X-ray volume renderings invert the final light intensity and take... =1, as shown below: 。 6. The CT implicit reconstruction method based on a single X-ray image according to claim 1, characterized in that, In step S5, during the inference phase, an X-ray image is input, and its corresponding implicit CT representation is predicted to achieve the subsequent tasks of new perspective synthesis and CT reconstruction. The specific process is as follows: During the inference phase, for a given single X-ray image, the latent space parameter z and the unwanted camera parameter c are predicted, and then the implicit 3D reconstruction result IR is obtained through WG. For CT reconstruction tasks, rasterized vertex coordinates of arbitrary resolution are uniformly selected within [-1, 1]^3 and fed into the MLP of IR to obtain CT reconstruction results. The model is fitted to the X-ray image input using an optimization method, that is, an optimal solution is obtained through optimization, as shown below: ； in, Input X-ray image; Rendering for volume.

Images