3D reconstruction method of sparse view X-ray images based on hybrid features

Abstract

The present invention discloses a sparse viewing angle X-ray image three-dimensional reconstruction method based on hybrid features. The method uses gradient values ​​as weights to guide sampling rays so that the sampling rays are distributed in areas with more obvious changes as much as possible. Three-plane decomposition features and hash coding features are mixed, and the hybrid features are sent to an attenuation field prediction network to predict attenuation values. An attenuation weight prediction network (W‑NAF) is used to predict the attenuation weight of each sampling pixel point on the X-ray path. The attenuation value and attenuation weight value at the sampling pixel point are obtained through the two networks. The predicted projection pixel value is generated by accumulating the attenuation values ​​along the sampling rays, and then the network is optimized by minimizing the difference between the predicted projection and the real X-ray image.

Description

Technical Field

[0001] The invention relates to medical image processing technology, and in particular to a sparse viewing angle X-ray light image three-dimensional reconstruction method based on mixed features. Background Art

[0002] X-ray 3D reconstruction combines traditional methods and deep learning methods to achieve excellent performance in medical imaging. However, they usually require X-rays emitted from hundreds of angles for training. This results in high radiation doses and poses significant health risks. In recent years, experts and scholars have made significant progress in achieving 3D reconstruction of medical images. Starting from the reconstruction method, the related algorithms are divided into the following categories according to different reconstruction ideas.

[0003] Traditional CBCT reconstruction methods: Traditional CBCT reconstruction algorithms are pioneering algorithms in the field of CBCT reconstruction, especially filtered back projection and iterative reconstruction algorithms, which represent fundamental advances in the field of X-ray imaging. These traditional reconstruction methods are usually based on complex mathematical and physical modeling. They have played a vital role in medical imaging. Filtered back projection is a widely used technique that preprocesses the projection data by high-pass filtering before back projection, and then uses Radon transform to convert the preprocessed projection back to image space for CBCT reconstruction. Yu et al. proposed an innovative back projection filter (BPF) algorithm for CBCT reconstruction of the region of interest, which effectively solves the truncation artifacts inherent in FDK. On the other hand, the iterative method uses algebraic reconstruction technology to minimize the objective function and iteratively improve the model accuracy. SART uses the root mean square error to improve the algebraic reconstruction technique (ART). ASD-POCS is an iterative method with a total variation regularizer.

[0004] Although these methods can provide satisfactory results, they usually rely on hundreds of predictions, making them less efficient and more resource-intensive. Moreover, in the context of sparse view X-ray reconstruction, these algorithms can suffer from significant issues such as artifacts and edge blurring, which affect the clarity and accuracy of the reconstructed image.

[0005] Supervised learning CBCT reconstruction methods: With the rapid development of computer technology, deep learning-based methods have revolutionized medical image reconstruction. Solving many limitations of traditional algorithms, XCTNet uses deep learning networks to reconstruct 3D volume data from a single X-ray image. Volume data is generated by extracting prior knowledge from 2D pixels and using a self-attention mechanism for adaptive feature optimization. X2CT-GAN uses a generative adversarial network framework to reconstruct 3D volume data from two orthogonal X-ray images. CCX-RayNet uses a fusion network combined with semantic priors to provide detailed anatomical structure and texture information for different organs, achieving accurate reconstruction of different semantics. Lin proposed a new deep intensity field network (DIF-Net) for CBCT reconstruction, demonstrating the potential of innovation in this field in deep learning. In IntraTomo, a framework combining learning-based and model-based hybrid methods is proposed, which has some effect in solving highly ill-posed inverse problems in CT reconstruction and improving image quality. Although these deep learning methods have achieved impressive results, they often require a large amount of training data. This limits their development. In the medical field, data sets are relatively precious, and patient privacy issues limit the access to data sets, which seriously hinders the development and widespread application of CBCT reconstruction technology. This data scarcity is a major obstacle to the development and improvement of deep learning methods in medical imaging.

[0006] Self-supervised learning CBCT reconstruction methods: Self-supervised learning methods have shown great promise in 3D reconstruction of medical images. IntraTomo combines the advantages of learning-based and model-based methods to iteratively reconstruct through a novel sinusoidal prediction module and a self-supervised geometry optimization module. Neural radiance fields are well-known for their implicit scene representation and are widely used in new view synthesis and 3D reconstruction. Due to the powerful expression ability of NeRF, MedNeRF combines NeRF with generative adversarial networks (GAN) to propose generative radiance fields for X-ray 3D reconstruction, which enables the reconstruction of CT projections from several or even single view X-rays. NeAT introduces Neural Adaptive Tomography (NeAT) to combine neural features with adaptive explicit representations for tomography reconstruction. Lee proposed an iterative neural reconstruction method for limited-angle computed tomography reconstruction, which combines coordinate-based neural representations with iterative reconstruction techniques. NAF proposed an algorithm based on neural radiance fields to simulate the attenuation process of X-rays in a self-supervised manner for X-ray 3D reconstruction, which improves the quality of reconstruction to a certain extent. In recent years, some new feature representations have also been used for 3D reconstruction to improve the efficiency of CBCT reconstruction. SAX-NeRF proposed a structure-aware X-ray neural radiation density field (SAX-NeRF) framework for sparse view X-ray 3D reconstruction. They designed a linear attention mechanism (Lineformer) and a local-global (MLG) ray sampling strategy as the backbone to effectively capture the internal structure of the object. These advanced NeRF-based technologies effectively solve the difficulties associated with training time and large amounts of training data. However, since radiation dose can cause harm to the human body, the radiation dose will be controlled within a certain range during X-ray scanning, which will cause a lot of noise in the generated X-ray images, especially the thickness of human tissue at different angles is different, and the reconstructed image still cannot clearly depict the soft tissue details inside the human body. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a method for three-dimensional reconstruction of sparse perspective X-ray images based on mixed features.

[0008] Technical solution: A method for three-dimensional reconstruction of sparse viewing angle X-ray images based on mixed features of the present invention comprises the following steps:

[0009] Step 1: For the X-ray image, gradient-guided ray sampling is performed;

[0010] First, the gradient value of each pixel i of the X-ray projection in the image is calculated; then, the light corresponding to the pixel is randomly selected from the total pixels of the X-ray projection with the corresponding sampling probability P(i); during the selection process, the sampling probability is increased in the area with higher gradient values, and the sampling probability is reduced in the area with lower gradient values; for example, the gradient value of each pixel point can be calculated first, and then the gradient values ​​of all the pixel points are accumulated to obtain the total gradient value, and the probability of each pixel point (that is, different areas) is the gradient value of this point divided by the total gradient value (as shown in formula (2));

[0011] Step 2: Predict the attenuation field to obtain the attenuation value and attenuation weight value of the sampled pixel. The specific method is as follows:

[0012] Step 2.1: Based on the rays sampled in step 1, extract their three-plane decomposition features T and hash code features M respectively. H ; Then the three-plane decomposition feature T and hash code feature M H The mixed feature H is obtained by concatenation and mixing;

[0013] Step 2.2, input the obtained mixed feature H into the neural attenuation field prediction network, which includes two parallel branch network modules: the attenuation weight prediction network W-NAF and the attenuation value prediction network A-NAF;

[0014] First, the attenuation value prediction network A-NAF predicts the attenuation value μ of each sampling pixel point on the output X-ray i and feature F p , then the feature F p With the hash code feature M H The mixed feature input attenuation weight prediction network W-NAF, the attenuation weight prediction network W-NAF predicts the attenuation weight W of the output sampling pixel i , the decay weight W i and attenuation value μ i Multiply to get the final attenuation value μ i ′;

[0015] Step 3: Based on the attenuation value μ of the sampled pixel point obtained in step 2 i and the attenuation weight W i , optimize the neural attenuation field prediction network.

[0016] In order to reduce the memory and time of training sampling rays while achieving the best training results, step 2 adopts a gradient-guided sampling strategy. By using the gradient value to guide the sampling process, this innovative strategy ensures that more structural information is captured and improves the effectiveness of X-ray projection training. The sampling weights are adjusted in a targeted manner based on the calculated gradient value, thereby ensuring a more effective and information-rich sampling process, thereby obtaining better training results and improving overall performance. The calculation formulas for the pixel gradient value and the sampling probability are as follows:

[0017]

[0018] G x is the horizontal gradient of pixel i, G y is the vertical gradient of pixel i, G i is the total gradient value of pixel i;

[0019]

[0020] P(i) is the sampling probability of pixel i, and N is the total number of pixels on the X-ray image. The probability of each pixel is used to guide the random sampling process, ensuring a higher sampling rate at locations with significant gradient values, thereby effectively capturing local structural features.

[0021] The many-to-one characteristic of the hash function is very consistent with the inherent sparse attenuation changes of human organs, so that hash coding can focus on relevant details and select appropriate spectra. The present invention uses hash coding as position coding to capture high-frequency details. However, due to the limited X-ray projection data and lack of prior information in the sparse view, the accurate representation of the three-dimensional neural attenuation field is hindered. Therefore, only using hash coding can capture some high-frequency details, but it cannot effectively display local details, and only using hash coding may cause local information loss.

[0022] To solve the above technical problems, step 2 of the present invention introduces a three-plane method for X-ray 3D reconstruction. The three-plane features are good at representing local details, while the hash coding features can effectively capture high-frequency information. The hash coding features are combined with the three-plane decomposition features, so that the network can express both high-frequency features and complex details. The process of obtaining the mixed feature H of the sampled pixel point is as follows:

[0023] Get the hash code feature M of the sampled pixel H :

[0024] M H (p; Θ) = [I(H 1 ),…,I(H L )] T (3)

[0025] In the above formula,

[0026] is a bitwise XOR operation, H() is a hash function, L is a multi-resolution voxel grid; π is a unique large prime number; P is the query point, c is the coordinate; I represents the linear interpolation of the feature vector from c;

[0027] Obtain the three-plane decomposition feature T of the sampled pixel point, that is, project the sampled pixel point onto three planes (xy, yz, xz) to obtain the two-dimensional position information, and then use bilinear interpolation to extract the feature T from the three planes xy ,T yz and T xz Then, T xy ,T yz and T xz Spliced ​​together to form a three-plane decomposition feature T:

[0028] T=T xy +T yz +T xz (5)

[0029] The hash code feature M H The hybrid feature H is obtained by splicing and mixing with the three-plane decomposition feature T. The expression is as follows:

[0030] H=T+M H (6)

[0031] The hybrid feature H effectively fuses high-frequency information and local detail information, significantly improving the accuracy and richness of 3D reconstruction. The hybrid feature is subsequently input into the MLP network to predict the attenuation value μ of the sampling point in 3D space, ensuring precise and detailed reconstruction and significantly improving the accuracy and effectiveness of 3D imaging.

[0032] Furthermore, the neural attenuation field prediction network uses a multi-jump method based on the MLP network to predict the attenuation weight of each sampling pixel point, thereby obtaining the attenuation value when the X-ray penetrates different organs and tissues;

[0033] The attenuation value prediction network A-NAF uses a 6-layer MLP network, and uses a multi-hop method to connect the hash code features to the output of each layer of the MLP network, and together as the input of the next layer of the MLP network; for example, the input of the second layer is the output and hash code features of the first layer, the input of the third layer is the output and hash code features of the second layer, and so on...; effectively reducing overfitting and minimizing information loss;

[0034] In the 6-layer MLP network, the activation functions of the middle layer and the last layer are ReLU function and Sigmod function respectively;

[0035] The expression is as follows:

[0036] F k =Sigmoid(ReLU(MLP(H k-1 +M H ,θ))) (7)

[0037] H k =F k ⊙M H (8)

[0038] Where F k is the output feature of the kth layer, H k is the input feature of the next layer, M H is the hash code feature of the input; ⊙ represents the connection operation.

[0039] Furthermore, the neural attenuation field prediction network uses a multi-jump method based on the MLP network to predict the attenuation weight of each sampling pixel point, thereby obtaining the attenuation value when the X-ray penetrates different organs and tissues.

[0040] The attenuation weight prediction network W-NAF uses a 6-layer MLP network, and the feature Fp output by the attenuation value prediction network A-NAF is combined with the hash code feature M H The concatenated mixture is used as the input of the first layer, and the position information is input into the other five-layer MLP network using a multi-hop method;

[0041] E k =ReLU(MLP(E k-1 +M H ,θ)) (9)

[0042] W k =Sigmod(E f ) (10)

[0043] The first input feature E 1 =F p +M H , where F p Represents the output features of the decay weight prediction network (W-NAF);

[0044] E k is the output feature of the kth intermediate layer of the MLP network, E f is the output feature of the last layer of the MLP network, W k is the weight of the corresponding sampling point.

[0045] Furthermore, the detailed method of step 3 is:

[0046] First, the attenuation value μ of the X-ray upsampled pixel isi and the attenuation weight W i multiply;

[0047] After obtaining the product, the pixel intensity is obtained by integrating along the X-ray direction;

[0048] μ i ′=W i ×μ i

[0049] Next, a new X-ray image is synthesized by attenuation according to Beer’s law, and the intensity of the X-ray passing through the material is reduced by exponential integration of the X-ray path;

[0050] Finally, the update is performed by minimizing the L1 loss between the real projection and the synthesized projection. The L1 loss can be better used for this task than the L2 loss. The loss function L1 is defined as follows

[0051]

[0052] In the above formula, μ i ′ refers to the final attenuation value; I r (r) refers to the actual pixel value; I s (r) refers to the predicted pixel value, and B represents a batch, that is, a batchsize.

[0053] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0054] 1. The present invention uses the gradient value as a weight to guide the sampling ray, so that the sampling ray is distributed as much as possible in the area with more obvious changes, ensuring that more structural information is captured, improving efficiency without affecting the reconstruction quality, and improving the effectiveness of X-ray projection training;

[0055] 2. The present invention uses a mixture of three-plane decomposition features and hash coding features, so that the network can express both high-frequency features and complex details, significantly improving the accuracy and effectiveness of imaging;

[0056] 3. The present invention sends the mixed features into the attenuation field prediction network to predict the attenuation value, and uses the attenuation weight prediction network to predict the attenuation weight of each sampling pixel point on the X-ray path (taking into account that different organs and tissues show different attenuation when X-rays penetrate), and obtains the attenuation value and attenuation weight value at the sampling pixel point through these two networks; and generates the predicted projection pixel value by accumulating the attenuation value along the sampling ray;

[0057] 4. Since the present invention is mainly used in transmission scenarios (X-rays, etc.), in order to improve reconstruction accuracy and efficiency, the present invention optimizes the network by minimizing the difference between the predicted projection and the actual X-ray image. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the overall network framework of the present invention;

[0059] Figure 2 It is a visual comparison diagram of the present invention and the prior art in the embodiment;

[0060] Figure 3 A comparison chart of the ablation experiment of the embodiment on the LIDC-IDRI dataset;

[0061] Figure 4 This is an ablation experiment of different modules of the present invention in the embodiment;

[0062] Figure 5 This is a set of images finally generated by using the solution of the present invention in the embodiment. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0064] In order to enhance the CBCT reconstruction effect of sparse view x-ray projection, the technical solution of the present invention mainly includes three stages. The first stage is gradient-guided ray sampling, which uses gradient values ​​as weights to guide sampling rays so that the sampling rays are distributed as much as possible in areas with more obvious changes; the second stage is the attenuation field prediction stage, which uses tri-plane decomposition features to represent local detail information, uses hash coding to capture high-frequency information, and mixes the two features to obtain hybrid features, which are sent to the attenuation field prediction network (A-NAF) for predicting attenuation values. Considering the different attenuation rates of different organs, an attenuation weight prediction network (W-NAF) is introduced to predict the attenuation weight of each sampling pixel point on the X-ray path, and the attenuation value and attenuation weight value at the sampling pixel point are obtained through these two networks; in the third stage, the predicted projection pixel value is generated by accumulating the attenuation value along the sampling ray, and then the network is optimized by minimizing the difference between the predicted projection and the real X-ray image.

[0065] like Figure 1 As shown, the above content of the present invention specifically includes the following steps:

[0066] Step 1: For the X-ray image, gradient-guided ray sampling is performed;

[0067] First, the gradient value of each pixel i in the X-ray projection of the image is calculated; then the light corresponding to the pixel is randomly selected from the total pixels of the X-ray projection with the corresponding sampling probability; during the selection process, the sampling probability is increased in the area with higher gradient values, and the sampling probability is reduced in the area with lower gradient values;

[0068] Step 2: Perform attenuation field prediction to obtain the attenuation value and attenuation weight value at the sampling pixel point. The specific method is as follows:

[0069] Step 2.1: Based on the rays sampled in step 1, extract their three-plane decomposition features T and hash code features M respectively. H ; Then the three-plane decomposition feature T and hash code feature M H The mixed feature H is obtained by concatenation and mixing;

[0070] Step 2.2, input the obtained mixed feature H into the neural attenuation field prediction network, which includes two parallel branch network modules: the attenuation weight prediction network W-NAF and the attenuation value prediction network A-NAF;

[0071] First, the attenuation value prediction network A-NAF predicts the attenuation value μ of each sampling pixel point on the output X-ray i and feature F p , then the feature F p With the hash code feature M H The mixed feature input attenuation weight prediction network W-NAF, the attenuation weight prediction network W-NAF predicts the attenuation weight W of the output sampling pixel i , the decay weight W i and feature F p Multiply to get the final attenuation value μ i ;

[0072] Step 3: Based on the attenuation value μ of the sampled pixel point obtained in step 2 i and the attenuation weight W i , optimize the neural attenuation field prediction network.

[0073] Figure 1 The complete process of the present invention is shown in detail. The first stage is as follows Figure 1 As shown in (a) and (b), the CBCT scanner captures x-ray projections from different angles and then obtains sampling rays through a gradient-guided sampling strategy. Figure 1 As shown in (c), (d), (e) and (f), the hash coding information and the three-plane features are mixed to form a mixed feature of the network input. An attenuation value prediction network (A-NAF) and an attenuation weight prediction network (W-NAF) are also designed. The network uses a multi-hop connection strategy to accurately estimate the attenuation value. In the third stage, the network is optimized by comparing the real projection and the synthetic projection to ensure the quality of the X-ray image reconstruction.

[0074] The calculation formulas for the pixel gradient value and the sampling probability in step 1 of this embodiment are as follows:

[0075]

[0076] G x is the horizontal gradient of pixel i, G y is the vertical gradient of pixel i, G i is the total gradient value of pixel i;

[0077]

[0078] P(i) is the sampling probability of pixel i, and N is the total number of pixels on the X-ray image.

[0079] The process of obtaining the mixed feature H of the sampled pixel points in step 2 of this embodiment is as follows:

[0080] Get the hash code feature M of the sampled pixel H :

[0081] M H (p; Θ) = [I(H 1 ),…,I(H L )] T (3)

[0082] In the above formula,

[0083] is a bitwise XOR operation, H is a hash function, L is a multi-resolution voxel grid; π is a unique large prime number; P is a query point, c is a coordinate; I represents linear interpolation of feature vectors from c;

[0084] Get the three-plane decomposition feature T of the sampled pixel point, that is, project the sampled pixel point onto three planes (xy, yz, xz) to obtain the two-dimensional position information, and then use bilinear interpolation to extract features Txy, Tyz and Txz from the three planes, and then splice Txy, Tyz and Txz together to form a three-plane decomposition feature T:

[0085] T=T xy +T yz +T xz (5)

[0086] The hash code feature M H The hybrid feature H is obtained by splicing and mixing with the three-plane decomposition feature T. The expression is as follows:

[0087] H=T+M H (6)

[0088] Existing NeRF-based methods usually use shallow MLP to predict the attenuation of each sampling point on the X-ray path. These methods generate sampling points by uniformly sampling at equal distances on the X-ray path, treating all sampling points equally. However, this existing method treats all sampling points equally, ignoring that different organs and tissues exhibit different attenuations when X-rays penetrate. To address the defects of the prior art, step 2 of this embodiment adopts a neural attenuation field prediction network (reducing artifacts). The neural attenuation field prediction network uses a multi-jump method based on the MLP network (which can reduce information loss and overfitting problems) to predict the attenuation weight of each sampling pixel point, and then obtains the attenuation value when the X-ray penetrates different organs and tissues. The neural attenuation field prediction network has two parallel branches: the attenuation field prediction network (A-NAF) and the attenuation weight prediction network (W-NAF).

[0089] The attenuation value prediction network A-NAF uses a 6-layer MLP network. The mixed feature H is connected to the output of each layer of the MLP network using a multi-hop method, and together they serve as the input of the next layer of the MLP network. In the 6-layer MLP network, the activation functions of the middle layer and the last layer are ReLU function and Sigmod function respectively.

[0090] The expression is as follows:

[0091] F k =Sigmoid(ReLU(MLP(H k-1 +M H ,θ))) (7)

[0092] H k =F k ⊙M H (8)

[0093] Where F k is the output feature of the kth layer, H k is the input feature of the next layer, M H is the hash code feature of the input; ⊙ represents the connection operation.

[0094] The decay weight prediction network W-NAF uses a 6-layer MLP network. The feature Fp output by the decay value prediction network A-NAF is consistent with the hash code feature M H The concatenated mixture is used as the input of the first layer, and the position information is input into the other five-layer MLP network using a multi-hop method;

[0095] E k =ReLU(MLP(E k-1 +M H ,θ)) (9)

[0096] W k =Sigmod(Ef ) (10)

[0097] The first input feature E 1 =F p +M H , where F p Represents the output features of the decay weight prediction network (W-NAF);

[0098] E k is the output feature of the kth intermediate layer of the MLP network, E f is the output feature of the last layer of the MLP network, W k is the weight of the corresponding sampling point.

[0099] Most 3D reconstruction work in natural scenes uses volume rendering to generate rendered color images. Unlike the 3D reconstruction method in visible light scenes, the present invention is mainly applied to transmission scenes, using the attenuation accumulation method to predict the pixel values ​​on the projected image. In order to achieve better reconstruction results, it is necessary to optimize the above network. The detailed method is as follows:

[0100] First, the attenuation value μ of the X-ray upsampled pixel obtained in the network prediction phase is i and the attenuation weight W i multiply;

[0101] After obtaining the product, the pixel intensity is obtained by integrating along the X-ray direction;

[0102] μ i ′=W i ×μ i

[0103] Next, a new X-ray image is synthesized by attenuation according to Beer’s law, and the intensity of the X-ray passing through the material is reduced by exponential integration of the X-ray path;

[0104] Finally, the update is performed by minimizing the L1 loss between the real projection and the synthesized projection. The loss function L1 is defined as follows

[0105]

[0106] In the above formula, μ i ′ refers to the final attenuation value; I r (r) refers to the actual pixel value; I s (r) refers to the predicted pixel value, and B represents a batch, that is, a batchsize.

[0107] Example 1

[0108] This embodiment compares the present invention with existing NeRF-based deep learning methods, including NeRF, TensoRF, NAF, IntraTomo and SAX-NeRF.

[0109] from Figure 2 As shown in (c) and (d), it can be found that the results of NeRF and TensoRF reconstruction have a lot of blur and artifacts. These methods are usually suitable for 3D reconstruction in natural scenes, which makes it difficult for them to reconstruct high-quality 3D volume data from X-ray images. Figure 2 As shown in (b), IntraTomo uses a novel sinusoidal prediction module and a geometric optimization module to iteratively reconstruct the density field, but the result still has artifacts. Figure 2 As shown in (e), although NAF uses hash coding to capture high-frequency features to achieve high-quality 3D reconstruction, there are still some defects. Figure 2 As shown in (f), SAX-NeRF has been improved on this basis, effectively capturing local and global structural information through the local-global ray sampling strategy and the transformer-based attention mechanism on the ray to produce satisfactory 3D reconstruction results. Figure 2 As shown in (g), the present invention combines the gradient-guided ray sampling strategy and hybrid features to capture high-frequency information and local structural details, producing excellent results.

[0110] like Figure 2 As shown, the reconstruction effect of the present invention is significantly better than other methods, and is 6 to 8 times faster than the existing SAX-NeRF method, highlighting its accuracy and efficiency.

[0111] In order to prove the effectiveness of the present invention, the PSNR and SSIM indicators of each method are shown in Table 1. From Table 1, it can be seen that the technical solution of the present invention is also superior to other advanced methods.

[0112] Table 1

[0113]

[0114] Example 2

[0115] To study the effectiveness of each step of the present invention, this embodiment conducts ablation experiments on the LIDC-IDRI dataset. The baseline is a simple attenuation field method for X-ray 3D reconstruction.

[0116] For qualitative evaluation, see Figure 3 , Figure 3 (a) shows the use of the proposed attenuation field prediction network (A-NAF) to estimate the attenuation value; Figure 3 (b) in the figure indicates the use of the decay weight prediction network (W-NAF) to estimate the decay weight. Figure 3(e)Full in indicates that a+b and three planes are used to construct mixed features as input. Figure 3 As shown in (b) of , the basic method without any optimization strategy will produce obvious artifacts; Figure 3 As shown in (c), by integrating the attenuation field prediction network (A-NAF) of the present invention to predict the attenuation value (base+a), the artifacts are significantly reduced; Figure 3 As shown in (d), further enhancement is achieved by adding the attenuation weight prediction network (W-NAF) (base+a+b) of the present invention, thereby greatly improving the reconstruction quality; Figure 3 As shown in (e), finally combining the three-plane features to create a hybrid feature input produced a significant performance improvement (Full = base + a + b + c).

[0117] The quantitative evaluation is shown in Table 2. From Table 2, we can see that the reconstruction performance gradually improves with the implementation of each strategy.

[0118] Table 2

[0119]

[0120] The main comparison object of the experimental results of this embodiment is SAX-NeRF, which is significantly better than the experimental results of SAX-NeRF on most data sets, and the technical solution of the present invention is comparable to the latest SAX-NeRF method on a few data sets. When using the same GPU as SAX-NeRF, the present invention has a higher time cost. The timetable and GPU usage of each method are shown in Table 3.

[0121] Table 3

[0122]

[0123] To verify the robustness of the present invention, a series of experiments were conducted with increasing and decreasing the number of projections, ranging from 20 to 40 views. The experimental results are shown in Table 4 and compared with the baseline.

[0124] Table 4

[0125]

[0126] Example 3

[0127] This embodiment uses PyTorch to implement the technical solution of the present invention on a PC equipped with an NVIDIA GeForce GTX 3090 GPU. The Adam optimizer is used, and the initial learning rate is 1×10 -3 , gradually reduced to 1×10 -4. The network is trained with a batch size of 1024 sampling rays and 1500 epochs. 192 sampling points are sampled along each ray path. A 6-layer MLP is used to construct two branch networks: A-NAF and W-NAF, and a multi-hop connection strategy is used in the middle layer. For the three-plane decomposition features, this embodiment uses a size of 256×256×16 to represent the three-plane features. For each scene, 50 projections are used for training and 50 projections are used for testing. To ensure the fairness of the experiments, all experiments are conducted under the same settings.

[0128] In terms of data sets, this embodiment conducts a large number of experiments on the public data sets LIDC-IDRI and X3D data sets to verify the effectiveness of the present invention.

[0129] The LIDC-IDRI dataset only provides CT scan volume data, and the necessary training data is generated by digitally reconstructing radiographic images (DRR). For the generation of training data, this embodiment uses the tomography tool Tigre to generate projection data from 0 to 180 degrees at equiangular intervals. The distance from the source to the detector is set to 2000 mm, and the distance from the source to the object is set to 700 mm. The resolution of the three-dimensional CT volume data is 128×128×128, and the resolution of the projection image is 256×256.

[0130] The effect of gradient guided sampling by the present invention is as follows: Figure 4 As shown, the X-ray image generated by the present invention using Tigre is as follows Figure 5 shown.

Claims

1. A method for 3D reconstruction of sparse view X-ray images based on mixed features, characterized in that: The following steps are involved: Step 1: For the X-ray image, gradient-guided ray sampling is performed; First, the gradient value of each pixel i in the X-ray projection of the image is calculated; then the light corresponding to the pixel is randomly selected from the total pixels of the X-ray projection with the corresponding sampling probability P(i); during the selection process, the sampling probability is increased in the area with higher gradient values, and the sampling probability is reduced in the area with lower gradient values; Step 2: Predict the attenuation field to obtain the attenuation value and attenuation weight value of the sampled pixel. The specific method is as follows: Step 2.1: Based on the rays sampled in step 1, extract their three-plane decomposition features T and hash code features M respectively. H ; Then the three-plane decomposition feature T and hash code feature M H The mixed feature H is obtained by concatenation and mixing; Step 2.2: Input the obtained mixed feature H into the neural attenuation field prediction network. The neural attenuation field prediction network includes two parallel branch network modules: the attenuation weight prediction network W-NAF and the attenuation value prediction network A-NAF. First, the attenuation value prediction network A-NAF predicts the attenuation value μ of each sampling pixel point on the output X-ray. i and feature F p , then the feature F p With the hash code feature M H The mixed feature input attenuation weight prediction network W-NAF, the attenuation weight prediction network W-NAF predicts the attenuation weight W of the output sampling pixel i , the decay weight W i and attenuation value μ i Multiply to get the final attenuation value μ i '; The specific method is as follows: The neural attenuation field prediction network uses a multi-jump method based on the MLP network to predict the attenuation weight of each sampling pixel point, thereby obtaining the attenuation value when the X-ray penetrates different organs and tissues; The attenuation value prediction network A-NAF uses a 6-layer MLP network, and uses a multi-hop method to connect the hash code features to the output of each layer of the MLP network, and together as the input of the next layer of the MLP network; in the 6-layer MLP network, the activation functions of the middle layer and the last layer are ReLU function and Sigmod function respectively; The expression is as follows: F k =Sigmoid(ReLU(MLP(H k-1 +M H ,θ))) (7) H k =F k ⊙M H (8) Where F k is the output feature of the kth layer, H k is the input feature of the next layer, M H Hash encoding features for the input; ⊙ represents a connection operation; the neural attenuation field prediction network uses a multi-jump method based on the MLP network to predict the attenuation weight of each sampling pixel point, and then obtains the attenuation value when the X-ray penetrates different organs and tissues The attenuation weight prediction network W-NAF uses a 6-layer MLP network, and the feature Fp output by the attenuation value prediction network A-NAF is combined with the hash code feature M H The concatenated mixture is used as the input of the first layer, and the position information is input into the other five-layer MLP network using a multi-hop method; E k =ReLU(MLP(E k-1 +M H ,θ)) (9) W k =Sigmod(E f ) (10) The first input feature E1 = F p +M H , where F p Represents the output features of the decay weight prediction network W-NAF; E k is the output feature of the kth intermediate layer of the MLP network, E f is the output feature of the last layer of the MLP network, W k is the weight of the corresponding sampling point; Step 3: Based on the attenuation value μ of the sampled pixel point obtained in step 2 i and the attenuation weight W i , optimize the neural attenuation field prediction network.

2. The method for 3D reconstruction of sparse viewing angle X-ray images based on hybrid features according to claim 1, characterized in that: The calculation formulas for the pixel gradient value and the sampling probability in step 1 are as follows: G x is the horizontal gradient of pixel i, G y is the vertical gradient of pixel i, G i is the total gradient value of pixel i; P(i) is the sampling probability of pixel i, and N is the total number of pixels on the X-ray image.

3. The method for 3D reconstruction of sparse viewing angle X-ray images based on hybrid features according to claim 1, characterized in that: The process of obtaining the mixed feature H of the sampled pixel points in step 2 is as follows: Get the hash code feature M of the sampled pixel H : M H (p;Θ)=[I(H1),…,I(H L )] T (3) In the above formula, is a bitwise XOR operation, H() is a hash function, L is a multi-resolution voxel grid; π is a unique large prime number; p is a query point, c is a coordinate; I represents the linear interpolation of the feature vector from c; the three-plane decomposition feature T of the sampled pixel point is obtained, that is, the sampled pixel point is projected onto three planes (xy, yz, xz) to obtain the two-dimensional position information, and then the feature T is extracted from the three planes using bilinear interpolation xy ,T yz and T xz Then, T xy ,T yz and T xz Spliced ​​together to form a three-plane decomposition feature T: T=T xy +T yz +T xz (5) The hash code feature M H The hybrid feature H is obtained by splicing and mixing with the three-plane decomposition feature T. The expression is as follows: H=T+M H (6)。 4. The method for 3D reconstruction of sparse viewing angle X-ray images based on hybrid features according to claim 1, characterized in that: The detailed method of step 3 is: First, the attenuation value μ of the X-ray upsampling pixel is i and the attenuation weight W i multiply; After obtaining the product, the pixel intensity is obtained by integrating along the X-ray direction; m i ′=W i ×m i Next, a new X-ray image is synthesized by attenuation according to Beer’s law, and the intensity of the X-ray passing through the material is reduced by exponential integration of the X-ray path; Finally, the update is performed by minimizing the L1 loss between the real projection and the synthesized projection. The loss function L1 is defined as follows In the above formula, μ i ′ refers to the final attenuation value; I r (r) refers to the actual pixel value; I s (r) refers to the predicted pixel value, and B represents a batch, that is, a batchsize.

Images