Three-dimensional ct reconstruction method based on single-view x-ray

Abstract

The application discloses a three-dimensional CT reconstruction method based on a single-view X-ray film, constructs a three-dimensional CT image reconstruction neural network model, and comprises a feature extraction network based on a deep residual convolutional neural network and an implicit neural representation network based on a multilayer full connection; the feature extraction network based on the deep residual convolutional neural network is used to extract multilevel image features from the X-ray film, and fixed low-resolution volume features are generated based on the highest layer features through three-dimensional convolution; and then the target three-dimensional image is modeled as the implicit neural representation network based on the multilayer full connection, that is, for any point in a three-dimensional space, the position code of the point, the pixel image features corresponding to the point projection and the voxel features corresponding to the point are input into the implicit neural representation network, and the corresponding voxel density value is output, so that a continuous high-resolution three-dimensional image estimation task is realized. The application can reconstruct and generate effective and reliable three-dimensional CT images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical computer vision, in particular to a three-dimensional computed tomography (CT) image reconstruction method based on a single-view X-ray film. BACKGROUND

[0002] The three-dimensional image reconstruction technology based on a single-view X-ray film can effectively recover the three-dimensional geometric structure of a patient from a single X-ray film image, and can be used in dose planning in radiotherapy, or preoperative planning and intraoperative guidance in minimally invasive intervention, to help doctors obtain three-dimensional pathological information with certain clinical value under limited imaging conditions, and provide a reference for clinical diagnosis and treatment. Meanwhile, the technology can help patients reduce the risk of radiation exposure. Traditional CT image reconstruction performs full rotation projection around the target object for each layer by an X-ray device, and then recovers the X-ray absorption coefficient of each pixel in the cross-sectional image based on back projection algorithm, and the three-dimensional image is obtained by stacking multiple layers.

[0003] In current computer vision research, deep convolutional neural networks are used for three-dimensional image reconstruction research under low dose and sparse view. However, for the reconstruction task based on a single-view X-ray film, the existing deep learning method uses feature stacking or low-dimensional manifold transformation to realize cross-modal feature transformation, and then uses a three-dimensional convolutional neural network to realize three-dimensional image reconstruction. These schemes lack spatial imaging principles or semantic explanations of the organizational structure, and the quality of the generated image is low. In addition, the model proposed by the existing work has high complexity, and it is difficult to optimize the parameters, and it is also difficult to adapt to high-resolution three-dimensional image reconstruction. Therefore, the three-dimensional CT image reconstruction method proposed by the present application can effectively overcome the above problems, and provide a reliable CT image reconstruction result for a given single-view X-ray film. SUMMARY

[0004] In order to overcome the deficiencies of the above-mentioned prior art, the present application provides a three-dimensional computed tomography (CT) image reconstruction method based on a single-view X-ray film, which is a single-view three-dimensional reconstruction method based on implicit neural representation. Under the condition of a given single-view X-ray film, a high-resolution three-dimensional CT image can be generated, and a reliable CT image reconstruction result can be provided, which can provide three-dimensional structural information to assist diagnosis and treatment.

[0005] The core of the present application is that the present application proposes a CT image space consistent implicit neural representation network based on X-ray film, and solves the continuous high-resolution three-dimensional image estimation task. The three-dimensional CT image reconstruction neural network model constructed by the present application includes a feature extraction network based on a deep residual convolutional neural network and an implicit neural representation network based on a multi-layer full connection. The present application first extracts multi-scale image features from the given X-ray film through the feature extraction network based on the deep residual convolutional neural network, and generates fixed low-resolution volume features based on the highest layer features through three-dimensional convolution. Subsequently, the target three-dimensional image is modeled as an implicit neural representation network based on a multi-layer full connection, that is, for any point in the three-dimensional space, the position encoding of the point, the pixel image feature corresponding to the point projection, and the voxel feature corresponding to the point are input into the implicit neural representation network, and the corresponding voxel density value is output.

[0006] The present application provides a method for reconstructing a three-dimensional CT image based on a single-view X-ray film, which includes the following steps:

[0007] 1) Constructing a three-dimensional CT image reconstruction neural network model, including a feature extraction network based on a deep residual convolutional neural network and an implicit neural representation network based on a multi-layer full connection; and performing feature extraction;

[0008] For any point in the three-dimensional space of the three-dimensional CT image, the extracted features include:

[0009] Point projection corresponding pixel feature, denoted as F(φ(v i ));Where F is a multi-scale image feature;φ is a projection model;v i is the three-dimensional coordinates of the point;

[0010] Voxel feature, denoted as

[0011] Position encoding, denoted as z(v i );

[0012] 2) Feature splicing: splice all the extracted features in the feature channel dimension to obtain the feature vector corresponding to the point v i , denoted as:

[0013] 3) Training the feature extraction network and the implicit neural representation network through supervised learning to obtain the trained three-dimensional CT image reconstruction neural network model;

[0014] The loss function L of the model is defined as follows:

[0015]

[0016] Where h represents an implicit neural representation network composed of a multi-layer full connection and a nonlinear activation layer;c *(v i ) represents the voxel density value true value at point v i ; n represents the number of points sampled per training parameter; represents the square of the vector 2-norm;

[0017] Model training, that is, optimizing the parameters of the feature extraction network and the implicit neural representation network by minimizing the loss function, obtains a trained three-dimensional CT image reconstruction neural network model;

[0018] 4) using the trained three-dimensional CT image reconstruction neural network model for online three-dimensional image reconstruction; comprising:

[0019] 41) inputting a given single-view X-ray film into the feature extraction network in the trained three-dimensional CT image reconstruction neural network model to extract features;

[0020] 42) selecting a target resolution for reconstructing a three-dimensional image, for each voxel in the three-dimensional image grid, extracting point projection corresponding pixel features and point corresponding voxel features, and joint position coding for feature splicing;

[0021] 43) inputting the spliced features into the implicit neural representation network in the trained three-dimensional CT image reconstruction neural network model, outputting CT image predicted voxel values, and finally obtaining a reconstructed three-dimensional CT image.

[0022] Compared with the prior art, the beneficial effects of the present application are:

[0023] Using the single-view image reconstruction method of the present application, a corresponding high-quality three-dimensional CT image can be generated under the condition of only giving a single X-ray film, and the implicit neural representation effectively overcomes the difficulty of training a three-dimensional convolutional neural network for high-resolution images, and the trained model can output a three-dimensional CT image at a specified high resolution.

[0024] Unlike traditional voxel-based three-dimensional shape representation, the present application avoids the computational and storage costs brought by three-dimensional convolutional neural networks in existing deep learning-based volume reconstruction methods, and establishes a correspondence between three-dimensional voxels and local pixel features through a given projection model, with clear spatial consistency. The present application is trained end-to-end on paired data consisting of CT image data and digital reconstructed radiographs in the sagittal plane. In the test phase, given an input X-ray film and a three-dimensional spatial point query in the voxel grid at the target resolution, the model can generate an effective and reliable CT image reconstruction result. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of the method of the present application. DETAILED DESCRIPTION

[0026] The present application will be further described by examples in conjunction with the accompanying drawings, but the present application is not limited in any way by the scope of the present application.

[0027] The present application proposes a CT image space consistent implicit neural representation based on X-ray film, which solves the continuous high-resolution three-dimensional image estimation task. The three-dimensional CT image reconstruction neural network model constructed by the present application includes a feature extraction network based on a deep residual convolutional neural network and an implicit neural representation network based on a multi-layer full connection. The present application first extracts multi-scale image features from the given X-ray film through the feature extraction network based on the deep residual convolutional neural network, and generates fixed low-resolution volume features based on the highest layer features through three-dimensional convolution. Then, the target three-dimensional image is modeled as an implicit neural representation network based on a multi-layer full connection, that is, for any point in the three-dimensional space, the position encoding of the point, the pixel image feature corresponding to the point projection, and the voxel feature corresponding to the point are input into the implicit neural representation network, and the corresponding voxel density value is output.

[0028] Figure 1 The present application provides a method for reconstructing a three-dimensional CT image based on a single-view X-ray film. In specific implementation, the method includes the following steps:

[0029] 1) Construct a three-dimensional CT image reconstruction neural network model; and extract features;

[0030] The three-dimensional CT image reconstruction neural network model constructed by the present application includes a feature extraction network f based on a deep residual convolutional neural network and an implicit neural representation network h based on a multi-layer full connection.

[0031] The present application adopts a feature extraction network based on Resnet-34. In specific implementation of the present application, a three-dimensional CT image with a resolution of 128x128x128 is considered, and for the coordinates v of any point in the three-dimensional space of the three-dimensional CT image i , the features extracted for the point include the following three parts:

[0032] a) Point projection corresponding pixel feature: given a single-view X-ray film x (with a resolution of 128x128), input the feature extraction network based on the deep residual convolutional neural network, and output multi-scale image features F=f(x) at different resolutions, such as image features with resolutions of 64x64, 32x32, 16x16, 8x8, and 4x4. For the three-dimensional coordinates v i , the pixel position φ(v i ) on the two-dimensional plane is calculated according to the given projection model φ, and the point projection corresponding position φ(v i ) is extracted on the previously extracted multi-scale image features F at different resolutions. i) one-dimensional feature vector in the lower dimension, and the splicing is completed on the feature channel to obtain the pixel feature F(φ(v i )) corresponding to the point projection.

[0033] b) voxel feature: according to the multi-scale image features F extracted at different resolutions, the high-level image feature with the smallest scale is extracted, the feature is copied in the projection dimension to form a three-dimensional feature, and then the three-dimensional convolution is performed to obtain a voxel feature with a fixed resolution such as a voxel feature with a resolution of 32*32*32, and according to the three-dimensional point coordinates v i The corresponding voxel feature vector is extracted

[0034] c) position encoding: position encoding is used for the coordinate value, so that the network can more effectively learn high-frequency information, and each point corresponds to a position encoding vector:

[0035]

[0036] k=0,1,2,

[0037] wherein L is a hyperparameter, and is usually set as L=6. represents the point v i The coordinate in the kth dimension, for a three-dimensional space point v i has a coordinate dimension of 3.

[0038] 2) feature splicing:

[0039] After completing the feature extraction, all the extracted features are spliced in the feature channel dimension, and each point v i corresponds to a feature vector wherein F is a multi-scale image feature, φ is a given projection model, is a voxel feature, and z is a position encoding.

[0040] 3) training the feature extraction network f and the implicit neural representation network h through supervised learning to obtain a trained three-dimensional CT image reconstruction neural network model;

[0041] The three-dimensional CT image reconstruction neural network model is trained on the CT image and the corresponding X-ray sheet data set. The loss function L rec of the model is defined as follows:

[0042]

[0043] wherein h represents an implicit neural representation network composed of multiple fully connected layers and nonlinear activation layers, c * (v i ) represents the point v iwhere n denotes the number of sampled points per training parameter. During the training process, an adaptive voxel sampling method based on octree sampling is used to improve the training efficiency. The strategy of octree sampling is based on the density distribution of the voxel space. For any node in the octree, we calculate the information entropy H V =∑ v∈V c * (v)log c * (v), where V denotes the voxel space represented by the current node of the octree, c * (v) is the voxel density value truth of any voxel v, and log is the logarithm calculation. When the information entropy H V of the node is greater than or equal to 1, the splitting continues, otherwise the splitting stops. After the octree is established, random sampling is performed in each leaf node of the octree, and the number of sampling points in the entire three-dimensional voxel space is equal to the number of leaf nodes of the octree. The adaptive spatial sampling scheme encourages more voxels in regions with structural and density changes to be sampled, while avoiding excessive sampling of background or regions with uniform density. denotes the square of the vector 2-norm.

[0044] In the specific implementation process, the three-dimensional CT image reconstruction neural network model proposed by the present application is optimized by the adaptive sampling method above and the loss function L rec Optimizing the three-dimensional CT image reconstruction neural network model proposed by the present application, the parameters involved in the optimization include all the neural network parameters θ f contained in the feature extraction network f and the parameters θ h of the implicit neural representation network h.

[0045] 4) Using the trained three-dimensional CT image reconstruction neural network model for online three-dimensional image reconstruction:

[0046] In the online test phase, for a given single-view X-ray image input, the feature extraction network in the three-dimensional CT image reconstruction neural network model extracts multi-scale image features and fixed resolution volume features, then selects the target resolution of the reconstructed image, extracts the pixel features corresponding to the point projection and the voxel features corresponding to the point for each voxel in the three-dimensional image grid, performs feature splicing with position coding, inputs the spliced features into the implicit neural representation network, outputs the predicted voxel value, and finally obtains the reconstructed three-dimensional CT image. For example, if the target resolution of the reconstructed three-dimensional CT image is specified as 128x128x128, a three-dimensional grid with a size of 128x128x128, i.e., a total of 2097152 voxels, can be established. For each voxel, the pixel features corresponding to the point projection and the voxel features corresponding to the point are extracted according to the position and the features obtained by the feature extraction network, feature splicing is performed with position coding, and the implicit neural representation network is input to output the voxel value, and finally a gray-scale image with a size of 128x128x128, i.e., the final predicted three-dimensional CT image, is obtained.

[0047] The above embodiments are only used to illustrate the present application, but not to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions belong to the scope of the present application, and the patent protection scope of the present application should be defined by the claims.

Claims

1. A method for three-dimensional CT reconstruction based on single view X-ray images, characterized in that, The three-dimensional CT image reconstruction neural network model is constructed, including a feature extraction network based on a deep residual convolutional neural network and an implicit neural representation network based on a multi-layer full connection; the multi-scale image features are extracted from the X-ray film through the feature extraction network based on the deep residual convolutional neural network, and the fixed low-resolution volume features are generated based on the highest layer features through three-dimensional convolution; the target three-dimensional image is modeled as the implicit neural representation network based on the multi-layer full connection, the position encoding of any point in the three-dimensional space, the pixel image features corresponding to the point projection, and the voxel features corresponding to the point are input into the implicit neural representation network, and the corresponding voxel density value is output, so that the continuous high-resolution three-dimensional image estimation task is realized; the method comprises the following steps: 1) constructing a three-dimensional CT image reconstruction neural network model; multi-scale image features are extracted from the X-ray film through the feature extraction network based on the deep residual convolutional neural network; For any point in the three-dimensional space of the three-dimensional CT image, the extracted features include: The point projection corresponds to a pixel feature, denoted as F(φ(v i )); wherein F is a multi-scale image feature; φ is the projection model; v i are the three-dimensional coordinates of the points; voxel features, denoted as Position encoding, denoted as z(v i ); 2) Perform feature concatenation: concatenate all extracted features in the feature channel dimension to get point v i The corresponding feature vector is denoted as: 3) training the feature extraction network and the implicit neural representation network through supervised learning; Loss function L of the model rec is defined as follows: where h denotes the implicit neural representation network consisting of multiple fully connected and nonlinear activation layers; c * (v i ) denotes the ground truth value of the voxel density at point v i ; n denotes the number of sampled points per training parameter; denotes the squared 2-norm of a vector. The model training is to optimize the parameters of the feature extraction network and the implicit neural representation network by minimizing the loss function, so as to obtain the trained three-dimensional CT image reconstruction neural network model; 4) using the trained three-dimensional CT image reconstruction neural network model for online three-dimensional image reconstruction; comprising: 41) inputting the given single-view X-ray film into the feature extraction network in the trained three-dimensional CT image reconstruction neural network model to extract the features; 42) selecting the target resolution of the reconstructed three-dimensional image, extracting the pixel features corresponding to the point projection and the voxel features corresponding to the point for each voxel in the three-dimensional image grid, and splicing the features with the position encoding; 43) inputting the spliced features into the implicit neural representation network in the trained three-dimensional CT image reconstruction neural network model to output the voxel value of the CT image prediction, and finally obtaining the reconstructed three-dimensional CT image.

2. The method of claim 1, wherein the method is based on a single view X-ray slice. The pixel features corresponding to the point projection are extracted, which specifically comprises: Inputting the single-view X-ray film x into the feature extraction network based on the deep residual convolutional neural network to output multi-scale image features at different resolutions, denoted as F = f(x); According to the projection model φ, the pixel position φ(v i ) on the two-dimensional plane is calculated, and the one-dimensional feature vector corresponding to the position φ(v i ) of the point projection is extracted on the multi-scale image features of different resolutions extracted in advance, the splicing is completed on the feature channel, and the pixel feature F(φ(v i )) corresponding to the point projection is obtained; v i is the three-dimensional coordinates of the point.

3. The method of claim 1, wherein the single-view X-ray slice-based 3D CT reconstruction method is characterized by, The voxel feature is extracted, specifically, high-level image features obtained by the previous feature extraction network for the X-ray film are copied in the projection dimension to form three-dimensional features, and then three-dimensional convolution is performed to obtain fixed-resolution volume features and according to the three-dimensional point coordinates v i The corresponding volume feature vector is extracted 4. The method of claim 1, wherein the single-view X-ray slice-based 3D CT reconstruction method is characterized by, The position encoding vector corresponding to each point is extracted, denoted as: k＝0,1,2， where L is a hyperparameter; representing point v i coordinate in the kth dimension.

5. The method for 3D CT reconstruction based on single view X-ray image according to claim 4, characterized in that, The hyperparameter L is set to 6; the coordinate dimension of the three-dimensional space point v i is 3.

6. The three-dimensional CT reconstruction method based on single-viewpoint X-ray as described in claim 1, characterized in that, During the training process, the adaptive voxel sampling method based on octree sampling is used to improve the training efficiency; the octree sampling is based on the density distribution of the voxel space.

7. The method of claim 6, wherein the method is a method of three-dimensional CT reconstruction from a single view x-ray image. For any node in the octree, the information entropy H of the part of voxel space represented by the node is calculated V V =∑ v∈V c * (v)logc * (v);when the information entropy H V ≥1, continue to split; otherwise, stop splitting, and the octree is completed.​ 8. The method of claim 6, wherein the method is based on a single view X-ray slice. Random sampling is performed in each leaf node of the octree; the number of three-dimensional space sampling points is equal to the number of leaf nodes of the octree. ​

Images