Preoperative three-dimensional CT registration method based on single X-ray image

Abstract

The invention belongs to the field of medical imaging and artificial intelligence, and discloses a preoperative three-dimensional CT registration method based on a single X-ray image. According to the method, the CT of the current pose of the patient is recovered by using an X-ray image, and then CT registration in a three-dimensional space is performed by using the recovered CT and the preoperative CT, so that the problem of dimension mismatching in a previous registration method is solved. The invention provides a single-view-angle CT reconstruction network which is used for converting a two-dimensional X-ray image into a three-dimensional CT, and then registration of the CT generated by the network and the preoperative CT is realized by using an ICP (Inductively Coupled Plasma) algorithm. The CT registration in the three-dimensional space is realized only by using the X-ray image, the CT registration precision is effectively improved, the submillimeter-level CT registration is realized, the position of the anatomical structure of a patient can be accurately determined especially in the spine orthopedic surgery, and doctors are helped to accurately and efficiently implement the surgery.

Description

Technical Field

[0001] The present invention belongs to the field of medical imaging and artificial intelligence, and in particular relates to a preoperative three-dimensional CT registration method based on a single X-ray image. Background Art

[0002] CT (computed tomography) registration is a technique that aligns CT scans of the same patient collected at different time points or with different devices. In orthopedic surgery, doctors will make precise preoperative plans based on the patient's preoperative CT scan, such as determining the best implant path and angle for screws, simulating the removal of diseased intervertebral discs, etc. Since the patient's position during surgery is different from that when the preoperative CT scan was taken, CT registration is required. By CT registration, the preoperative CT scan is made consistent with the patient's position during surgery, which can accurately determine the positional relationship of the anatomical structure, provide accurate three-dimensional positioning information for the intraoperative navigation system, and help doctors monitor the relative position of the instrument and the target tissue in real time. Through precise navigation after registration, doctors can effectively reduce errors caused by visual occlusion or tissue displacement during surgery, avoid important structures such as bone marrow, blood vessels, and nerves, and perform surgery accurately and efficiently.

[0003] The registration method widely used in orthopedic surgery is to take two X-ray images of the patient in the frontal and lateral view during surgery, and compare them with the bone features in the simulated projection (DRR) of the preoperative CT in these two positions to calculate the rotation and offset of the patient's position, that is, the registration transformation matrix, and then apply the registration transformation matrix to the preoperative CT to complete the CT registration so that the preoperative CT is consistent with the patient's current posture. However, CT is a three-dimensional space, and X-ray images and DRR images are two-dimensional spaces. Due to the complex anatomical structure of the human body and the overlap of tissues during the projection process, two-dimensional image matching cannot fully and accurately describe the alignment relationship in three-dimensional space. The small errors caused by two-dimensional space matching will be magnified in three-dimensional space. At the same time, X-ray images are easily interfered by artifacts and noise, which will affect the extraction and matching of bone features. The registration accuracy of this method is usually at the millimeter level.

[0004] In order to improve the accuracy of CT registration, researchers have been exploring new registration methods. For example, patent CN114511597B has made improvements based on the above method. When comparing the front and side view X-ray images and DRR images, both types of images are downsampled to multiple resolution sizes, and the difference between the images is calculated from different scales. A mutual information measurement method is also proposed to evaluate the similarity of registration. Patent CN118037793B proposes to use the feature extraction capability of neural networks to take multiple X-ray images of the patient during registration, extract the anatomical structure features in the two-dimensional X-ray image and the three-dimensional CT, and correspond them in the feature space to complete the CT registration.

[0005] Different from the existing CT registration methods that compare the differences between X-ray images and DRR images in two-dimensional space or directly extract the features of two-dimensional X-ray images and match them with three-dimensional CT features, the present invention proposes a preoperative three-dimensional CT registration method based on a single X-ray image. First, an X-ray image is used to restore the three-dimensional CT of the patient's current posture, and then the restored three-dimensional CT and the preoperative CT are used to perform CT registration in three-dimensional space, which solves the problem of dimensional mismatch in the previous methods that directly use two-dimensional X-ray images and three-dimensional CT for registration, and effectively improves the accuracy of CT registration. Specifically, the present invention proposes a CT reconstruction method based on single-view projection, which converts a two-dimensional X-ray image into a three-dimensional CT by training a single-view CT reconstruction network, and then uses the ICP (iterative closest point) algorithm to register the network reconstructed CT with the preoperative CT in three-dimensional space. The present invention overcomes the problem of dimensional mismatch in previous registration methods and can improve the accuracy of registration to the sub-millimeter level. Summary of the invention

[0006] The present invention provides a preoperative 3D CT registration method based on a single X-ray image. Using a single X-ray image, CT registration can be achieved in 3D space, effectively improving the registration accuracy. The CT registration method proposed in the present invention includes a single-view CT reconstruction network and an ICP algorithm. The overall architecture is as follows: Figure 1 The specific steps are as follows:

[0007] Step 1: Build the "X-ray-CT" dataset

[0008] Obtain the patient's preoperative CT and an X-ray image taken in the embrasure position during the operation. Perform displacement and rotation of the preoperative CT in various postures, simulate projection of the preoperative CT according to the imaging parameters of the X-ray image to generate the corresponding DRR image to construct a dataset for training and verifying the single-view CT reconstruction network.

[0009] Step 2: Build a single-view CT reconstruction network

[0010] The single-view CT reconstruction network converts two-dimensional X-ray images into three-dimensional CT. The single-view CT reconstruction network includes an encoder module, a feature enhancement module, a decoder module and a feature conversion module. The network architecture is as follows: Figure 2 shown.

[0011] (1) Encoder module: The encoder module includes a channel expansion module and four downsampling modules with the same structure. It extracts features and gradually downsamples the two-dimensional X-ray image to convert it into a compact representation with a higher number of channels. The channel expansion module is mainly composed of convolutional layers and residual blocks. The input of the convolutional layer is a two-dimensional X-ray image with an input channel of 1 and an output channel of 210. The convolutional layer is used to increase the number of channels of the two-dimensional X-ray image and enhance the feature representation capability of the single-view CT reconstruction network. The residual block is divided into residual block A and residual block B, which are connected in the form of "residual block A-residual block B-residual block B-residual block A". The structure of residual block A is as follows: Figure 3 As shown, residual block A doubles the number of channels, and the structure of residual block B is as follows Figure 4 As shown in the figure, the residual block B keeps the number of channels unchanged to enhance the feature extraction capability of the single-view CT reconstruction network. The downsampling module is mainly composed of the maximum pooling layer and the residual block, which are connected in the form of "maximum pooling layer-residual block B-residual block B-residual block B". The 2×2 maximum pooling layer can halve the size of the feature map, and the residual block B can extract deep features. In the process of downsampling, the number of feature channels remains unchanged, the size of the feature map gradually decreases, and the single-view CT reconstruction network extracts deeper feature information.

[0012] (2) Feature enhancement module: The feature enhancement module is used to mine the dependencies between feature maps and enhance the single-view CT reconstruction network's understanding of global information. After multi-layer feature extraction by the encoder module, the underlying semantic features in the input feature map are obtained, which contain the structural information required for accurate reconstruction. A self-attention-based feature enhancement module is added to the bottom layer of the single-view CT reconstruction network to fully capture the long-distance dependencies in the feature map, enhance the single-view CT reconstruction network's understanding of global information, and achieve accurate CT reconstruction. The input of the feature enhancement module is the feature map output by the encoder module. The feature map is first positionally encoded and then sent to the Transformer module for processing. The Transformer module consists of a normalization layer, a multi-head attention layer, and a feedforward network layer. They are connected in the form of "normalization layer-multi-head attention layer-normalization layer-feedforward network layer". The multi-head attention layer captures the dependencies between the feature maps. After the normalization layer stabilizes the training process, the feature representation capability is finally enhanced after the feedforward network layer. Finally, the feature enhancement module outputs the enhanced feature map.

[0013] (3) Decoder module: The decoder module is used to gradually restore the spatial resolution of the enhanced feature map, and integrate the encoded features through the feature conversion module, gradually converting the enhanced feature map into a complete CT. The decoder module includes an upsampling module and a one-dimensional convolution layer. The upsampling module corresponds to the downsampling module in the encoder module and gradually restores the size of the enhanced feature map. Then, the number of channels is reduced from 840 to 128 through a 1×1 convolution layer, and finally a complete three-dimensional CT is output.

[0014] (4) Feature conversion module: The feature conversion module is used to solve the problem of inconsistent information dimensions between the projection domain and the image domain. The input of the feature conversion module is the projection domain feature map of different resolutions output by the encoder module. After the projection domain feature map is input, it first passes through a feature screening module. The feature screening module consists of a global average pooling layer, a one-dimensional convolution layer, and a Sigmoid activation function. The global average pooling layer can reduce the amount of calculation. The one-dimensional convolution layer adaptively adjusts the weights of different semantics in the projection domain feature map. The Sigmoid activation function performs channel weighting on the input projection domain features to obtain the filtered information-rich projection domain features. Then, the maximum pooling layer, 2D residual layer, and deconvolution layer are used to convert the projection domain features into CT features in the image domain. Finally, the feature conversion module outputs the semantically converted image domain feature map to the decoder module.

[0015] Step 3: Train the single-view CT reconstruction network

[0016] In the dataset established in step 1, paired DRR images and CT are used as input and supervision of the single-view CT reconstruction network, respectively. The DRR image is named x, and the corresponding CT is named q. The loss function used is composed of MSE loss and MAE loss. and is the weight parameter of the loss, as shown in formula (1.1):

[0017] (1.1)

[0018] After training to a certain number of generations, Convergence, the training of the single-view CT reconstruction network is completed. When the training is completed, an X-ray image taken in the embrasure position during the operation obtained in step 1 is input into the single-view CT reconstruction network, and its corresponding three-dimensional CT is output.

[0019] Step 4: Apply the ICP algorithm for CT registration in 3D space

[0020] Input an intraoperative X-ray image obtained in step 1 and taken in the embrasure position into the single-view CT reconstruction network trained in step 3, and output a three-dimensional CT. First, the preoperative CT and the CT generated by the single-view CT reconstruction network are converted into point cloud representations, namely the source point cloud and the target point cloud. The objective function of the ICP algorithm is shown in formula (1.2):

[0021] (1.2)

[0022] in, and are the points in the source point cloud and the target point cloud respectively, is the total number of points in the source point cloud, is the rotation matrix, is the translation vector. The ICP algorithm minimizes this objective function in each iteration. First, for each point in the source point cloud, find the nearest point in the target point cloud as the corresponding point pair. Then, based on the current corresponding point pair, calculate the optimal rotation matrix according to the principle of minimizing the objective function. and translation vectors , so that the source point cloud is aligned with the target point cloud after transformation. and Applied to the source point cloud, if the transformation change is less than the preset threshold or reaches the maximum number of iterations, the algorithm terminates. Otherwise, repeatedly search for corresponding point pairs and calculate the rotation matrix and translation vectors , and applied to the source point cloud. The final registration transformation matrix can accurately describe the translation and rotation relationship between the preoperative CT and the network-generated CT in space. The registration transformation matrix is ​​applied to the preoperative CT to complete the CT registration.

[0023] Effects and benefits of the present invention:

[0024] (1) The present invention proposes a three-dimensional CT registration method based on a single X-ray image, which realizes CT registration in three-dimensional space using only X-ray images, effectively improving the accuracy of CT registration and achieving sub-millimeter level CT registration. Especially in spinal orthopedic surgery, it can accurately determine the position of the patient's anatomical structure and help doctors perform surgery accurately and efficiently.

[0025] (2) The present invention only requires one X-ray to achieve preoperative CT registration. Compared with the previous CT registration method that requires at least two X-rays, one in the frontal view and the other in the lateral view, the operation process is simplified, which is of great significance for some scenes in orthopedic surgery where it is difficult to take X-rays in the lateral view. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is an overall architecture diagram of the single X-ray and preoperative CT registration method of the present invention.

[0027] Figure 2 It is an architecture diagram of the single-view CT reconstruction network of the present invention.

[0028] Figure 3 It is an architecture diagram of the residual block A in the single-view CT reconstruction network of the present invention.

[0029] Figure 4 It is an architecture diagram of the residual block B in the single-view CT reconstruction network of the present invention. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0031] First, under the premise of ensuring the privacy and rights of the patient, the cooperating hospital obtains the patient's preoperative CT and an emmetropia X-ray image taken during the operation. According to step 1, a dataset is constructed to train and verify the single-view CT reconstruction network.

[0032] Then, follow step 2 and combine Figure 2 Building a single-view CT reconstruction network: First, build an encoder module, input a two-dimensional X-ray image, expand the number of channels to 210 through the convolution layer of the channel expansion module, and extract rich features through multiple residual blocks cascaded. Subsequently, the feature map enters four downsampling modules with the same structure in turn, and the size of the feature map is gradually reduced from 512×512 to 32×32. Secondly, build a feature enhancement module after the encoder module, capture long-distance dependencies through the self-attention mechanism, and combine position encoding and multi-head attention to improve the modeling ability of global information. Next, build a decoder module, symmetrically design based on the downsampling structure of the encoder module, replace the downsampling layer with the upsampling layer, gradually restore the feature map resolution to 512×512, and finally reduce the number of channels to 128 through 1×1 convolution to output high-quality CT. Finally, build a feature conversion module, which connects the encoder module and the decoder module to solve the semantic difference problem between the projection domain and the image domain.

[0033] Secondly, train the single-view CT reconstruction network according to step 3. Specific details: 300 paired DRR images and CT data are selected as the training set, and 10 cases are selected as the validation set. According to the process of step 3, the training takes 7 hours to complete under the RTX4090 GPU.

[0034] After completing the training and verification of the network, input an orthoscopic X-ray image taken during surgery into the network according to step 3 to obtain a 3D CT. Then, according to step 4, convert both the CT generated by the network and the preoperative CT into point clouds, use the ICP algorithm to calculate the registration transformation matrix, and apply the registration change matrix to the preoperative CT to complete the CT registration.

Claims

1. A preoperative 3D CT registration method based on a single X-ray image, characterized in that: This preoperative 3D CT registration method based on a single X-ray image includes two parts: a single-view CT reconstruction network and an ICP algorithm; Here are the steps: Step 1: Build the "X-ray-CT" dataset Obtain the patient's preoperative CT and an X-ray image taken in the embrasure position during the operation; perform displacement and rotation of the preoperative CT in various postures, simulate projection of the preoperative CT according to the imaging parameters of the X-ray image to generate the corresponding DRR image to construct a data set for training and verifying the single-view CT reconstruction network; Step 2: Build a single-view CT reconstruction network The single-view CT reconstruction network converts a two-dimensional X-ray image into a three-dimensional CT. The single-view CT reconstruction network includes an encoder module, a feature enhancement module, a decoder module, and a feature conversion module. (1) Encoder module: The encoder module includes a channel expansion module and four downsampling modules with the same structure. It extracts features and gradually downsamples the two-dimensional X-ray image to convert it into a compact representation with a higher number of channels. The channel expansion module is mainly composed of a convolutional layer and a residual block. The input of the convolutional layer is a two-dimensional X-ray image with 1 input channel and 210 output channels. The convolutional layer is used to increase the number of channels of the two-dimensional X-ray image and enhance the feature representation capability of the single-view CT reconstruction network. The residual block is divided into residual block A and residual block B. According to the "residual block A-residual The residual block A doubles the number of channels, and the residual block B keeps the number of channels unchanged to enhance the feature extraction capability of the single-view CT reconstruction network. The downsampling module is mainly composed of a maximum pooling layer and a residual block, which are connected in the form of "maximum pooling layer-residual block B-residual block B-residual block B". The 2×2 maximum pooling layer halves the size of the feature map, and the residual block B can extract deep features. In the process of downsampling, the number of feature channels remains unchanged, and the size of the feature map gradually decreases. The single-view CT reconstruction network extracts deeper feature information. (2) Feature enhancement module: The feature enhancement module is used to mine the dependencies between feature maps and enhance the single-view CT reconstruction network's understanding of global information. After multi-layer feature extraction by the encoder module, the underlying semantic features of the input feature map are obtained, which contain the structural information required for accurate reconstruction. A self-attention-based feature enhancement module is added to the bottom layer of the single-view CT reconstruction network to fully capture the long-distance dependencies in the feature map, enhance the single-view CT reconstruction network's understanding of global information, and achieve accurate CT reconstruction. The input of the feature enhancement module is the feature map output by the encoder module. The feature map is first positionally encoded and then sent to the Transformer module for processing. The Transformer module consists of a normalization layer, a multi-head attention layer, and a feedforward network layer. They are connected in the form of "normalization layer-multi-head attention layer-normalization layer-feedforward network layer". The multi-head attention layer captures the dependencies between the feature maps. After the normalization layer stabilizes the training process, the feature representation capability is finally enhanced after the feedforward network layer. The final feature enhancement module outputs the enhanced feature map; (3) Decoder module: The decoder module is used to gradually restore the spatial resolution of the enhanced feature map and integrate the encoded features through the feature conversion module to gradually convert the enhanced feature map into a complete CT. The decoder module includes an upsampling module and a one-dimensional convolutional layer. The upsampling module corresponds to the downsampling module in the encoder module and gradually restores the size of the enhanced feature map. Then, the number of channels is reduced from 840 to 128 through a 1×1 convolutional layer, and finally a complete three-dimensional CT is output. (4) Feature conversion module: The feature conversion module is used to solve the problem of inconsistent information dimensions between the projection domain and the image domain. The input of the feature conversion module is the projection domain feature map of different resolutions output by the encoder module. After the projection domain feature map is input, it first passes through a feature screening module. The feature screening module consists of a global average pooling layer, a one-dimensional convolution layer, and a Sigmoid activation function. The global average pooling layer is used to reduce the amount of calculation. The one-dimensional convolution layer adaptively adjusts the weights of different semantics in the projection domain feature map. The Sigmoid activation function performs channel weighting on the input projection domain feature map to obtain a filtered information-rich projection domain feature map. Then, the maximum pooling layer, the 2D residual layer, and the deconvolution layer are used to convert the filtered information-rich projection domain feature map into a CT feature map in the image domain. Finally, the feature conversion module outputs the CT feature map of the image domain after semantic conversion to the decoder module; Step 3: Train the single-view CT reconstruction network In the dataset established in step 1, paired DRR images and CT are used as input and supervision of the single-view CT reconstruction network, respectively. The DRR image is named x, and the corresponding CT is named q. The loss function used is composed of MSE loss and MAE loss. and is the weight parameter of the loss, as shown in formula (1.1): (1.1) After training to a certain number of generations, Convergence, the single-view CT reconstruction network training is completed; when the training is completed, an X-ray image taken in the embrasure position during the operation obtained in step 1 is input into the single-view CT reconstruction network, and its corresponding three-dimensional CT is output; Step 4: Apply the ICP algorithm for CT registration in 3D space Input an intraoperative X-ray image obtained in step 1 and taken in the embrasure position into the single-view CT reconstruction network trained in step 3, and output a three-dimensional CT. First, the preoperative CT and the CT generated by the single-view CT reconstruction network are converted into point cloud representations, namely, source point cloud and target point cloud. The objective function of the ICP algorithm is shown in formula (1.2): (1.2) in, and are the points in the source point cloud and the target point cloud respectively, is the total number of points in the source point cloud, is the rotation matrix, is the translation vector; the ICP algorithm minimizes the objective function in each iteration; first, for each point in the source point cloud, find the nearest point in the target point cloud as the corresponding point pair; Then, based on the current corresponding point pair, the optimal rotation matrix is ​​calculated according to the principle of minimizing the objective function. and translation vectors , so that the source point cloud is aligned with the target point cloud after transformation; the calculated and Applied to the source point cloud, if the transformation change is less than the preset threshold or reaches the maximum number of iterations, the algorithm terminates; otherwise, repeatedly search for corresponding point pairs and calculate the rotation matrix and translation vectors , and applied to the source point cloud step; the final registration transformation matrix can accurately describe the translation and rotation relationship between the preoperative CT and the network-generated CT in space, and the registration transformation matrix is ​​applied to the preoperative CT to complete the CT registration.

Images