2D / 3D coronary artery automatic registration method, system and medium

Abstract

The application provides a 2D / 3D coronary artery automatic registration method, system and medium, comprising: carrying out blood vessel structure segmentation and center line extraction on a patient's preoperative three-dimensional CTA image and corresponding intraoperative two-dimensional XCA image respectively, taking the center line containing blood vessel geometric topological structure information as a blood vessel feature; decomposing the corresponding 2D / 3D blood vessel center line into starting point information and shape information, the starting point information passing through a starting point feature extraction branch, and the shape information passing through a shape feature extraction branch, and carrying out feature extraction and feature fusion by two information flow channels; inputting the double-flow features into a deep neural network, predicting the angle deflection between two adjacent points as a displacement field, deforming the three-dimensional coronary artery structure obtained preoperatively according to the displacement field, and projecting the three-dimensional coronary artery structure onto the intraoperative two-dimensional image plane to complete registration. The application firstly uses a deep neural network to carry out 2D / 3D coronary artery registration, is stable and reliable, has high operation efficiency, and is accurate in result.

Description

Technical Field

[0001] The present invention relates to the field of medicine, and more specifically, to a 2D / 3D coronary artery automatic registration method, system, and medium. More specifically, it relates to a 2D / 3D coronary artery automatic registration method based on a deep neural network for dual-stream feature extraction and angular displacement prediction. Background Art

[0002] In PCI (percutaneous coronary intervention) surgery, intraoperative XCA angiography images are often used as guidance images. They have the advantages of being dynamic and real-time, artifact-free, and providing clear anatomical structures. However, they are essentially two-dimensional projection images, which lose spatial three-dimensional structural information and have severe tissue overlap. Therefore, it is difficult to intuitively guide doctors in performing a series of complex operations such as guidewire advancement and stent placement. Preoperative CTA, as a three-dimensional reconstruction imaging method, can present the overall structure and shape of the coronary arteries in three dimensions. Therefore, incorporating the three-dimensional information contained in preoperative CTA images into the surgery can make up for the inherent shortcomings of XCA angiography, providing more intuitive, complete, and real-time three-dimensional coronary structure images as guidance for PCI surgery, greatly improving the efficiency and accuracy of the surgery.

[0003] 2D / 3D coronary artery image registration is of great clinical significance. A literature search revealed a range of published techniques. In their paper "Deformable 2D-3D registration of vascular structures in a one-view scenario," Groher et al. used the Gold and Rangarajan method to establish soft correspondences between point pairs. They used the Euclidean distance as the optimization objective and employed the BFGS optimizer to control the optimization process and avoid falling into a global minimum.

[0004] In their paper "Simultaneous correspondence and non-rigid 3D reconstruction of the coronary tree from single X-ray images," Serradell et al. first sampled points in both three-dimensional and two-dimensional space using k-clustering. They then established correspondences between the points using distance and direction information, and then used the Hungarian algorithm for matching. They then leveraged weak prior knowledge to add Gaussian perturbations to the existing three-dimensional structure, generating a series of basis samples. This transformed the high-dimensional deformation field predictions into weighted predictions of the basis samples, reducing the number of prediction parameters and accelerating the iterative optimization process.

[0005] In their paper "A 3D / 2D registration of the coronary arteries based on tree topology consistency matching," Liu et al. considered the relationship between the coronary artery tree structure and vascular hierarchy. While removing false bifurcations, they rigorously aligned the upstream and downstream branches of each true bifurcation, thereby constructing a global match to avoid optimization errors. Thin-plate spline deformation was used to control motion and optimize the Euclidean distance between corresponding points.

[0006] In their paper "Oriented Gaussian mixture models for nonrigid 2D / 3D coronary artery registration," Baka et al. treat the centerline point sets of 3D and 2D space as two independent Gaussian distribution models. To register the two, they aim to make the two Gaussian distributions as close as possible. This allows them to construct a Gaussian mixture model and optimize the parameters to minimize the L2 distance. These methods, all based on iterative optimization, are complex, require a large number of prediction parameters, and are slow to optimize. They are also prone to non-convergence and local minima, and their robustness remains to be improved. Currently, there is still no method that achieves both effective and fast registration. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the present invention aims to provide a 2D / 3D coronary artery automatic registration method, system and medium.

[0008] According to the present invention, a 2D / 3D coronary artery automatic registration method is provided, comprising the following steps:

[0009] Preprocessing steps: Vascular structure segmentation and centerline extraction are performed on the patient's preoperative 3D CTA images and the corresponding intraoperative 2D XCA images, respectively. The centerline containing the vascular geometric topology information is used as the vascular feature;

[0010] Dual-stream feature extraction steps: The 3D vascular centerline structure corresponding to the 3D CTA image and the 2D vascular centerline structure corresponding to the 2D XCA image are decomposed into starting point information and morphological information. The starting point information passes through the starting point feature extraction branch, and the morphological information passes through the morphological feature extraction branch. Feature extraction and feature fusion are performed through these two information flow channels to form dual-stream features.

[0011] Angular displacement prediction step: The dual-stream features are input into the deep neural network to predict the angular deflection between two adjacent points as the displacement field. Based on this, the three-dimensional coronary artery structure acquired before surgery is deformed and modeled, and then projected onto the intraoperative two-dimensional imaging plane to complete the registration.

[0012] Preferably, the pre-processing step comprises:

[0013] CTA image training steps: Randomly select CTA images, perform voxel-by-voxel vessel annotation, and train a convolutional neural network (3D U-net) for 3D coronary artery segmentation based on the original image and corresponding labels;

[0014] CTA processing steps: Use the trained 3D U-net to segment the CTA of the new data, use the morphological method to extract the centerline, and use the smoothing algorithm to further smooth the extracted CTA centerline;

[0015] XCA image training steps: Randomly select XCA images and perform pixel-by-pixel vessel annotation. Based on the original image and the corresponding labels, a convolutional neural network (2D U-net) for 2D coronary artery segmentation is trained;

[0016] XCA processing steps: Use the trained 2D U-net to segment the XCA of the new data, use the morphological method to extract the centerline, and use the smoothing algorithm to further smooth the extracted XCA centerline.

[0017] Preferably, the dual-stream feature extraction step includes:

[0018] CTA coordinate representation steps: The centerline of the CTA vascular segment is represented as a set of continuous points C = {a1,…,a n}, where a n =(x n ,y n ,z n ) is represented by the Euclidean coordinates of a point in three-dimensional space; it can be decomposed into the starting point information a1 and the morphological information {a2-a1,…,a n -a n-1}, where the shape is represented by the set of offsets between two adjacent points;

[0019] XCA coordinate representation steps: The center line of the XCA blood vessel segment is represented as a set of continuous points X = {b1,…,b n}, where b n =(x n ,y n ) is represented by the Euclidean coordinates of the point in two-dimensional space; it can be decomposed into the starting point information b1 and the morphological information {b2-b1,…,b n -b n-1}, where the shape is represented by the set of offsets between two adjacent points;

[0020] CTA feature fusion steps: Input the CTA centerline starting point information set with a dimension of N×3×1 into the CTA starting point feature extraction branch, and use a series of one-dimensional convolution kernels of size 1 for convolution to obtain a CTA starting point feature with a dimension of N×m×1; Input the CTA centerline morphological information set with a dimension of N×3×(n-1) into the CTA morphological feature extraction branch, and use a series of one-dimensional convolution kernels of size 3 for convolution to obtain a CTA morphological feature with a dimension of N×m×(n-1); Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion;

[0021] XCA feature fusion steps: Input the XCA centerline starting point information set with a dimension of N×2×1 into the XCA starting point feature extraction branch, and use a series of one-dimensional convolution kernels of size 1 for convolution to obtain an XCA starting point feature with a dimension of N×m×1; Input the XCA centerline morphological information set with a dimension of N×2×(n-1) into the XCA morphological feature extraction branch, and use a series of one-dimensional convolution kernels of size 3 for convolution to obtain an XCA morphological feature with a dimension of N×m×(n-1); Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion;

[0022] Splicing and fusion step: Splice the CTA features and XCA features in the channel dimension to obtain a cross-modal fusion feature with a dimension of N×2m×(n-1).

[0023] Preferably, the angular displacement prediction step includes:

[0024] Angular offset output step: The cross-modal fusion feature that combines CTA and XCA information is input into the angular displacement prediction network. The angular displacement prediction network is based on a series of one-dimensional convolution kernels of size 3, divided into two parts: downsampling and upsampling. It uses a skip connection structure and finally outputs the angular offset of a point on the centerline relative to the previous point.

[0025] Morphological restoration step: Based on the output angle offset, the deformed three-dimensional coronary artery morphology is restored according to the following formula:

[0026]

[0027] Among them, θ and is the change of two angles in space, and r is the distance between two adjacent points. During the training phase, a Gaussian mixture model is established to measure the similarity of two point sets:

[0028]

[0029] Among them, L GMM Represents error, PCTA represents the CTA point set, P XCA represents the XCA point set, and σ represents the smoothing parameter.

[0030] According to the present invention, a 2D / 3D coronary artery automatic registration system is provided, comprising the following modules:

[0031] Preprocessing module: vascular structure segmentation and centerline extraction are performed on the patient's preoperative 3D CTA images and the corresponding intraoperative 2D XCA images, respectively. The centerline containing the vascular geometric topology information is used as the vascular feature;

[0032] Dual-stream feature extraction module: The 3D vascular centerline structure corresponding to the 3D CTA image and the 2D vascular centerline structure corresponding to the 2D XCA image are decomposed into starting point information and morphological information. The starting point information passes through the starting point feature extraction branch, and the morphological information passes through the morphological feature extraction branch. Feature extraction and feature fusion are performed through these two information flow channels to form dual-stream features.

[0033] Angular displacement prediction module: The dual-stream features are input into a deep neural network to predict the angular deflection between two adjacent points as the displacement field. Based on this, the three-dimensional coronary artery structure acquired before surgery is deformed and modeled, and then projected onto the intraoperative two-dimensional imaging plane to complete the registration.

[0034] Preferably, the preprocessing module includes:

[0035] CTA image training module: Randomly select CTA images, perform voxel-by-voxel vessel annotation, and train a convolutional neural network (3D U-net) for 3D coronary artery segmentation based on the original image and corresponding labels;

[0036] CTA processing module: Use the trained 3D U-net to segment the CTA of new data, use morphological methods to extract the centerline, and use the smoothing algorithm to further smooth the extracted CTA centerline;

[0037] XCA image training module: randomly selects XCA images, performs pixel-by-pixel vessel annotation, and trains a convolutional neural network (2D U-net) for two-dimensional coronary artery segmentation based on the original image and corresponding labels;

[0038] XCA processing module: Use the trained 2D U-net to segment the XCA of new data, use the morphological method to extract the centerline, and use the smoothing algorithm to further smooth the extracted XCA centerline.

[0039] Preferably, the dual-stream feature extraction module includes:

[0040] CTA coordinate representation module: The centerline of a CTA vascular segment is represented as a set of continuous points C = {a1,…,a n}, where a n =(x n ,y n ,z n ) is represented by the Euclidean coordinates of a point in three-dimensional space; it can be decomposed into the starting point information a1 and the morphological information {a2-a1,…,a n -a n-1}, where the shape is represented by the set of offsets between two adjacent points;

[0041] XCA coordinate representation module: The centerline of the XCA blood vessel segment is represented as a set of continuous points X = {b1,…,b n}, where b n =(x n ,y n ) is represented by the Euclidean coordinates of the point in two-dimensional space; it can be decomposed into the starting point information -1 and the morphological information {b2-b1,…,b n -b n-1}, where the shape is represented by the set of offsets between two adjacent points;

[0042] CTA feature fusion module: The CTA centerline starting point information set with a dimension of N×3×1 is input into the CTA starting point feature extraction branch, and convolution is performed using a series of one-dimensional convolution kernels of size 1 to obtain a CTA starting point feature with a dimension of N×m×1. The CTA centerline morphological information set with a dimension of N×3×(n-1) is input into the CTA morphological feature extraction branch, and convolution is performed using a series of one-dimensional convolution kernels of size 3 to obtain a CTA morphological feature with a dimension of N×m×(n-1). Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion.

[0043] XCA feature fusion module: The XCA centerline starting point information set with a dimension of N×2×1 is input into the XCA starting point feature extraction branch, and a series of one-dimensional convolution kernels of size 1 are used for convolution to obtain an XCA starting point feature with a dimension of N×m×1. The XCA centerline morphological information set with a dimension of N×2×(n-1) is input into the XCA morphological feature extraction branch, and a series of one-dimensional convolution kernels of size 3 are used for convolution to obtain an XCA morphological feature with a dimension of N×m×(n-1). Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion.

[0044] Splicing and fusion module: Splice the CTA features and XCA features in the channel dimension to obtain a cross-modal fusion feature with a dimension of N×2m×(n-1).

[0045] Preferably, the angular displacement prediction module includes:

[0046] Angle offset output module: This module inputs the cross-modal fusion features that combine CTA and XCA information into the angular displacement prediction network. The angular displacement prediction network is based on a series of one-dimensional convolution kernels of size 3, divided into downsampling and upsampling parts, and uses a skip connection structure to output the angular offset of a point on the centerline relative to the previous point.

[0047] Morphological Restoration Module: Based on the output angle offset, the deformed 3D coronary artery morphology is restored according to the following formula:

[0048]

[0049] Among them, θ and is the change of two angles in space, and r is the distance between two adjacent points. During the training phase, a Gaussian mixture model is established to measure the similarity of two point sets:

[0050]

[0051] Among them, L GMM Represents error, P CTA represents the CTA point set, P XCA represents the XCA point set, and σ represents the smoothing parameter.

[0052] According to the present invention, a computer-readable storage medium storing a computer program is provided, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention has the advantages of stability, reliability, high computational efficiency and accurate results.

[0055] 2. The present invention utilizes a 2D / 3D coronary artery registration network based on deep learning to perform dual-stream feature extraction of morphological information and starting point information, enriching feature information while simultaneously predicting angular displacement.

[0056] 3. The present invention abandons the complex explicit length constraints in traditional optimization functions, greatly simplifies the network training process, accelerates convergence, makes the registration network model more robust, and realizes automatic and rapid registration of coronary artery structures, laying a good foundation for PCI surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0058] Figure 1 Flowchart of the 2D / 3D coronary artery automatic registration method based on dual-stream feature extraction and angular displacement prediction deep neural network.

[0059] Figure 2 Figure 2: The structural diagram of the 2D / 3D coronary artery registration network.

[0060] Figure 3 This is the effect diagram of the registration. DETAILED DESCRIPTION

[0061] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0062] like Figures 1 to 3 As shown, the present invention addresses the deficiencies in the prior art and provides a 2D / 3D coronary artery automatic registration method based on a dual-stream feature extraction and angular displacement prediction deep neural network, comprising the following steps:

[0063] Step 1: Preprocessing: Vascular structure segmentation and centerline extraction are performed on the patient's preoperative 3D CTA (CT coronary angiography) image and the corresponding intraoperative 2D X-ray coronary angiography (XCA) image. The centerline containing the vascular geometric topology information is used as the vascular feature.

[0064] Step 2: Dual-stream feature extraction: The corresponding 2D / 3D vascular centerline structure is decomposed into starting point information and morphological information. The information is then passed through the starting point feature extraction branch and the morphological feature extraction branch respectively, and feature extraction and feature fusion are performed by the two information flow channels.

[0065] Step 3: Angular displacement prediction: The dual-stream features are input into a deep neural network to predict the angular deflection between two adjacent points as the displacement field. Based on this, the 3D coronary artery structure acquired before surgery is deformed and modeled, and then projected onto the intraoperative 2D imaging plane to complete the registration.

[0066] Furthermore, the step 1 includes the following steps:

[0067] Step 11: Randomly select some CT coronary angiography images and perform voxel-by-voxel vessel annotation. Based on the original images and corresponding labels, train a convolutional neural network (3D U-net) for 3D coronary artery segmentation.

[0068] Step 12: Use the trained coronary artery segmentation network 3D U-net to segment the CTA of the new data. Based on this, use the morphological method to extract the centerline, and use the smoothing algorithm to further smooth the extracted CTA centerline.

[0069] Step 13: Randomly select some X-ray coronary angiography images and perform pixel-by-pixel vessel annotation. Based on the original images and the corresponding labels, train a convolutional neural network (2D U-net) for two-dimensional coronary artery segmentation.

[0070] Step 14: Use the trained coronary artery segmentation network 2D U-net to segment the XCA of the new data. On this basis, use the morphological method to extract the centerline, and use the smoothing algorithm to further smooth the extracted XCA centerline.

[0071] Furthermore, the step 2 includes the following steps:

[0072] Step 21: The centerline of the CTA vessel segment can be represented as a set of continuous points C = {a1,…,a n}, where a n =(x n ,y n ,z n ) is represented by the Euclidean coordinates of a point in three-dimensional space. It can be decomposed into the starting point information a1 and the shape information {a2-a1,…,a n -a n-1}, where the shape is represented by the set of offsets between two adjacent points;

[0073] Step 22: The centerline of the XCA blood vessel segment can be represented as a set of continuous points X = {b1,…,b n}, where b=(x n ,y n ) is represented by the Euclidean coordinates of a point in two-dimensional space. It can be decomposed into the starting point information b1 and the shape information {b2-b1,…,b n -b n-1}, where the shape is represented by the set of offsets between two adjacent points;

[0074] Step 23: Input the CTA centerline starting point information set of dimension N×3×1 into the CTA starting point feature extraction branch and perform convolution using a series of one-dimensional convolution kernels of size 1 to obtain CTA starting point features of dimension N×m×1. Input the CTA centerline morphological information set of dimension N×3×(n-1) into the CTA morphological feature extraction branch and perform convolution using a series of one-dimensional convolution kernels of size 3 to obtain CTA morphological features of dimension N×m×(n-1). Finally, the N×m×1 starting point features are replicated along the last dimension to N×m×(n-1) and added to the morphological features for feature fusion.

[0075] Step 24: Input the XCA centerline starting point information set of dimension N×2×1 into the XCA starting point feature extraction branch, and convolve it using a series of one-dimensional convolution kernels of size 1 to obtain an XCA starting point feature of dimension N×m×1. Input the XCA centerline morphological information set of dimension N×2×(n-1) into the XCA morphological feature extraction branch, and convolve it using a series of one-dimensional convolution kernels of size 3 to obtain an XCA morphological feature of dimension N×m×(n-1). Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion.

[0076] Step 25: Concatenate the CTA features and the XCA features in the channel dimension to obtain a cross-modal fusion feature with a dimension of N×2m×(n-1).

[0077] Furthermore, the step 3 includes the following steps:

[0078] Step 31: Input the cross-modal fusion features that combine CTA and XCA information into the angular displacement prediction network. The angular displacement prediction network is based on a series of one-dimensional convolution kernels of size 3, divided into two parts: downsampling and upsampling. It uses a skip connection structure and finally outputs the angular offset of a point on the centerline relative to the previous point.

[0079] Step 32: Based on the output angle offset, restore the deformed three-dimensional coronary artery shape according to the following formula:

[0080]

[0081] Among them, θ and is the change in two angles in space, and r is the distance between two adjacent points. During the training phase, a Gaussian mixture model is established to measure the similarity of two point sets:

[0082]

[0083] Among them, L GMM Represents error, P CTArepresents the CTA point set, P XCA represents the XCA point set, and σ represents the smoothing parameter.

[0084] The present invention also provides a 2D / 3D coronary artery automatic registration system, comprising the following modules:

[0085] Preprocessing module: vascular structure segmentation and centerline extraction are performed on the patient's preoperative 3D CTA images and the corresponding intraoperative 2D XCA images, respectively. The centerline containing the vascular geometric topology information is used as the vascular feature;

[0086] Dual-stream feature extraction module: The 3D vascular centerline structure corresponding to the 3D CTA image and the 2D vascular centerline structure corresponding to the 2D XCA image are decomposed into starting point information and morphological information. The starting point information passes through the starting point feature extraction branch, and the morphological information passes through the morphological feature extraction branch. Feature extraction and feature fusion are performed through these two information flow channels to form dual-stream features.

[0087] Angular displacement prediction module: The dual-stream features are input into a deep neural network to predict the angular deflection between two adjacent points as the displacement field. Based on this, the three-dimensional coronary artery structure acquired before surgery is deformed and modeled, and then projected onto the intraoperative two-dimensional imaging plane to complete the registration.

[0088] The pre-processing module comprises:

[0089] CTA image training module: Randomly select CTA images, perform voxel-by-voxel vessel annotation, and train a convolutional neural network (3D U-net) for 3D coronary artery segmentation based on the original image and corresponding labels;

[0090] CTA processing module: Use the trained 3D U-net to segment the CTA of new data, use morphological methods to extract the centerline, and use the smoothing algorithm to further smooth the extracted CTA centerline;

[0091] XCA image training module: randomly selects XCA images, performs pixel-by-pixel vessel annotation, and trains a convolutional neural network (2D U-net) for two-dimensional coronary artery segmentation based on the original image and corresponding labels;

[0092] XCA processing module: Use the trained 2D U-net to segment the XCA of new data, use the morphological method to extract the centerline, and use the smoothing algorithm to further smooth the extracted XCA centerline.

[0093] The dual-stream feature extraction module includes:

[0094] CTA coordinate representation module: The centerline of a CTA vascular segment is represented as a set of continuous points C = {a1,…,a n}, where a n =(x n ,y n ,z n ) is represented by the Euclidean coordinates of a point in three-dimensional space; it can be decomposed into the starting point information a1 and the morphological information {a2-a1,…,a n -a n-1}, where the shape is represented by the set of offsets between two adjacent points;

[0095] XCA coordinate representation module: The centerline of the XCA blood vessel segment is represented as a set of continuous points X = {b1,…,b n}, where b n =(x n ,y n ) is represented by the Euclidean coordinates of the point in two-dimensional space; it can be decomposed into the starting point information b1 and the morphological information {b2-b1,…,b n -b n-1}, where the shape is represented by the set of offsets between two adjacent points;

[0096] CTA feature fusion module: The CTA centerline starting point information set with a dimension of N×3×1 is input into the CTA starting point feature extraction branch, and convolution is performed using a series of one-dimensional convolution kernels of size 1 to obtain a CTA starting point feature with a dimension of N×m×1. The CTA centerline morphological information set with a dimension of N×3×(n-1) is input into the CTA morphological feature extraction branch, and convolution is performed using a series of one-dimensional convolution kernels of size 3 to obtain a CTA morphological feature with a dimension of N×m×(n-1). Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion.

[0097] XCA feature fusion module: The XCA centerline starting point information set with a dimension of N×2×1 is input into the XCA starting point feature extraction branch, and a series of one-dimensional convolution kernels of size 1 are used for convolution to obtain an XCA starting point feature with a dimension of N×m×1. The XCA centerline morphological information set with a dimension of N×2×(n-1) is input into the XCA morphological feature extraction branch, and a series of one-dimensional convolution kernels of size 3 are used for convolution to obtain an XCA morphological feature with a dimension of N×m×(n-1). Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion.

[0098] Splicing and fusion module: Splice the CTA features and XCA features in the channel dimension to obtain a cross-modal fusion feature with a dimension of N×2m×(n-1).

[0099] The angular displacement prediction module includes:

[0100] Angle offset output module: This module inputs the cross-modal fusion features that combine CTA and XCA information into the angular displacement prediction network. The angular displacement prediction network is based on a series of one-dimensional convolution kernels of size 3, divided into downsampling and upsampling parts, and uses a skip connection structure to output the angular offset of a point on the centerline relative to the previous point.

[0101] Morphological Restoration Module: Based on the output angle offset, the deformed 3D coronary artery morphology is restored according to the following formula:

[0102]

[0103] Among them, θ and is the change of two angles in space, and r is the distance between two adjacent points. During the training phase, a Gaussian mixture model is established to measure the similarity of two point sets:

[0104]

[0105] Among them, L GMM Represents error, P GTA represents the CTA point set, P XCA represents the XCA point set, and σ represents the smoothing parameter.

[0106] This example was implemented on a computer with a 64-bit Linux system, 64GB of RAM, an Intel(R) Xeon E5-2630 v3 CPU, and an Nvidia Titan X Pascal GPU. The entire method is based on the PyTorch deep learning framework and implemented in the Python programming language. The collected clinical data is loaded into the network model to complete the registration process. The renderings show two examples (black represents the projection of the CTA centerline to be registered, and gray represents the target XCA centerline. The first row shows the initial state before registration, and the second row shows the final result after registration). This method can quickly and efficiently achieve registration and alignment of CTA and XCA structures, with an average error of 0.8 mm.

[0107] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A 2D / 3D coronary artery automatic registration method, characterized in that: The steps include: Preprocessing steps: Vascular structure segmentation and centerline extraction are performed on the patient's preoperative 3D CTA images and the corresponding intraoperative 2D XCA images, respectively. The centerline containing the vascular geometric topology information is used as the vascular feature; Dual-stream feature extraction steps: The 3D vascular centerline structure corresponding to the 3D CTA image and the 2D vascular centerline structure corresponding to the 2D XCA image are decomposed into starting point information and morphological information. The starting point information passes through the starting point feature extraction branch, and the morphological information passes through the morphological feature extraction branch. Feature extraction and feature fusion are performed through these two information flow channels to form dual-stream features. Angular displacement prediction: The dual-stream features are fed into a deep neural network to predict the angular deflection between two adjacent points as a displacement field. Based on this, deformation modeling is performed on the preoperatively acquired 3D coronary artery structure and projected onto the intraoperative 2D imaging plane for registration. The pre-processing step comprises: CTA image training steps: Randomly select CTA images, perform voxel-by-voxel vessel annotation, and train a convolutional neural network (3D U-net) for 3D coronary artery segmentation based on the original image and corresponding labels; CTA processing steps: Use the trained 3D U-net to segment the CTA of the new data, use the morphological method to extract the centerline, and use the smoothing algorithm to further smooth the extracted CTA centerline; XCA image training steps: Randomly select XCA images and perform pixel-by-pixel vessel annotation. Based on the original image and the corresponding labels, a convolutional neural network (2D U-net) for 2D coronary artery segmentation is trained; XCA processing steps: Use the trained 2D U-net to segment the XCA of the new data, use the morphological method to extract the centerline, and use the smoothing algorithm to further smooth the extracted XCA centerline; The dual-stream feature extraction step includes: CTA coordinate representation steps: The centerline of the CTA vascular segment is represented as a set of continuous points C = {a1,…,a n }, where a n =(x n ,y n ,z n ) is represented by the Euclidean coordinates of a point in three-dimensional space; it can be decomposed into the starting point information a1 and the morphological information {a2-a1,…,a n -a n-1 }, where the shape is represented by the set of offsets between two adjacent points; XCA coordinate representation steps: The center line of the XCA blood vessel segment is represented as a set of continuous points X = {b1,…,b n }, where b n =(x n ,y n ) is represented by the Euclidean coordinates of the point in two-dimensional space; it can be decomposed into the starting point information b1 and the morphological information {b2-b1,…,b n -b n-1 }, where the shape is represented by the set of offsets between two adjacent points; CTA feature fusion steps: Input the CTA centerline starting point information set with a dimension of N×3×1 into the CTA starting point feature extraction branch, and use a series of one-dimensional convolution kernels of size 1 for convolution to obtain a CTA starting point feature with a dimension of N×m×1; Input the CTA centerline morphological information set with a dimension of N×3×(n-1) into the CTA morphological feature extraction branch, and use a series of one-dimensional convolution kernels of size 3 for convolution to obtain a CTA morphological feature with a dimension of N×m×(n-1); Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion; XCA feature fusion steps: Input the XCA centerline starting point information set with a dimension of N×2×1 into the XCA starting point feature extraction branch, and use a series of one-dimensional convolution kernels of size 1 for convolution to obtain an XCA starting point feature with a dimension of N×m×1; Input the XCA centerline morphological information set with a dimension of N×2×(n-1) into the XCA morphological feature extraction branch, and use a series of one-dimensional convolution kernels of size 3 for convolution to obtain an XCA morphological feature with a dimension of N×m×(n-1); Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion; Splicing and fusion step: Splice the CTA features and XCA features in the channel dimension to obtain a cross-modal fusion feature with a dimension of N×2m×(n-1); The method is based on the Pytorch deep learning framework and uses the Python programming language.

2. The 2D / 3D coronary artery automatic registration method according to claim 1, characterized in that: The angular displacement prediction step comprises: Angular offset output step: The cross-modal fusion feature that combines CTA and XCA information is input into the angular displacement prediction network. The angular displacement prediction network is based on a series of one-dimensional convolution kernels of size 3, divided into two parts: downsampling and upsampling. It uses a skip connection structure and finally outputs the angular offset of a point on the centerline relative to the previous point. Morphological restoration step: Based on the output angle offset, the deformed three-dimensional coronary artery morphology is restored according to the following formula: Among them, θ and is the change of two angles in space, and r is the distance between two adjacent points. During the training phase, a Gaussian mixture model is established to measure the similarity of two point sets: Among them, L GMM Represents error, P CTA represents the CTA point set, P XCA represents the XCA point set, and σ represents the smoothing parameter.

3. A 2D / 3D coronary artery automatic registration system, characterized in that: Includes the following modules: Preprocessing module: vascular structure segmentation and centerline extraction are performed on the patient's preoperative 3D CTA images and the corresponding intraoperative 2D XCA images, respectively. The centerline containing the vascular geometric topology information is used as the vascular feature; Dual-stream feature extraction module: The 3D vascular centerline structure corresponding to the 3D CTA image and the 2D vascular centerline structure corresponding to the 2D XCA image are decomposed into starting point information and morphological information. The starting point information passes through the starting point feature extraction branch, and the morphological information passes through the morphological feature extraction branch. Feature extraction and feature fusion are performed through these two information flow channels to form dual-stream features. Angular displacement prediction module: This module inputs dual-stream features into a deep neural network to predict the angular deflection between two adjacent points as a displacement field. Based on this, deformation modeling is performed on the preoperatively acquired 3D coronary artery structure and projected onto the intraoperative 2D imaging plane for registration. The pre-processing module comprises: CTA image training module: Randomly select CTA images, perform voxel-by-voxel vessel annotation, and train a convolutional neural network (3D U-net) for 3D coronary artery segmentation based on the original image and corresponding labels; CTA processing module: Use the trained 3D U-net to segment the CTA of new data, use morphological methods to extract the centerline, and use the smoothing algorithm to further smooth the extracted CTA centerline; XCA image training module: randomly selects XCA images, performs pixel-by-pixel vessel annotation, and trains a convolutional neural network (2D U-net) for two-dimensional coronary artery segmentation based on the original image and corresponding labels; XCA processing module: Use the trained 2D U-net to segment the XCA of new data, use morphological methods to extract the centerline, and use the smoothing algorithm to further smooth the extracted XCA centerline; The dual-stream feature extraction module includes: CTA coordinate representation module: The centerline of a CTA vascular segment is represented as a set of continuous points C = {a1,…,a n }, where a n =(x n ,y n ,z n ) is represented by the Euclidean coordinates of a point in three-dimensional space; it can be decomposed into the starting point information a1 and the morphological information {a2-a1,…,a n -a n-1 }, where the shape is represented by the set of offsets between two adjacent points; XCA coordinate representation module: The centerline of the XCA blood vessel segment is represented as a set of continuous points X = {b1,…,b n }, where b n =(x n ,y n ) is represented by the Euclidean coordinates of the point in two-dimensional space; it can be decomposed into the starting point information b1 and the morphological information {b2-b1,…,b n -b n-1 }, where the shape is represented by the set of offsets between two adjacent points; CTA feature fusion module: The CTA centerline starting point information set with a dimension of N×3×1 is input into the CTA starting point feature extraction branch, and convolution is performed using a series of one-dimensional convolution kernels of size 1 to obtain a CTA starting point feature with a dimension of N×m×1. The CTA centerline morphological information set with a dimension of N×3×(n-1) is input into the CTA morphological feature extraction branch, and convolution is performed using a series of one-dimensional convolution kernels of size 3 to obtain a CTA morphological feature with a dimension of N×m×(n-1). Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion. XCA feature fusion module: The XCA centerline starting point information set with a dimension of N×2×1 is input into the XCA starting point feature extraction branch, and a series of one-dimensional convolution kernels of size 1 are used for convolution to obtain an XCA starting point feature with a dimension of N×m×1. The XCA centerline morphological information set with a dimension of N×2×(-1) is input into the XCA morphological feature extraction branch, and a series of one-dimensional convolution kernels of size 3 are used for convolution to obtain an XCA morphological feature with a dimension of N×m×(n-1). Finally, the N×m×1 starting point feature is copied along the last dimension to N×m×(n-1) and added to the morphological feature for feature fusion. Splicing and fusion module: Splice the CTA features and XCA features in the channel dimension to obtain a cross-modal fusion feature with a dimension of N×2m×(n-1); The system is based on the Pytorch deep learning framework and uses the Python programming language.

4. The 2D / 3D coronary artery automatic registration system according to claim 3, characterized in that: The angular displacement prediction module includes: Angle offset output module: This module inputs the cross-modal fusion features that combine CTA and XCA information into the angular displacement prediction network. The angular displacement prediction network is based on a series of one-dimensional convolution kernels of size 3, divided into downsampling and upsampling parts, and uses a skip connection structure to output the angular offset of a point on the centerline relative to the previous point. Morphological Restoration Module: Based on the output angle offset, the deformed 3D coronary artery morphology is restored according to the following formula: Among them, θ and is the change of two angles in space, and r is the distance between two adjacent points. During the training phase, a Gaussian mixture model is established to measure the similarity of two point sets: Among them, L GMM Represents error, P CTA represents the CTA point set, P XCA represents the XCA point set, and σ represents the smoothing parameter.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images