Registration method between 2D (two-dimensional) and 3D (three-dimensional) images without supervised learning

Abstract

The invention discloses an unsupervised learning 2D and 3D image registration method, and belongs to the field of computer medical image processing. According to the method, a CNN-Transform parallel network and improved projection space transformation based on a parallel projection mode are constructed, specifically, 3D TOF and 2D DSA data are obtained and preprocessed, the improved projection space transformation based on the parallel projection mode is constructed, a parallel CNN-Transform double-branch network (CTNet) is constructed, and the method is used for evaluating feature similarity, training the CTNet and achieving precise registration of the 2D DSA and the 3D TOF. The system is used for interventional operation navigation, provides three-dimensional guidance for catheter path planning and embolism treatment, is used for complex lesion assessment, and provides multi-aspect support for disease diagnosis and treatment. According to the method, global and local features can be extracted more comprehensively for feature similarity evaluation, and the registration performance is improved; the projection image is generated by adopting the improved projection space transformation based on the parallel projection mode, and the generated projection image is more matched with the TOF imaging principle.

Description

Technical Field

[0001] The present invention relates to an unsupervised learning registration method between 2D and 3D images, and belongs to the field of computer medical image processing. Background Art

[0002] In image-guided surgical interventions, geometric transformations of 3D images are performed to achieve optimal alignment with 2D images, providing better visualization and more accurate diagnostic evidence. Currently, commonly used 3D imaging technologies include computed tomography (CT) and magnetic resonance imaging (MRI), while commonly used 2D imaging technologies include projection X-rays and digital subtraction angiography (DSA). Image registration technologies primarily focus on 2D / 2D and 3D / 3D registration, and progress in 2D / 3D registration has been relatively slow.

[0003] The goal of 2D / 3D image registration is to find a spatial transformation that aligns the projection of a 3D image with the 2D image as closely as possible. Traditional optimization-based methods are widely used for 2D / 3D registration in interventional surgery due to their high accuracy, but they are sensitive to initialization and easily fall into local optimality. With the continuous development of deep learning technology, methods based on convolutional neural networks (CNNs) and transformers have been introduced into the registration task. Compared with traditional optimization-based methods, deep learning-based methods have shown significant advantages in registration performance. CNN-based methods have successfully overcome the limitations of traditional methods and have been widely used in image registration tasks.

[0004] However, CNN architectures typically focus on extracting local information within a limited receptive field, making it challenging to capture the global information required to achieve high-precision registration results; Transformer captures long-range dependencies in the input sequence through a multi-head self-attention mechanism and is good at modeling global information, but is insufficient in capturing local features, resulting in the loss of key image details; serially combining CNN and Transformer to overcome their respective limitations and effectively extract global and local information from the image, still fails to achieve ideal results in extracting global information, resulting in insufficient registration accuracy; Projective Spatial Transformer (ProST) is used to generate 2D digitally reconstructed radiographs (DRR) images from 3D objects, using cone-beam projection, which is inconsistent with the principle of TOF imaging, resulting in information loss and affecting registration performance.

[0005] 3D and 2D imaging are widely used in disease diagnosis and treatment. Image registration has extensive practical applications in medical image processing and analysis, observing disease progression and treatment effects, surgical navigation, and 3D reconstruction of sequential images. Numerous existing 2D / 3D registration methods exist, but each has its own limitations. A new 2D / 3D registration method is urgently needed to further improve the accuracy of registration. Summary of the Invention

[0006] In order to solve the problem of insufficient accuracy of existing 2D / 3D registration methods in image registration, the purpose of the present invention is to provide an unsupervised learning registration method between 2D and 3D images, which realizes the accurate registration of 2D DSA and 3D TOF by constructing a CNN-Transformer parallel network and an improved projection space transformation based on parallel projection.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A method for registration between 2D and 3D images using unsupervised learning, comprising the following steps:

[0009] Step 1: Acquire 3D TOF and 2D DSA data;

[0010] Step 2: Preprocessing 3D TOF and 2D DSA data;

[0011] Step 3: Construct an improved projective spatial transform (iProST) based on parallel projection.

[0012] Step 4: Build a parallel CNN-Transformer dual-branch network (CTNet) for feature similarity evaluation;

[0013] The CTNet consists of a global feature encoder, a local feature encoder, and a consistency evaluation module;

[0014] Step 5: Training CTNet, which includes the following sub-steps:

[0015] Step 5.1: Set the maximum number of training rounds and initialize the pose parameters θ.

[0016] Step 5.2 Determine the TOF image I projected by iProST m Compared with the preprocessed DSA image I f Normalized Cross Correlation (NCC) loss between and matching loss of CTNet output;

[0017] Update the CTNet parameters and repeat step 5.2 until the maximum number of training rounds is reached;

[0018] Step 6: Determine the registration parameters between 2D and 3D images;

[0019] Input the pre-processed TOF image and DSA image into the trained iProST and CTNet to obtain the final pose parameters and complete the TOF image and DSA image registration;

[0020] It also includes step seven, applying the TOF image and DSA image registered in step six for interventional surgery navigation, providing three-dimensional guidance for catheter path planning and embolization therapy, combining dynamic and static information for complex lesion assessment, improving the accuracy of aneurysm rupture risk assessment and vascular malformation classification, and providing multi-faceted support for disease diagnosis and treatment.

[0021] Beneficial effects:

[0022] 1. The present invention discloses an unsupervised learning method for 2D and 3D image registration. By constructing a parallel CNN-Transformer dual-branch network, compared with a serial CNN-Transformer network, it can more comprehensively extract global and local features from TOF images and DSA images for feature similarity assessment, thereby improving registration performance.

[0023] 2. The present invention discloses an unsupervised learning method for registration between 2D and 3D images, which uses an improved projection space transformation based on parallel projection to generate a projection image. Compared with the projection space transformation based on cone beam, the generated projection image is more consistent with the imaging principle of TOF. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of an unsupervised learning registration method between 2D and 3D images of the present invention;

[0025] Figure 2 Schematic diagram of iProST in an unsupervised learning registration method between 2D and 3D images of the present invention;

[0026] Figure 3 This is a diagram of the CTNet architecture in an unsupervised learning method for registration between 2D and 3D images of the present invention;

[0027] Figure 4 This is a schematic diagram comparing the visualization results of different registration methods for internal data sets in the embodiment;

[0028] Figure 5 Schematic diagram comparing the quantitative registration results of different methods for internal data sets in the embodiment. DETAILED DESCRIPTION

[0029] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below with reference to the accompanying drawings and examples.

[0030] Example 1:

[0031] Taking patients with moyamoya disease whose cerebral vascular system is severely damaged as an example, the unsupervised learning registration method between 2D and 3D images of the present invention is applied to realize the registration between 3D TOF and 2D DSA, such as Figure 1 As shown, the following steps are included:

[0032] Step 1: Acquire 3D TOF and 2D DSA data;

[0033] Step 2: Preprocessing 3D TOF and 2D DSA data;

[0034] Use the pre-trained nnU-Net to segment the cerebral blood vessels in the TOF image V and remove the brain tissue information in the TOF image;

[0035] The DSA image I is inverted so that the transformed DSA image is similar to the TOF image, and the high-intensity signal represents the vascular system;

[0036] The Z-score method was used to normalize the grayscale distribution of the TOF image and the inverted DSA image;

[0037] The normalized TOF image is cropped to 256×256×256 to obtain the preprocessed TOF image V′, and the normalized DSA image is cropped to 256×256 to obtain the preprocessed DSA image I f ;

[0038] Step 3: Construct an improved projective spatial transform (iProST) based on parallel projection.

[0039] iProST Figure 2 As shown in the figure, when generating the geometric projection of 3D TOF, the same settings as ProST are retained, such as normalized coordinates, spatial grid of control points, and projection layer. Parallel projection is used to replace the original DRR projection method in ProST, generating rays from multiple source points, and the directions of these rays are consistent;

[0040] Given a fixed 2D image I fand a varying 3D volume V, rigid 2D / 3D image registration is to find the spatial pose parameters θ∈SE(3) by applying a mapping function F to V based on θ and the acquired image I m As similar as possible to I f , as shown in formula (1):

[0041]

[0042] Among them, θ is a 6-degree-of-freedom parameter vector consisting of three rotation parameters and three translation parameters, and θ is (r x ,r y ,r z ,t x ,t y ,t z ), the initial value of θ is randomly set and iteratively updated until convergence. P(·) is the projective space transformation. P(·) takes θ and V as input and outputs I m , the dimension of P(·) is related to I f Keep consistent, S(·) represents the value used to evaluate I m and I f A function of the similarity between them, which promotes the optimization of θ during back propagation;

[0043] Step 4: Build a parallel CNN-Transformer dual-branch network (CTNet) for feature similarity evaluation;

[0044] like Figure 3 As shown in Figure 1, the network consists of a global feature encoder (GFE), a local feature encoder (LFE), and a consistency evaluation (CE) module;

[0045] In the embodiment, GFE is composed of two Swin Transformer modules, which are used to respectively m and I f Extract global features of the image;

[0046] The shifting window mechanism in the Swin Transformer has the ability to model long-range dependencies, and the layered architecture is adaptable to multi-resolution inputs, eliminating the need for additional scaling operations while preserving key information.

[0047] In the embodiment, LFE is composed of a weight-shared shallow feature extraction (WSFE) module and two high-frequency separation (HFS) modules;

[0048] The WSFE module consists of multiple convolution blocks and pooling blocks. m and I f Extracting shared shallow features. The HFS module is mainly composed of multiple inverted residual (IR) blocks. It separates the high-frequency local features and surface-level features in the shallow features and can capture texture details with specific physical meanings.

[0049] In an embodiment, the CE module includes a Convolutional Multi-Attention (CMA) module and a Multi-Layer Perceptron (MLP) for evaluating the consistency between global similarity and local similarity;

[0050] GFE respectively from I m and I f Extract the global eigenvector g from m and g f , LFE from I m and I f Extract the feature vector l m and l f , the two global features g extracted m 、g f And two local features l m 、l f Perform subtraction operations respectively, get the difference and then square it, as shown in formula (2):

[0051]

[0052] Global Features is input into MLP to determine the global feature similarity, local feature It is input into the CMA module to determine the local feature similarity, as shown in formula (3):

[0053]

[0054] Among them, ψ g is the global similarity estimation, ψ l is the local similarity estimation, MLP(·) is the multi-layer perceptron, and CMA(·) is the convolutional multi-attention module;

[0055] In the embodiment, CTNet is deployed on a computer with an NVIDIA RTX 1080Ti GPU and implemented using the deep learning framework Pytorch;

[0056] Step 5: Training CTNet, which includes the following sub-steps:

[0057] Step 5.1: Set the maximum number of training rounds and initialize the pose parameters θ.

[0058] In this embodiment, a pose parameter θ is randomly sampled i , three rotation parameters (r x ,r y ,r z ) and three translation parameters (t x ,t y ,t z ), and further fine-tune the rotation parameters and set them to be learnable. Randomly select a TOF data V and a corresponding sagittal or coronal DSA image as a fixed image I f ;

[0059] The learning rate is 1×10 -4 , the momentum is 0.9, α is set to 0.5, and the training process involves 200 epochs, each epoch iterates 50 times;

[0060] Step 5.2 Determine the TOF image I projected by iProST m Compared with the preprocessed DSA image I f Normalized Cross Correlation (NCC) loss between and matching loss of CTNet output, NCC loss As shown in formula (4), the matching loss As shown in formula (5), the total loss As shown in formula (6):

[0061]

[0062] in, is the NCC loss function, which is used to measure the motion image I m With fixed image I f The similarities between The smaller it is, the higher the similarity between the two images. is the matching loss function, which is used to measure the difference between global similarity and local similarity. The smaller it is, the closer the global features are to the local features. If the two images are completely aligned, the global similarity is equal to the local similarity. The value of is 0, ABS(·) is the absolute value of the input, and α is the weight parameter for balancing different loss terms;

[0063] Update the CTNet parameters and repeat step 5.2 until the maximum number of training rounds is reached;

[0064] Step 6: Determine the registration parameters between 2D and 3D images;

[0065] Input the pre-processed TOF image and DSA image into the trained CTNet to obtain the final pose parameters and complete the matching of TOF image and DSA image;

[0066] In the embodiment, the visualization results based on the method of the present invention and the existing advanced method as a comparison are as follows: Figure 4 shown, where the blood vessels are highlighted in red in the projected image;

[0067] Compared with ProST-based methods, all iProST-based methods generate clearer vascular projection images with richer and more complete details. The projection principle of iProST is more consistent with the imaging method of TOF images. In particular, when CTNet and iProST are combined, the blood vessels in the projected images in two different views are the clearest and most complete. The method of the present invention can extract effective information more comprehensively and significantly improve the registration accuracy.

[0068] In the embodiment, the quantitative results based on the method of the present invention and the existing advanced method as a comparison are as follows Figure 5 As shown in the figure, when only iProST or ProST is used, the performance of all three evaluation indicators is suboptimal, which indicates that it is necessary to add modules to further refine the registration. When ProST is used as the projection method, the average NCC value of CTNet as the feature extraction module is 0.0401 higher than the average NCC value of CDNet. When iProST is used as the projection method, the difference becomes 0.0519. In summary, CTNet is superior to CDNet in feature extraction ability. When CTNet uses iProST, the SSIM and COSINE values on the two views are higher than those when ProST is used, and the average NCC value of CTNet using iProST is 0.0076 higher than that using ProST. iProST is more in line with the principle of TOF imaging. CTNet using iProST has the highest average value in all three indicators and has excellent registration performance.

[0069] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for registration between 2D and 3D images using unsupervised learning, characterized by: By building a CNN-Transformer parallel network and improving the projection space transformation based on parallel projection, accurate registration of 2D DSA and 3D TOF is achieved. The specific steps include: Step 1: Acquire 3D TOF and 2D DSA data; Step 2: Preprocessing 3D TOF and 2D DSA data; Step 3: Construct an improved projective spatial transform (iProST) based on parallel projection. Step 4: Build a parallel CNN-Transformer dual-branch network (CTNet) for feature similarity evaluation; The CTNet consists of a global feature encoder, a local feature encoder, and a consistency evaluation module; Step 5: Training CTNet, which includes the following sub-steps: Step 5.1: Set the maximum number of training rounds and initialize the pose parameters θ. Step 5.2 Determine the TOF image I projected by iProST m Compared with the preprocessed DSA image I f Normalized Cross Correlation (NCC) loss between and matching loss of CTNet output; Update the CTNet parameters and repeat step 5.2 until the maximum number of training rounds is reached; Step 6: Determine the registration parameters between 2D and 3D images; The pre-processed TOF image and DSA image are input into the trained iProST and CTNet to obtain the final pose parameters and complete the TOF image and DSA image registration.

2. The unsupervised learning registration method between 2D and 3D images according to claim 1, characterized in that: Step seven: Use the TOF image and DSA image registered in step six for interventional surgical navigation, provide three-dimensional guidance for catheter path planning and embolization therapy, combine dynamic and static information for complex lesion assessment, improve the accuracy of aneurysm rupture risk assessment and vascular malformation classification, and provide multi-faceted support for disease diagnosis and treatment.

Images