DSA platform-based multi-state real-time multi-mode image accurate fusion method

Abstract

The invention belongs to the technical field of DSA, and discloses a DSA platform-based multi-state real-time multi-mode image accurate fusion method. After an angiography image is collected on the basis of DSA equipment under the motion condition, accurate DSA blood vessel subtraction image enhanced display is achieved, meanwhile, hepatic artery extraction in a three-dimensional image in preoperative CT is completed, dynamic intelligent fusion of a precise anatomical structure in a preoperative multi-mode CT image and intraoperative local blood vessel development is combined, and the dynamic and intelligent fusion of the dynamic and intelligent fusion of the dynamic and intelligent fusion of the dynamic and intelligent fusion of the dynamic and intelligent fusion of the dynamic and intelligent fusion of the dynamic and intelligent fusion of the dynamic and intelligent fusion is achieved. The intrahepatic lesions and important blood vessels are accurately and visually displayed, operation planning is completed, fusion of three-dimensional information and two-dimensional DSA is utilized, and transvascular and percutaneous interventional operations are accurately guided.

Description

Technical Field

[0001] This invention relates to the field of DSA technology, and in particular to a method for accurate fusion of multi-modal real-time multi-images based on a DSA platform. Background Technology

[0002] Interventional liver therapy, utilizing percutaneous, transvascular, and transcirculatory approaches, employs methods such as embolization, ablation, internal irradiation, and shunt establishment. It has become an important clinical local treatment option for patients with advanced cancer and cirrhosis, effectively reducing tumor burden, lowering portal pressure, and improving clinical prognosis. However, clinical practice of interventional liver therapy still faces several bottlenecks that urgently need to be overcome: First, accuracy is insufficient. For some poorly vascularized lesions, localization is difficult, and the feeding arteries are hard to identify. Respiratory movements also affect the accuracy, requiring repeated adjustments to the instrument's angle and direction to reach the lesion and blood vessels. Second, intelligence is lacking. 2D-DSA is affected by overlapping vascular images, rapid contrast agent clearance, and respiratory movements, making the identification of the lesion's feeding vessels and catheter insertion path primarily rely on personal experience. During puncture, manual measurement is required to plan the puncture path. Finally, different interventional procedures rely on either DSA or CT-guided navigation, increasing patient discomfort and reducing efficacy. Given the current limitations of interventional liver treatment, how to achieve combined percutaneous, transvascular, and transcirculatory therapies on DSA equipment while improving intelligence and precision has always been a promising research direction and hot topic in the field of interventional therapy. Currently, cone-beam computed tomography (CBCT) on DSA equipment can achieve CT-like imaging functions, including lesion localization, intraoperative efficacy evaluation, and even guidance for percutaneous puncture and shunt establishment, making the concept of "one-stop" interventional treatment possible. However, CBCT imaging still has inherent limitations such as low image resolution and non-real-time imaging, making it difficult to use for multimodal image fusion. Therefore, it is necessary to seek more intelligent and precise imaging solutions.

[0003] However, to achieve the above objectives, the following technical challenges still need to be overcome: how to accurately fuse preoperative multimodal three-dimensional image data on DSA two-dimensional images in order to superimpose lesion and important anatomical structure information required for intraoperative combined treatment. Summary of the Invention

[0004] The purpose of this invention is to provide a method for accurate fusion of multi-modal real-time multi-images based on the DSA platform.

[0005] The above-mentioned technical objective of this invention is achieved through the following technical solution: a method for accurate fusion of multi-modal real-time multi-images based on the DSA platform, comprising the following steps: Step 1: Based on real-time registration and analysis of X-ray sequence images, motion effects are removed to obtain accurate DSA images; Step 2: Segment and extract preoperative CT multi-phase images to establish liver contour, arteries, veins and lesion segmentation structure, forming a three-dimensional liver structure route map; Step 3: Using a projection fusion algorithm, the anatomical structure and vascular information of the three-dimensional liver structure roadmap are fused and displayed in real time in the precise DSA image, thus achieving image fusion.

[0006] By employing the above technical solution, in order to more clearly display enhanced blood vessels on DSA images, this invention performs real-time registration and analysis of X-ray sequence images, removing image subtraction caused by motion effects to obtain a clearer DSA image sequence, thereby enhancing the display of blood vessels in the region of interest. Furthermore, it detects the movement of the vascular tissue, providing a foundation for the next step of fusing the preoperative CT-based three-dimensional vascular structure (i.e., roadmap) with the 2D vascular projection structure described in the book.

[0007] Preoperative segmentation and extraction of multimodal CT images are crucial for surgical planning and guiding intraoperative interventional procedures. Compared to intraoperative X-rays or CBCT, preoperative CT images provide high-resolution information on liver anatomy, arterial and venous visualization, and lesion shape and location. Preoperative CT arterial and portal venous phase images enhance the arteries and veins of the liver, respectively. By fusing and registering these images and segmenting the vessels, combined with lesion segmentation, a three-dimensional structure of the liver shape, arterial and venous structures, and lesion extent can be obtained. This facilitates surgical planning and allows for the precise guidance of transvascular and percutaneous interventional procedures by fusing three-dimensional information with two-dimensional DSA.

[0008] Current grayscale image registration techniques are not suitable for registration between DSA and CT images. Compared to grayscale-based registration algorithms, vascular structure-based registration methods can more effectively utilize the correspondences of anatomical structures, improving the robustness of registration. Typically, the anatomical structures in a 3D image, especially vascular structures, are referred to as a "roadmap." 2D / 3D registration between intraoperative 2D-DSA and preoperative 3D-CTA involves transforming the roadmap into a 2D image, which can enhance the visualization of blood vessels and guide interventional treatment.

[0009] A further provision of the present invention is as follows: In step one, the sequence image registration uses an unsupervised registration network based on MSE as the loss. The inputs are a floating image and a fixed image. The floating image is a DSA image without contrast agent, and the fixed image is an image with contrast agent. The loss function of the registration network includes the mean square error of grayscale of the deformed floating image and the fixed image after registration, as well as the smoothing loss function of the deformation field. The output is the deformation field between the two, thus completing the registration of the two DSA images and forming an enhanced angiography.

[0010] A further provision of the present invention is as follows: In step one, the shape of each respiratory period of the respiratory cycle is extracted during the patient's respiratory cycle, and the respiratory phase state corresponding to the DSA image at a certain time point is obtained through analysis; the sequence images are registered, and the patient's motion change pattern is obtained by combining the anatomical structure position; the patient's respiratory motion change pattern can be calculated by segmenting the liver tissue in the real-time DSA fluoroscopy sequence during interventional surgery; during interventional surgery, the real-time respiratory phase can be obtained by calculating the mean square error between the current DSA image frame and the key frame of a respiratory cycle of the patient.

[0011] A further feature of this invention is that, in step two, based on the preoperative multi-phase enhanced CT images of the liver in the arterial and venous phases, the shape of the liver, the location and shape of the lesion, and the anatomical structure of the arteries and veins are extracted, and three-dimensional segmentation and visualization are performed. During segmentation, the vascular mask and the gold standard vascular / bile duct mask are extracted using a skeletal layer to form a three-dimensional liver structure roadmap.

[0012] A further setting of the present invention is as follows: In step three, the spatial projection relationship between the precise 2D DSA image and the 3D three-dimensional liver structure route map is initialized according to the DSA equipment parameters. The 2D image of the individual located at the rotation center or the center of the world coordinate system o point on the DSA projection board is transformed by the distance SID from the emission source to the imaging panel, the distance SOD from the emission source to the imaging object, and the rotation of the C-arm relative to the isocentricity, and is determined by the angles LAO / RAO (α) and CRAN / CAUD (β). The projection transformation is simplified to Formula 1 as follows:

[0013] Where μ and υ represent pixel positions on the DSA image, and x, y, z represent points in world coordinates.

[0014] A further provision of the present invention is that, in step three, points in the CT image... m', y', w' The transformation relationship with the world coordinate system is used T w express, T w This also includes the patient's positional changes during CT scans and interventions. Assuming the transformation between the two is a rigid transformation, the transformation relationship between the CT image and the C-arm world coordinate system is expressed by Equation 2:

[0015] With intraoperative CBCT scanning already completed, the above coordinate correspondence is obtained by directly registering the 3DCBCT and CT images. T w .

[0016] A further provision of the present invention is that, in step three, the angiography image is generated based on a 3DCT image simulation, as expressed by formula 3:

[0017] in, I 0 and I These are the initial energy and the decay energy, respectively. l It is the penetration path of X-rays. m It is the attenuation coefficient, calculated from the CT values ​​of the image. m w It is the attenuation coefficient of water.

[0018] A further provision of the present invention is that, in step three, the image fusion method involves setting the three-dimensional liver structure route map as follows: I The corresponding arteries and veins are A、V The goal of direct solution is to solve for the optimal deformation field. f(μ',υ',w') This results in the image or structure after deformation. I'= f(μ',υ',w')°I The projected images according to Formulas 1 and 2 are similar to DSA / X-ray images. Tp°TW(I')~DSA .

[0019] The invention is further configured as follows: in step one, a 2D image registration network based on VoxelMorph is used to register real-time X-ray sequence images; in step two, a V-Net structure network based on multi-resolution multi-head extraction is used to complete image segmentation and blood vessel extraction.

[0020] The beneficial effects of this invention are as follows: Based on the acquisition of angiography images by DSA equipment under motion conditions, this invention achieves precise DSA angiography image enhancement display. At the same time, it completes the extraction of the hepatic artery from the three-dimensional images in the preoperative CT. Combined with the precise anatomical structure in the preoperative multi-mode CT images and the dynamic intelligent fusion of local vascular imaging during the operation, it accurately and visually displays intrahepatic lesions and important blood vessels, which helps to complete surgical planning and utilize the fusion of three-dimensional information and two-dimensional DSA to accurately guide transvascular and percutaneous interventional surgery. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of unsupervised DSA image registration in step one of the present invention.

[0022] Figure 2 This is a schematic diagram of DSA image sequence registration and dynamic analysis according to the present invention, wherein the dashed line represents the obtained real-time respiratory phase.

[0023] Figure 3 This invention utilizes enhanced CT images of the liver to create the outline of the liver parenchyma and segment intrahepatic blood vessels.

[0024] Figure 4 This is a schematic diagram of the 3D liver blood vessel segmentation network of the present invention.

[0025] Figure 5 A. The relationship between the changes in 3D image coordinates and DSA image coordinates; B. The relationship between 3D and 2D projection. Detailed Implementation

[0026] A method for accurate fusion of multi-modal, real-time, multi-image data based on the DSA platform includes the following steps: Step 1: Based on real-time registration and analysis of X-ray sequence images, motion effects are removed to obtain accurate DSA images.

[0027] Specifically, image registration uses an unsupervised registration network based on MSE as the loss function. The inputs are a floating image (DSA image without contrast agent) and a fixed image (image with contrast agent). The network's loss function includes the mean square error of grayscale between the registered floating and fixed images, as well as a smoothing loss function for the deformation field. The output is the deformation field between the two images, completing the registration of the two DSA images and enhancing the angiography. The network's loss function includes the mean square error of grayscale between the registered floating and fixed images and a smoothing loss function for the deformation field. After training with liver DSA images from different cases, the network can complete the registration of two DSA images at a rate greater than 5 frames per second, enhancing the angiography. Additionally, a 2D image registration network based on VoxelMorph is used to register real-time X-ray image sequences.

[0028] During the patient's respiratory cycle, the shape of each respiratory period is extracted, and the respiratory phase state corresponding to a DSA image at a certain time point is obtained through analysis. The sequence images are registered and combined with the anatomical structure position to obtain the patient's motion change pattern. By segmenting the liver tissue in the real-time DSA fluoroscopy sequence during interventional surgery, the patient's respiratory motion change curve during the operation can be calculated. Figure 2 The curve in the upper part of the image shows that longitudinal changes in the liver (pleura) are an important indicator for extracting a patient's respiration. During interventional procedures, for the current DSA image frame, the real-time respiratory phase (e.g., ...) can be obtained by calculating the mean square error between it and a keyframe of a respiratory cycle for that patient. Figure 2 (As shown by the dashed line in the image).

[0029] Step 2 involves segmenting and extracting preoperative CT multi-phase images to establish the liver contour, arteries, veins, and lesion segmentation structure, forming a three-dimensional liver structure roadmap.

[0030] Specifically, based on preoperative multi-phase enhanced CT images of the liver in the arterial and venous phases, the liver shape, lesion location and shape, and arterial and venous anatomical structures were extracted for 3D segmentation and visualization. During segmentation, the vascular mask and the gold standard vascular / bile duct mask were extracted using a skeletonization layer to form a 3D liver structure roadmap. The image segmentation and vascular extraction were performed using a V-Net network based on multi-resolution multi-head extraction.

[0031] Step 3: Using a projection fusion algorithm, the anatomical structure and vascular information of the three-dimensional liver structure roadmap are fused and displayed in real time in the precise DSA image, thus achieving image fusion.

[0032] Specifically, the spatial projection relationship between the precise 2D DSA image and the 3D liver structure route map is initialized based on the DSA equipment parameters. A C-arm fluoroscopy is a single-direction X-ray projection, while CT is a 3D reconstruction obtained under rotation; both are formed by X-ray fluoroscopy. Therefore, at a certain angle, a fluoroscopic image in that direction can be simulated from a CT image. Figure 5 Display A shows a schematic diagram of the C-arm's projection in the world coordinate system, representing the 2D image of an individual located at the center of rotation or the center of the world coordinate system (point o) on the DSA projection panel. This projection transformation is determined by the distance SID from the emission source to the imaging panel, the distance SOD from the emission source to the imaging object, and the C-arm's rotation relative to the isocentricity, and is further determined by the angles LAO / RAO (α) and CRAN / CAUD (β). The projection transformation is simplified using Equation 1:

[0033] Where μ and υ represent pixel positions on the DSA image, and x, y, z represent points in world coordinates. Points in CT images m', y', w' The transformation relationship with the world coordinate system is used T w This indicates that (the transformation parameter can be obtained from a DICOM standard image file). T w This also includes the patient's positional changes during CT scans and interventions. When the patient's position is the same, it is assumed that the transformation relationship between the position during intervention and the CT scan is a rigid transformation, i.e., a transformation with 6 degrees of freedom. This transformation can be obtained by using a rigid registration method with initial assumptions of the change parameters, or it can be obtained through electromagnetic marking and electromagnetic tracking methods.

[0034] Here, it is assumed that the patient's physical condition is the same during the CT scan and the intervention; therefore, the transformation between the two can be represented by a rigid transformation. Under this assumption, the transformation relationship between the CT image and the C-arm world coordinate system is expressed by Equation 2:

[0035] With intraoperative CBCT scanning already completed, the above coordinate correspondence is obtained by directly registering the 3DCBCT and CT images. T w 。 Note that since the DSA images and CBCT scans use the same equipment, and the patient's position remains unchanged, this coordinate transformation can be directly used to determine the coordinate transformation relationship between the CT and C-arm systems.

[0036] To simulate the generation of angiographic images, it is necessary to calculate the intensity accumulation effect of X-rays penetrating the human body. Considering that different tissues exhibit different absorption rates, and based on the attenuation along the penetration path at different tissue densities, according to the imaging principles of CT images, this projection image can be generated based on 3D CT images, expressed by Equation 3:

[0037] in, I 0 and I These are the initial energy and the decay energy, respectively. l It is the penetration path of X-rays. m It is the attenuation coefficient, calculated from the CT values ​​of the image. m w It is the attenuation coefficient of water.

[0038] The image fusion method involves setting a three-dimensional liver structure roadmap as... I The corresponding arteries and veins are A、V The goal of direct solution is to solve for the optimal deformation field. f(μ',υ',w') This results in the image or structure after deformation. I'= f(μ',υ',w')° I The projected images according to Formulas 1 and 2 are similar to DSA / X-ray images. Tp°TW(I')~DSA.

Claims

1. A method for multi-modal image fusion based on a DSA platform, characterized in that: It comprises the following steps, Step one, based on X-ray sequence image real-time registration and analysis, remove the motion effect, obtain accurate DSA image; Step two, preoperative CT multi-phase image segmentation, extraction, establishment of liver contour, artery, vein and lesion segmentation structure, form three-dimensional liver structure route map; Step three, through projection fusion algorithm, the three-dimensional liver structure route map anatomic structure vascular information is fused in real time in the accurate DSA image, realize image fusion. 2.The DSA platform-based polymorphic real-time multi-modal image accurate fusion method of claim 1, characterized in that: In step one, the sequence image registration uses unsupervised registration network based on MSE as loss, input floating image, fixed image, floating image is DSA image without contrast medium, fixed image is image with contrast medium, the loss function of registration network includes the gray mean square error of the deformed floating image and fixed image after registration and the smoothing loss function of deformation field, the output is the deformation field between the two, complete the registration of two DSA images, form the enhancement of angiography. 3.The DSA platform-based polymorphic real-time multi-modal image accurate fusion method of claim 2, characterized in that: In step one, in the patient's respiratory cycle, the shape of each respiratory period is extracted, and the corresponding respiratory phase state of the DSA image at a certain time point is obtained by analysis; the sequence image is registered, the patient's motion change rule is obtained by combining the anatomic structure position, and the liver tissue in the real-time DSA sequence in the interventional operation is segmented, which can calculate the respiratory motion change rule of the patient during the operation. In the interventional operation, the current DSA image frame can be obtained by calculating the mean square error between it and the key frame of the patient's respiratory cycle, and the real-time respiratory phase is obtained. 4.The DSA platform-based multi-state real-time multi-modal image accurate fusion method of claim 1, characterized in that: In step two, according to the preoperative multi-phase liver enhanced CT arterial phase and venous phase delay image, the shape of liver, lesion position and shape, arterial and venous anatomic structure are extracted, three-dimensional segmentation and visualization are carried out, and the skeleton of the blood vessel mask and gold standard blood vessel / cholangiolar mask is extracted during segmentation. Skeleton, form three-dimensional liver structure route map.

5. The multi-state real-time multi-mode image precise fusion method based on the DSA platform according to claim 1, characterized in that: In step three, according to the space projection relationship between the 2D accurate DSA image and the 3D three-dimensional liver structure route map initialized by the DSA device parameters, the individual at the center of rotation or the center of the world coordinate system o point gets the 2D image on the DSA projection board, the projection transformation and the distance SID from the emission source to the imaging panel and the distance SOD from the emission source to the imaging object as well as the angle LAO / RAO (α) and the angle CRAN / CAUD (β) determine the projection transformation, which is simply written as formula 1:

6. Wherein, μ and υ represent the pixel position on the DSA image, x, y, z represent the point in the world coordinate system.

7. The multi-state real-time multi-modal image precise fusion method based on the DSA platform according to claim 5, characterized in that: In step three, the point in the CT image μ', υ', w' and the transformation relationship of the world coordinate system is represented by T w , T w Also included is the transformation relationship of the patient's body position when CT scanning and when intervention, assuming that the transformation relationship between the two is rigid transformation, the transformation relationship of the CT image and the C-arm world coordinate system is represented by formula 2:

8. In the case that intraoperative CBCT scanning has been completed, the correspondence of the above coordinates is obtained by directly registering the 3D CBCT and CT images T w . 9.The DSA platform-based polymorphic real-time multi-modal image precise fusion method of claim 6, characterized in that: In step three, the angiography image is simulated and generated based on 3D CT image, which is represented by formula 3:

10. wherein, I 0 and I is an initial energy and is an attenuation energy, respectively, l is a penetration path of the X-ray, μ is an attenuation coefficient, calculated from a CT value of the image, μ w is an attenuation coefficient of water. 11.The DSA platform-based polymorphic real-time multi-modal image precise fusion method of claim 7, characterized in that: In step three, the image fusion is performed in the following manner: the three-dimensional liver structure roadmap is set as I , the corresponding artery and vein are A, V , the direct solution is to solve the optimization deformation field f(μ', υ', w') , so that the image or structure after deformation I'= f(μ', υ', w') ° I , the images after projection according to formula 1 and formula 2 are similar to the DSA / X-ray images Tp ° TW(I') ~ DSA . 12.The DSA platform-based multi-state real-time multi-mode image accurate fusion method of claim 1, characterized in that: In step one, the real-time X-ray sequence image is registered using the 2D image registration network based on VoxelMorph; In step two, the V-Net structure network based on multi-resolution multi-head extraction is used to complete image segmentation and blood vessel extraction.

Images