Novel OCT spatial reconstruction method
By combining XRAY angiography images and OCT sequence images, and using deep learning and epipolar geometry algorithms for multi-view matching and correction, a three-dimensional spatial model of the true pose of coronary vessels was reconstructed. This solves the problem that existing technologies cannot reconstruct the internal structure of blood vessels from all angles and provides screening data for vascular malformations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HORIMED TECH CO LTD
- Filing Date
- 2021-06-02
- Publication Date
- 2026-07-10
AI Technical Summary
Current technology cannot achieve full-angle three-dimensional reconstruction of coronary vessels, especially it cannot accurately reconstruct the internal structure of the vessels and the distortion of invisible parts.
By combining XRAY angiography images and OCT sequence images, deep learning is used for multi-view stereo matching, and the matching is corrected by the epipolar geometry algorithm to obtain the three-dimensional vascular skeleton. The skeleton is then rearranged by combining the coronary OCT sequence images to achieve the reconstruction of a three-dimensional spatial model with the true posture.
It enables full-angle three-dimensional reconstruction of coronary vessels, accurately reconstructs the internal structure of the vessels and the distortion of invisible parts, and provides a data basis for screening vascular malformations.
Smart Images
Figure CN115439597B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, and in particular to a novel OCT spatial reconstruction method based on XRAY and OCT images. Background Technology
[0002] Cardiovascular and cerebrovascular diseases are known as the leading cause of death and loss of life. With the continuous improvement of living standards and the increasing abundance of material resources, the proportion of obese individuals is on the rise, and these individuals are precisely the high-risk group for cardiovascular and cerebrovascular diseases. Technological advancements have driven continuous innovation in medical diagnostic techniques, and the rapid development of medical imaging technology has brought convenient measures and effective means for diagnosing various diseases. Imaging techniques such as coronary angiography (XRAY), computed tomography (CT), magnetic resonance imaging (MRI), intravascular ultrasound (IVUS), and optical coherence tomography (OCT) are all widely used in medical diagnosis.
[0003] Using two or more XRAY coronary artery images, spatial three-dimensional information can be reconstructed, thereby restoring the true three-dimensional structure of the coronary arteries. This reconstruction method is called multi-view geometric reconstruction. The spatial three-dimensional vascular model established by this method allows observation of the spatial structure information of the coronary arteries, including vessel direction, diameter, and stenosis, from an external spatial perspective. However, this method can only view external information and is powerless to obtain information within the vessel lumen.
[0004] 3D Reconstruction Based on OCT Sequence Images. This method utilizes multiple sequence images for 3D reconstruction. The acquired image data is optical rotation tomography data, so each image contains information about the internal structure of the blood vessel, from the inner wall to the lumen. The spatial model reconstructed from this sequence of images can reveal intravascular spatial information, including intravascular calcification and vascular occlusion. However, this method cannot reconstruct the spatial orientation of the blood vessel, so the spatial model does not represent the true morphological information.
[0005] The OCT spatial reconstruction method, which combines XRAY angiography images and OCT sequence images, utilizes both methods to detect the external and internal vascular structures respectively, constructing a 3D vascular model with realistic spatial orientation. However, in real-world scenarios, the 3D vascular model constructed using multi-view geometric reconstruction methods can only reveal the spatial structure and orientation of blood vessels from limited angles. Due to the unique orientation of blood vessels, we cannot capture images from all angles. Therefore, using multi-view geometric reconstruction alone cannot reveal whether the invisible parts of the blood vessels are distorted. Thus, improvements are needed to the method of 3D vascular reconstruction using XRAY angiography images and OCT sequence images. Summary of the Invention
[0006] Therefore, the purpose of this invention is to provide a novel OCT spatial reconstruction method. Based on XRAY angiography images and OCT sequence images, deep learning is used for multi-view stereo matching, and an epipolar geometry algorithm is used for matching correction. The three-dimensional vascular skeleton is obtained from the matched and corrected coronary angiography images, and the coronary OCT sequence images are rearranged, thereby realizing a three-dimensional spatial model with realistic posture.
[0007] To achieve the above objectives, the present invention provides a novel OCT spatial reconstruction method, comprising:
[0008] Acquire multiple coronary angiography images from different angles;
[0009] From the coronary angiography images acquired at each angle, coronary angiography images at the same moment in different cardiac cycles are extracted; and the vascular skeleton is extracted based on the coronary angiography images.
[0010] When extracting the vascular skeleton, deep learning is used for multi-view stereo matching at different angles, and epipolar geometry algorithm is used for matching correction. The three-dimensional vascular skeleton is obtained based on the matched and corrected coronary angiography image.
[0011] Obtain coronary artery OCT sequence images and rearrange them in conjunction with the orientation of the three-dimensional vascular skeleton;
[0012] The rearranged image is then volume-rendered to produce a realistic 3D spatial model.
[0013] Preferably, when acquiring multiple coronary angiography images from different angles, a positioning marker is set at the chest position, which is used to track and calibrate the acquired coronary angiography images.
[0014] In any of the above embodiments, the coronary angiography images at different angles are preferably processed using deep learning methods including the following:
[0015] Input the coronary angiography image obtained from any angle into a deep neural network to generate multiple reconstructed result point sets;
[0016] The reconstructed point cloud is generated based on the coronary angiography images obtained from each angle;
[0017] Based on the preset consistency loss function and the distance between any two point cloud sets, the multi-angle reconstruction result point cloud after consistency optimization is obtained.
[0018] In any of the above embodiments, preferably, the deep neural network has a preset conditional generation model that generates multiple reconstruction results based on multiple added noises; the conditional generation model includes the diversity constraints shown in the following formula:
[0019] loss div =max(0,||r1-r2||2-αEMD(S1,S2))
[0020] Where: r1 and r2 represent any two noises added to the coronary angiography image at the same angle; EMD(S1,S2) represents the bulldozing distance between the two reconstructed result point sets S1 and S2; α is the weight.
[0021] In any of the above embodiments, the consistency loss function is preferably calculated using the following formula:
[0022]
[0023] Where: n represents the number of input multi-angle images, i and j represent any two angle images, and CD(S) represents the number of input multi-angle images. i ,S j S represents the set of multiple reconstructed point clouds generated from images at any two angles. i and S j The chamfer distance between them.
[0024] In any of the above embodiments, the method for performing matching correction using an epipolar geometry algorithm includes the following:
[0025] In the reconstructed point clouds from any two angles, the 3D coordinates of key points are extracted. Based on the principles of multi-view geometry and the transformation relationship between physical coordinates and pixel coordinates, the following formula is used to calculate whether the coronary angiography images from any two angles match:
[0026]
[0027] Where P i jT Let P be the projection matrix. i In the j-th row, (x1,y1) and (x2,y2) are the coordinates of the same object projection point in the two angle images, respectively.
[0028] In any of the above embodiments, when rearranging the acquired coronary OCT sequence images in conjunction with the orientation of the three-dimensional vascular skeleton, the following method is used:
[0029] The envelope of the three-dimensional vascular skeleton is extracted; based on the number of image frames of the acquired coronary OCT sequence images, equal-interval interpolation is performed on the envelope; the sequence of the coronary OCT sequence images is matched with each interpolation on the envelope; and the coronary OCT sequence images are sorted according to their corresponding positions.
[0030] In any of the above embodiments, preferably, the coordinates of the obtained reconstructed point cloud are labeled using the intrinsic and extrinsic parameter matrix of the camera.
[0031] The novel OCT spatial reconstruction method disclosed in this application has at least the following advantages compared to existing technologies:
[0032] 1. The novel OCT spatial reconstruction method provided in this application combines deep learning with epipolar geometry algorithms. It utilizes deep learning algorithms to perform stereo matching on input multi-angle coronary angiography images, and then uses epipolar geometry algorithms for correction. This not only enables the prediction of invisible parts, but also ensures the accuracy of the three-dimensional reconstruction of visible parts, providing a data foundation for the subsequent screening of vascular malformations and other conditions.
[0033] 2. The novel OCT spatial reconstruction method provided in this application constructs multiple reconstruction result point clouds for coronary angiography images at any angle using a conditional generation model based on deep learning. After consistency optimization, a uniquely determined multi-angle reconstruction result point cloud is obtained. By using the multi-angle reconstruction result point clouds obtained from coronary angiography images at any two angles, multiple noises are added using the conditional generation model to generate multiple reconstruction results. This provides multiple prediction possibilities for invisible parts and provides a data foundation for screening local vascular malformations.
[0034] 3. The novel OCT spatial reconstruction method provided in this application uses a mark to track and calibrate the acquired image by attaching a mark to the patient's chest area in order to accurately obtain matching parameters and thus obtain an accurate spatial geometric structure.
[0035] 4. The novel OCT spatial reconstruction method provided in this application extracts the envelope line from the three-dimensional vascular skeleton; and facilitates the adjustment of coronary OCT sequence images by performing equidistant interpolation on the envelope line. Attached Figure Description
[0036] Figure 1 This is a schematic flowchart of a novel OCT spatial reconstruction method according to the present invention.
[0037] Figure 2(a) shows the point set of reconstruction results from any angle of a novel OCT spatial reconstruction method of the present invention;
[0038] Figure 2(b) shows the point set of the reconstruction result from another angle of the novel OCT spatial reconstruction method of the present invention;
[0039] Figure 3(a) is a schematic diagram of the vascular skeleton extracted by the present invention from one angle;
[0040] Figure 3(b) is a schematic diagram of the vascular skeleton extracted by the present invention from another angle;
[0041] Figure 4(a) is a schematic diagram of the vascular skeleton extracted by the present invention at one angle in a three-dimensional coordinate environment;
[0042] Figure 4(b) is a schematic diagram of the vascular skeleton extracted by the present invention from another angle in a three-dimensional coordinate environment;
[0043] Figure 5 This is a schematic diagram of coronary OCT sequence image rearrangement for a novel OCT spatial reconstruction method according to the present invention;
[0044] Figure 6 This is a rendered three-dimensional spatial model of the true pose of blood vessels, based on a novel OCT spatial reconstruction method of the present invention. Detailed Implementation
[0045] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0046] like Figure 1 As shown, one embodiment of the present invention provides a novel OCT spatial reconstruction method, comprising:
[0047] S1. Acquire multiple coronary angiography images from different angles; preferably, when acquiring multiple coronary angiography images from different angles, a positioning marker is set at the chest position, the positioning marker being used to track and calibrate the acquired coronary angiography images.
[0048] In one embodiment of the invention, before the experiment, a standard marker is established on the patient's chest area. A large C-arm angiography machine is used to continuously image the patient's coronary arteries after the contrast agent is injected, and then continuous imaging is performed from different angles.
[0049] S2. Extract coronary angiography images at the same moment in different cardiac cycles from the coronary angiography images acquired at each angle; and extract the vascular skeleton based on the coronary angiography images.
[0050] As shown in Figure 2, the extraction of the vascular skeleton includes coronary angiography images from different angles; deep learning is used for multi-view stereo matching, and epipolar geometry algorithm is used for matching correction. The three-dimensional vascular skeleton is obtained based on the matched and corrected coronary angiography images.
[0051] Furthermore, for coronary angiography images from different angles, deep learning was employed, including the following methods:
[0052] Input the coronary angiography image obtained from any angle into a deep neural network to generate multiple reconstructed result point sets;
[0053] The reconstructed point cloud is generated based on the coronary angiography images obtained from each angle;
[0054] Based on the preset consistency loss function and the distance between any two point cloud sets, the multi-angle reconstruction result point cloud after consistency optimization is obtained.
[0055] In any of the above embodiments, preferably, the deep neural network has a preset conditional generation model that generates multiple reconstruction results based on multiple added noises; the conditional generation model includes the diversity constraints shown in the following formula:
[0056] loss div =max(0,||r1-r2||2-αEMD(S1,S2))
[0057] Where: r1 and r2 represent any two noises added to the coronary angiography image at the same angle; EMD(S1,S2) represents the bulldozing distance between the two reconstructed result point sets S1 and S2; α is the weight.
[0058] In any of the above embodiments, the consistency loss function is preferably calculated using the following formula:
[0059]
[0060] Where: n represents the number of input multi-angle images, i and j represent any two angle images, and CD(S) represents the number of input multi-angle images. i ,S j S represents the set of multiple reconstructed point clouds generated from images at any two angles. i and S j The chamfer distance between them.
[0061] The coordinates of the reconstructed point cloud are labeled using the camera's intrinsic and extrinsic parameter matrices.
[0062] When using epipolar geometry algorithms for matching correction, the following methods are included:
[0063] In the reconstructed point clouds from any two angles, the 3D coordinates of key points are extracted. Based on the principles of multi-view geometry and the transformation relationship between physical coordinates and pixel coordinates, the following formula is used to calculate whether the coronary angiography images from any two angles match:
[0064]
[0065] Where P i jT Let P be the projection matrix. i In the j-th row, (x1,y1) and (x2,y2) are the coordinates of the same object projection point in the two angle images, respectively.
[0066] S3. Obtain coronary artery OCT sequence images and rearrange them in combination with the direction of the three-dimensional vascular skeleton;
[0067] When rearranging the acquired coronary OCT sequence images in conjunction with the orientation of the three-dimensional vascular skeleton, the following method is used:
[0068] Specifically, as shown in Figures 3 and 4, for spatial skeleton reconstruction: before performing spatial skeleton reconstruction, distortion correction, motion displacement recovery, feature extraction, and feature matching are required on the images used. Feature extraction includes: vessel boundary segmentation, vessel midline extraction, and bifurcation point extraction. Feature matching includes: vessel segment matching and vessel point matching, where vessel point matching can rely on epipolar constraint matching or SIFT matching based on texture features. After establishing the matching relationship, the coordinates of three-dimensional points in space can be calculated using the obtained geometric parameters and geometric principles. Then, the spatial skeleton is obtained by fitting continuous spatial points.
[0069] like Figure 5 As shown, the envelope of the three-dimensional vascular skeleton is extracted; based on the number of image frames of the acquired coronary OCT sequence images, equal-interval interpolation is performed on the envelope; the sequence of coronary OCT sequence images is matched with each interpolation on the envelope; the coronary OCT sequence images are then sorted according to their corresponding positions. The acquired OCT sequence image data, combined with the spatial skeleton orientation including changes in angle and curvature, is then rearranged.
[0070] like Figure 6 As shown, S4, the rearranged image is volume rendered to produce a three-dimensional spatial model with a realistic pose.
[0071] Volume rendering and reconstruction can be performed using either ray casting or texture rendering. When performing volume rendering, it's necessary to set the required transparency and color transfer functions. The gradient transfer function can be selected and set according to needs. The scene background for volume rendering can be set to white or black.
[0072] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A novel OCT spatial reconstruction method, characterized in that, include: Acquire multiple coronary angiography images from different angles; From the coronary angiography images acquired at each angle, coronary angiography images at the same moment in different cardiac cycles are extracted; and the vascular skeleton is extracted based on the coronary angiography images. When extracting the vascular skeleton, deep learning is used to perform multi-view stereo matching on coronary angiography images from different angles; and an epipolar geometry algorithm is used to perform matching correction. The three-dimensional vascular skeleton is obtained based on the matched and corrected coronary angiography images. Among these methods, deep learning is used for multi-view stereo matching of coronary angiography images from different angles, including: The coronary angiography images acquired at each angle are input into a deep neural network to generate multiple sets of reconstruction result points; Based on the preset consistency loss function and the distance between any two point cloud sets, the multi-angle reconstruction result point cloud after consistency optimization is obtained; The deep neural network generates a model with preset conditions, which generates multiple reconstruction results based on multiple noises added. The conditional generation model includes the diversity constraints shown in the following formula: in: and These represent any two noises added to coronary angiography images at the same angle; This represents the two reconstructed result point clouds generated. and The bulldozing distance; As weight; The consistency loss function is calculated using the following formula: ； in: This represents the number of input multi-angle images. and They represent The first multi-angle image The and the first indivual, Indicates the first The and the first The point cloud of the reconstruction result generated from the image at each angle and The chamfer distance between them; Obtain coronary artery OCT sequence images and rearrange them in conjunction with the orientation of the three-dimensional vascular skeleton; The rearranged image is then volume-rendered to produce a realistic 3D spatial model.
2. The novel OCT spatial reconstruction method according to claim 1, characterized in that, When acquiring multiple coronary angiography images from different angles, a positioning marker is set at the chest position. The positioning marker is used to track and calibrate the acquired coronary angiography images.
3. The novel OCT spatial reconstruction method according to claim 1, characterized in that, When using epipolar geometry algorithms for matching correction, the following methods are included: In the reconstructed point clouds from any two angles, the 3D coordinates of key points are extracted. Based on the principles of multi-view geometry and the transformation relationship between physical coordinates and pixel coordinates, the following formula is used to calculate whether the coronary angiography images from any two angles match: in Projection matrix The OK, , These are the coordinates of the same object's projection point in the two angle images.
4. The novel OCT spatial reconstruction method according to claim 1, characterized in that, When rearranging the acquired coronary OCT sequence images in conjunction with the orientation of the three-dimensional vascular skeleton, the following method is used: Extract the envelope of the three-dimensional vascular skeleton; Based on the number of image frames in the acquired coronary OCT sequence images, equal-interval interpolation is performed on the envelope; The sequence of coronary OCT images is mapped to each interpolation value along the envelope. The coronary artery OCT sequence images were sorted according to their corresponding locations.
5. A novel OCT spatial reconstruction method according to claim 3, characterized in that, The coordinates of the reconstructed point cloud are labeled using the camera's intrinsic and extrinsic parameter matrices.