Cardiovascular three-dimensional reconstruction system based on multi-modal image data fusion

Abstract

The present application relates to the technical field of three-dimensional modeling, in particular to a cardiovascular three-dimensional reconstruction system based on multi-modal image data fusion. The present application determines the composite matching cost between the corresponding two points based on the position difference between the three-dimensional prediction contour point and the two-dimensional observation edge point at each time, the spatial constraint of anatomical structure and the consistency of direction characteristics, and determines the best matching pair set between the three-dimensional prediction contour point and the two-dimensional observation edge point. Based on the composite matching cost of the matching point pair in the best matching pair set, the geometric error cost after removing the influence of the spatial constraint of anatomical structure and the spatial constraint of anatomical structure are removed, the projection deviation between the matching point pair is adjusted, the position of the three-dimensional prediction contour point of the three-dimensional reference model is updated, and the three-dimensional reconstruction model is obtained. The present application realizes accurate reconstruction of the cardiovascular model under the irregular heart rhythm of atrial fibrillation by using multi-dimensional constraint composite cost accurate matching and based on the double constraint update strategy of geometric error and anatomical resistance.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and more specifically to a cardiovascular 3D reconstruction system based on multimodal image data fusion. Background Technology

[0002] In catheter ablation surgery for atrial fibrillation (AF), a key step in achieving pulmonary vein isolation lies in the precise fusion of a preoperative high-resolution 3D CT model with intraoperative 2D X-ray fluoroscopy images to provide real-time, accurate 3D anatomical navigation. However, the heart is a continuously dynamically deforming organ, and its motion is regulated by complex electromechanical coupling mechanisms. Under AF rhythm, the highly irregular RR intervals result in different filling times and subsequent mechanical contraction amplitudes for each heartbeat. This makes it difficult for static CT models based on single-phase reconstruction to morphologically match the rapidly changing dynamic cardiac contours during surgery, posing a core technical challenge.

[0003] The prior art, patent document CN101799935A, discloses a dynamic three-dimensional reconstruction method for single-arm X-ray angiography images. Specifically, it involves extracting and correcting the overall translational displacement caused by respiration by analyzing the motion of non-cardiac markers in the image sequence, using a dynamic cardiovascular model for initial guidance, and then combining backprojection error and pericardial constraint for individualized optimization to estimate and compensate for complex local deformations caused by cardiac pulsation. However, in atrial fibrillation, the highly irregular RR intervals result in strong randomness in ventricular filling time and subsequent mechanical contraction amplitude, making it impossible to accurately match each pulsation pattern using only respiratory correction and simple pericardial constraint. This easily leads to severe anatomical alignment deviations between intraoperative images and the model. Simultaneously, irregular cardiac pulsation exacerbates uneven contrast agent distribution or flow artifacts, making it easier to form false edges on the image that are inconsistent with the actual anatomical boundaries, resulting in poor cardiovascular three-dimensional reconstruction. Summary of the Invention

[0004] To address the technical problem of poor cardiovascular 3D reconstruction results caused by anatomical deformation due to irregular cardiac contractions in atrial fibrillation scenarios, the present invention aims to provide a cardiovascular 3D reconstruction system based on multimodal image data fusion. The specific technical solution adopted is as follows: This invention proposes a cardiovascular three-dimensional reconstruction system based on multimodal image data fusion, the system comprising: The data acquisition module is used to acquire a preoperative three-dimensional cardiac anatomical model of the subject and fluoroscopic images at various times during the surgical period; the model is composed of different grid units; The initialization module is used to perform initial rigid registration of the preoperative three-dimensional cardiac anatomy model to obtain the three-dimensional reference model at the initial moment of the surgical period, and to use the three-dimensional reconstruction model of the previous moment of each non-initial moment of the surgical period as the three-dimensional reference model. The matching analysis module is used to determine the three-dimensional predicted contour points at each time step based on the projection relationship of the three-dimensional reference model relative to the imaging viewpoint, and to extract the two-dimensional observation edge points of the perspective image. Based on the positional difference between each three-dimensional predicted contour point and each two-dimensional observation edge point at each time step, the spatial constraints of the anatomical structure, and the consistency of directional features, the module determines the composite matching cost between corresponding two points and determines the best matching pair set between the three-dimensional predicted contour points and the two-dimensional observation edge points at each time step. The model update module is used to adjust the projection deviation between the matching point pairs based on the composite matching cost of the best matching pair set after removing the influence of the anatomical structure spatial constraints, and to update the position of the three-dimensional predicted contour points of the three-dimensional reference model, so as to obtain the three-dimensional reconstruction model at each time step.

[0005] Further, determining the three-dimensional predicted contour points at each time step includes: Obtain the camera's optical center point; for each mesh cell in the 3D reference model at each moment, obtain the unit vector of the direction vector from the camera's optical center point to the center point of the mesh cell, denoted as the unit line-of-sight vector; Obtain the unit normal vector of the mesh cell pointing to the outside of the 3D reference model; take the dot product of the unit view vector and the unit normal vector as the visibility index of the mesh cell; extract the common edge connecting two adjacent mesh cells with opposite visibility indexes from the 3D reference model as the 3D contour line. Sample all three-dimensional contour lines on the three-dimensional reference model at each time step to obtain the three-dimensional predicted contour points.

[0006] Further, determining the composite matching cost between the two corresponding points includes: The surface of the preoperative three-dimensional heart model contains several anatomical anchor points and vertices connecting different mesh units; the geodesic distance from each vertex of the preoperative three-dimensional heart model to the nearest anatomical anchor point is obtained, and the geodesic distance is negatively correlated and normalized to obtain the anatomical deformation resistance coefficient of the corresponding vertex. The unit normal vector of the grid cell containing each 3D predicted contour point at each time step is projected onto the 2D imaging plane and normalized to serve as the 2D projection normal vector. For each 3D predicted contour point and each 2D observed edge point at each time step, the directional deviation penalty coefficient between the corresponding two points is determined based on the directional consistency between the 2D projection normal vector of the 3D predicted contour point and the unit gradient vector of the 2D observed edge point. Using the anatomical deformation resistance coefficient of the corresponding vertex of the three-dimensional predicted contour point on the preoperative three-dimensional cardiac anatomy model, the distance between the two-dimensional projection normal vector of the three-dimensional predicted contour point and the unit gradient vector of the two-dimensional observation edge point is adjusted to obtain the distance cost between the two corresponding points. The directional deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point is weighted and summed with the distance cost to obtain the composite matching cost between the two corresponding points.

[0007] Further, determining the directional deviation penalty coefficient between two corresponding points includes: For each 3D predicted contour point and each 2D observed edge point at each time step, the angle between the 2D projection normal vector of the 3D predicted contour point and the unit gradient vector of the 2D observed edge point is used as the directional deviation angle between the two points. The cardiac cycle filling index is determined based on the time-varying physiological state of the heart at each moment, and the normal deviation threshold at each moment is calculated. If the directional deviation angle is less than or equal to the normal deviation threshold, the deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point is set to 1; if the directional deviation angle is greater than the normal deviation threshold, the directional deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point is determined based on the difference between the directional deviation angle and the normal deviation threshold.

[0008] Furthermore, the determination of the cardiac cycle filling index based on the time-varying physiological state of the heart at each moment includes: Acquire surface electrocardiogram signals during the surgical period, and perform R-wave peak detection on the signals to obtain R-wave moments; take the time interval between each two adjacent R-wave moments as a cardiac cycle; Calculate the ratio of the duration of the cardiac cycle preceding each moment to the average duration of the number of adjacent cardiac cycles preceding that moment, and use it as the relative fullness duration ratio. The time difference between each moment and the start moment of its corresponding cardiac cycle is taken as the cardiac phase delay; if the cardiac phase delay is less than or equal to a preset contraction cutoff threshold, the phase weighting factor is determined to be a first preset value; if the cardiac phase delay is greater than the preset contraction cutoff threshold, the phase weighting factor is determined to be a second preset value. The product of the relative filling duration ratio at each moment and the phase weighting factor is used as the cardiac cycle filling index.

[0009] Further, determining the optimal set of matching pairs between the three-dimensional predicted contour points and the two-dimensional observed edge points at each time step includes: A bipartite graph model is constructed based on the three-dimensional predicted contour points and two-dimensional observed edge points at each time step. The weight of the edge connecting a three-dimensional predicted contour point and a two-dimensional observed edge point in the model is set as the composite matching cost between the two points. The minimum weight matching of the bipartite graph model is solved using the Hungarian algorithm to obtain the optimal matching pair set consisting of several matching point pairs; wherein, the matching point pair includes a three-dimensional predicted contour point and a two-dimensional observed edge point.

[0010] Furthermore, the method for obtaining the geometric error cost includes: For each time step, the distance between the two-dimensional projection normal vector of the three-dimensional predicted contour point and the unit gradient vector of the two-dimensional observed edge point in each matching point pair in the best matching pair set is weighted and summed with the direction deviation penalty coefficient to obtain the local error cost between the corresponding two points. Calculate the arithmetic mean of the local error costs of all matching point pairs in the best matching pair set as the geometric error cost at each time step.

[0011] Furthermore, obtaining the three-dimensional reconstruction model at each time step includes: For each matching point pair in the best matching pair set at each time moment, the two-dimensional observation edge points in the matching point pair are mapped to the three-dimensional physical point coordinates on the imaging plane under the surgical space coordinate system using the camera imaging projection parameters; the ray pointing from the camera optical center to the three-dimensional physical point coordinates is taken as the target line-of-sight ray; the orthogonal projection point of the three-dimensional predicted contour points in the matching point pair on the target line-of-sight ray is taken as the spatial foot. The direction from the 3D predicted contour point of the matching point centering to the spatial perpendicular foot is taken as the direction of the line of sight projection traction vector, and the distance between the 3D predicted contour point of the matching point centering and the spatial perpendicular foot is taken as the magnitude of the line of sight projection traction vector. The geometric error costs of the matching point pairs are negatively correlated and normalized to obtain the global confidence weight; The product of the difference between constant 1 and the anatomical deformation resistance coefficient of the corresponding vertex of the three-dimensional predicted contour point in the matching pair on the preoperative three-dimensional cardiac anatomy model and the global confidence weight is calculated. The product is then multiplied by the line-of-sight projection traction vector to obtain the restricted deformation displacement vector. The position coordinates of the matched 3D predicted contour point are added to the components of the same dimension of the constrained deformation displacement vector to update the position coordinates of the 3D predicted contour point and obtain the reconstructed position. A 3D reconstruction model is constructed from the reconstructed positions of the 3D predicted contour points of the 3D reference model at each time step.

[0012] Furthermore, the extraction of two-dimensional observation edge points from the perspective image includes: Edge detection is performed on the perspective image at each time step, and pixels with gradient magnitudes greater than a preset gradient threshold are selected as edge candidate points; the edge candidate points are then sparsely sampled to obtain the two-dimensional observation edge points at the corresponding time step.

[0013] Furthermore, the first preset value is greater than the second preset value.

[0014] The present invention has the following beneficial effects: Firstly, the initialization module provides a starting point for spatiotemporal alignment, using the three-dimensional reconstruction model of the previous moment as a three-dimensional reference model. By utilizing the temporal continuity of cardiac motion, the current deformation prediction can inherit the best estimation state of the previous moment, thereby achieving continuous and smooth tracking of the complex motion trajectory of the heart under extremely irregular atrial fibrillation.

[0015] Secondly, by integrating the positional differences in geometric distance constraints between 3D predicted contour points and 2D observed edge points, the spatial constraints of anatomical structures that differentiate the deformation capabilities of different regions, and the consistency of directional features that can screen for anisotropic physiological motion, the composite matching cost can accurately distinguish between true anatomical edges and non-anatomical artifacts. This allows for the identification and elimination of erroneous candidates that do not conform to anatomical topological logic or physiological motion directions during point-to-point matching. By globally selecting the optimal set of matching pairs that best conforms to biomechanical principles, a highly reliable geometric reference is provided for subsequent accurate model deformation updates, fundamentally solving the matching failure problem caused by anatomical deformation uncertainty under atrial fibrillation rhythms.

[0016] Thirdly, by removing the influence of anatomical spatial constraints from the composite matching cost, the true pure geometric visual residuals between matching point pairs, i.e., geometric error costs, are restored. Combined with anatomical spatial constraints, differentiated local deformation restrictions are applied to different regions on the model surface. This ensures that the 3D reconstruction model can accurately capture the real heartbeat characteristics while maintaining a robust anatomical topology, effectively avoiding non-anatomical deformations caused by artifact interference or local high filling, thereby significantly improving the accuracy and robustness of cardiovascular 3D reconstruction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1This is a system structure diagram of a cardiovascular three-dimensional reconstruction system based on multimodal image data fusion, provided in one embodiment of the present invention. Figure 2 This is a system structure diagram of a composite matching cost provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of a computer device for a cardiovascular three-dimensional reconstruction device based on multimodal image data fusion, provided as an embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a cardiovascular three-dimensional reconstruction system based on multimodal image data fusion proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] The following description, in conjunction with the accompanying drawings, details a specific scheme for a cardiovascular three-dimensional reconstruction system based on multimodal image data fusion provided by the present invention.

[0022] Example 1: Please see Figure 1 The diagram illustrates a system block diagram of a cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to an embodiment of the present invention. The system includes: a data acquisition module 110, an initialization module 120, a matching analysis module 130, and a model update module 140.

[0023] The data acquisition module 110 is used to acquire the preoperative three-dimensional cardiac anatomical model of the subject and the fluoroscopic images at various times during the surgical period; the model is composed of different grid units.

[0024] Preoperative cardiac CT images of the subject are acquired, and a preoperative three-dimensional cardiac anatomy model is generated using image segmentation and three-dimensional reconstruction algorithms. This model is specifically represented as a discretized three-dimensional surface mesh structure, composed of a large number of tightly connected mesh units. In this embodiment, the mesh unit is typically a triangular facet, each defined by three vertices and their connecting edges, which together accurately fit the complex anatomical surface of the left atrium and its associated structures (including the opening regions of the four pulmonary veins and the left atrial appendage). The preoperative three-dimensional cardiac anatomy model is typically established in a standard anatomical coordinate system defined based on anatomical features, with the centroid of the left atrium as the origin, and the axes of the three-dimensional Cartesian coordinate system aligned with the anatomical axes of the human body (i.e., the left-right direction is the X-axis, the anteroposterior direction is the Y-axis, and the head-to-foot direction is the Z-axis).

[0025] An X-ray fluoroscopic imaging system (such as a C-arm) is used to acquire X-ray fluoroscopic images at a sampling rate of 10 Hz for each moment during the surgical procedure. To improve the signal-to-noise ratio in subsequent processing, Gaussian filtering is applied to the X-ray fluoroscopic images to obtain the corresponding fluoroscopic images at each moment. The surgical period refers to the time from the start of contrast agent injection to the end of the ablation procedure.

[0026] The initialization module 120 is used to perform initial rigid registration of the preoperative three-dimensional cardiac anatomy model to obtain the three-dimensional reference model at the initial moment of the surgical period, and to use the three-dimensional reconstruction model of the previous moment of each non-initial moment of the surgical period as the three-dimensional reference model.

[0027] At the initial moment of surgery, the preoperative 3D cardiac anatomy model exists in an independent scanning coordinate system, exhibiting significant spatial pose deviations from the surgical space coordinate system. Initial rigid registration provides a roughly correct initial model for subsequent non-rigid deformations. The method involves identifying rigid anatomical landmarks (such as vertebral bodies or tracheal carina) in fluoroscopic images and rigidly registering and aligning the standard anatomical coordinate system of the preoperative 3D cardiac anatomy model with the imaging coordinate system of the intraoperative X-ray fluoroscopic imaging equipment. This initial rigid registration transforms the preoperative model into a unified surgical space coordinate system, aligning the model's pose with the intraoperative image viewpoint. All subsequent model deformation and update calculations are performed within this unified surgical space coordinate system. The origin of the surgical space coordinate system is the C-arm rotation center, the Z-axis points to the detector, and the X and Y axes lie within the imaging plane. Rigid registration is a well-known technique in the field and will not be elaborated further here.

[0028] For moments after the initial moment during the surgical period (i.e., non-initial moments), the heart has undergone continuous nonlinear deformation with the cardiac cycle. By using the three-dimensional reconstruction model of the previous moment as the current three-dimensional reference model and utilizing the temporal continuity of cardiac motion, the current deformation prediction can inherit the best estimation state of the previous moment, thereby significantly improving the smoothness and computational efficiency of dynamic tracking.

[0029] It should be noted that if the fluoroscopic image field of view is small and does not include rigid anatomical landmarks such as the spine, the user will be prompted to manually specify the initial correspondence between the preoperative 3D cardiac anatomy model and the intraoperative fluoroscopic image. Alternatively, the known spatial position of the catheter electrode can be used as an auxiliary registration basis, or a new fluoroscopic image containing rigid anatomical landmarks such as the spine can be acquired.

[0030] The matching analysis module 130 is used to determine the three-dimensional predicted contour points at each time step based on the projection relationship of the three-dimensional reference model relative to the imaging viewpoint, and to extract the two-dimensional observation edge points of the perspective image; based on the positional difference between each three-dimensional predicted contour point and each two-dimensional observation edge point at each time step, the spatial constraints of the anatomical structure, and the consistency of directional features, it determines the composite matching cost between corresponding two points, and determines the optimal matching pair set between the three-dimensional predicted contour points and the two-dimensional observation edge points at each time step.

[0031] The most prominent geometric contour features from the current perspective—namely, the 3D predicted contour points and the 2D observed edge points with strong edge features—are used as the observation benchmarks for the actual anatomical structures. The positional difference between the 3D predicted contour points and the 2D observed edge points forms the basic geometric distance constraint. The anatomical structure spatial constraint considers the differences in deformation capacity of different anatomical regions (such as the relatively fixed pulmonary vein anchorage area and the more mobile atrial appendage free area). The consistency of directional features ensures that deformation conforms to the anisotropic laws of physiological motion. The composite matching cost constructed by combining the above three factors can effectively distinguish between real anatomical edges and non-anatomical artifacts (such as contrast agent flow artifacts). Furthermore, during point-to-point matching, erroneous candidates that do not conform to anatomical topological logic or physiological motion direction are identified and eliminated. The best matching pair set that best conforms to biomechanical laws is selected globally, providing a highly reliable geometric reference for subsequent accurate model deformation updates.

[0032] The model update module 140 is used to adjust the projection deviation between the matching point pairs based on the composite matching cost of the best matching pair set after removing the influence of the anatomical structure spatial constraint and the geometric error cost and anatomical structure spatial constraint, and update the position of the three-dimensional predicted contour points of the three-dimensional reference model to obtain the three-dimensional reconstruction model at each time step.

[0033] Since anatomical spatial constraints are artificial biases introduced to force priority registration of anchorage regions during the matching search phase, their values ​​are significantly amplified in high-resistance areas such as the pulmonary vein region, easily misclassifying normal anchorage matches as high-error artifacts. Therefore, by removing the influence of anatomical spatial constraints from the composite matching cost, the true pure geometric visual residuals between matching point pairs, i.e., the geometric error cost, can be restored. Subsequently, by combining anatomical spatial constraints with differential local deformation restrictions applied to different regions of the model surface, this dual-constraint strategy ensures that the reconstructed mesh model can accurately capture the characteristics of real heartbeats while maintaining a robust anatomical topology, effectively avoiding non-anatomical deformations caused by artifact interference or local high weights.

[0034] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining three-dimensional predicted contour points includes: obtaining the camera optical center point; for each mesh cell in the three-dimensional reference model at each time moment, obtaining the unit vector of the direction vector pointing from the camera optical center point to the center point of the mesh cell, denoted as the unit line-of-sight vector; obtaining the unit normal vector pointing from the mesh cell to the outside of the three-dimensional reference model; using the dot product of the unit line-of-sight vector and the unit normal vector as the visibility index of the mesh cell; extracting the common edge connecting two adjacent mesh cells with visibility indexes of opposite signs from the three-dimensional reference model as the three-dimensional contour line; sampling all three-dimensional contour lines on the three-dimensional reference model at each time moment to obtain three-dimensional predicted contour points.

[0035] It should be noted that the camera's optical center point refers to the spatial physical location of the X-ray source emission point in an X-ray fluoroscopic imaging system. Based on the principle of backplane culling in 3D graphics, a visibility criterion is determined; a value greater than 0 indicates that the mesh cell is facing away from the camera and is in the observer's blind spot; a value greater than 0 indicates that the mesh cell is facing the camera and is within the observer's visible range. The 3D contour line is the boundary between the front and back sides of the model, corresponding to the most prominent outer contour feature in the projection. In this embodiment of the invention, all 3D contour lines extracted from the 3D reference model at each moment are sampled at equal intervals, with the sampling interval set to 1 mm, that is, a 3D predicted contour point is taken every 1 mm.

[0036] Preferably, in some possible implementations of this invention, the method for obtaining two-dimensional observation edge points includes: performing edge detection on the perspective image at each time step, selecting pixels with gradient magnitudes greater than a preset gradient threshold as candidate edge points; and performing sparse sampling on the candidate edge points to obtain the two-dimensional observation edge points at the corresponding time step. It should be noted that this embodiment uses the Sobel operator to perform edge detection on the perspective image to obtain the gradient magnitude of each pixel. Two-dimensional observation edge points are locations with significant anatomical contour features (such as the outline of a heart or the boundary of a contrast agent). To reduce the complexity of subsequent matching calculations and maintain the uniformity of feature distribution, sparse sampling is performed on the extracted candidate edge points. In this embodiment, one two-dimensional observation edge point is selected every three pixels. The implementer can set the sampling interval according to specific circumstances.

[0037] In this embodiment of the invention, the preset gradient threshold is usually adaptively determined based on the global grayscale statistical characteristics of the perspective image at each time moment. The preset gradient threshold at each time moment is 1.5 times the average gradient magnitude of all pixels in the projection image at each time moment. The implementer can set it according to the specific situation.

[0038] Please see Figure 2 The diagram illustrates a system structure diagram of a composite matching cost provided by an embodiment of the present invention, including: a dissection resistance analysis unit 131, a direction penalty unit 132, and a cost analysis unit 133.

[0039] The anatomical resistance analysis unit 131 is used to analyze the surface of the preoperative three-dimensional heart model, which contains several anatomical anchor points and vertices connecting different mesh units. The geodesic distance from each vertex of the preoperative three-dimensional heart model to the nearest anatomical anchor point is obtained, and the geodesic distance is negatively correlated and normalized to obtain the anatomical deformation resistance coefficient of the corresponding vertex.

[0040] It should be noted that several anatomical anchor points are pre-defined on the surface of the preoperative three-dimensional cardiac anatomical model through manual annotation or automatic feature extraction algorithms. These feature points need to be relatively fixed in position during the cardiac cycle with minimal deformation. In this embodiment of the invention, the center point of the central region of the four pulmonary vein openings on the posterior wall of the left atrium is used as the anatomical anchor point. The geodesic distance from each vertex of the preoperative three-dimensional cardiac model to the nearest anatomical anchor point is obtained using a fast travel algorithm, which can characterize the mechanical constraint strength propagating along the atrial wall. Simultaneously, based on anatomical principles, the pulmonary vein opening region has extremely high mechanical stability due to its close connection with the mediastinum; as the distance increases, the atrial wall gradually becomes more detached. Therefore, the geodesic distance is negatively correlated and normalized to obtain the anatomical deformation resistance coefficient, which represents the ability of the vertex position to undergo spatial deformation. The anatomical deformation resistance coefficient represents the spatial constraint of the anatomical structure at the vertex position in the model. The larger this value, the closer the vertex is to the anchor structure, and the smaller the degree of deformation allowed.

[0041] In this embodiment of the invention, the ratio of the geodesic distance corresponding to each vertex of the preoperative three-dimensional cardiac model to a preset spatial attenuation constant is calculated. The negative of this ratio is used as the exponent of an exponential function with the natural constant as the base, thereby achieving negative correlation and normalization of the geodesic distance to obtain the anatomical deformation resistance coefficient. The preset spatial attenuation constant characterizes the rate attenuation of the anatomical deformation constraint on the cardiac surface with increasing distance. Geodesic distances from the left atrial appendage apex (free point) to the nearest anatomical anchor point were collected from preoperative 3D cardiac models of different individuals. The arithmetic mean of these geodesic distances was calculated. Based on the aforementioned method for calculating the anatomical deformation resistance coefficient, it was determined that when the geodesic distance reaches this arithmetic mean, the resistance coefficient should decay to a minimum value (e.g., 0.05, indicating almost no constraint). By inversely solving the formula for calculating the anatomical deformation resistance coefficient, a preset spatial attenuation constant was obtained. Based on the calculation results and combined with engineering experience, a preset spatial attenuation constant of 25 mm was preferably set to accommodate the atrial anatomical dimensions of most atrial fibrillation patients.

[0042] The orientation penalty unit 132 is used to project the unit normal vector of the grid cell containing each 3D predicted contour point at each time step onto the 2D imaging plane and normalize it as a 2D projection normal vector. For each 3D predicted contour point and each 2D observed edge point at each time step, the orientation deviation penalty coefficient between the corresponding two points is determined based on the orientation consistency between the 2D projection normal vector of the 3D predicted contour point and the unit gradient vector of the 2D observed edge point.

[0043] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the directional deviation penalty coefficient includes: for each three-dimensional predicted contour point and each two-dimensional observed edge point at each time moment, taking the angle between the two-dimensional projection normal vector of the three-dimensional predicted contour point and the unit gradient vector of the two-dimensional observed edge point as the directional deviation angle between the two corresponding points; determining the cardiac cycle filling index based on the time-varying physiological state of the heart at each time moment, and calculating the normal deviation threshold at each time moment; if the directional deviation angle is less than or equal to the normal deviation threshold, then setting the deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point to 1; if the directional deviation angle is greater than the normal deviation threshold, then determining the directional deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point based on the difference between the directional deviation angle and the normal deviation threshold.

[0044] It should be noted that the process involves obtaining the camera projection matrix of the 3D predicted contour point at its corresponding time, and extracting the rotation component matrix from the camera projection matrix; obtaining the normal vector of the mesh cell containing the 3D predicted contour point; multiplying this normal vector by the rotation component matrix to obtain the transformed normal vector in the camera coordinate system; projecting the transformed normal vector onto the 2D imaging plane and normalizing it to obtain the 2D projection normal vector representing the local tangential direction of the model surface on the imaging plane. The orientation deviation angle reflects the geometric consistency between the 3D predicted contour point in the 2D projection normal direction and the gradient direction of the 2D observed edge point on the imaging plane.

[0045] In atrial fibrillation, although the atria lose effective contraction, the strong contraction and relaxation of the ventricles (affected by preload, i.e., the RR interval) exert a significant passive traction and torsional effect on the left atrium through the movement of the mitral valve annulus. This invention utilizes the RR interval to quantify this passive traction effect, thereby predicting atrial geometric deformation. In this embodiment of the invention, the method for obtaining the cardiac cycle filling index includes: acquiring the surface electrocardiogram signal during the surgical period and performing R-wave peak detection on the signal to obtain the R-wave moment; taking the time interval between every two adjacent R-wave moments as a cardiac cycle; calculating the ratio of the duration of the cardiac cycle preceding each moment to the average duration of a preset number of adjacent cardiac cycles preceding that moment, as the relative filling duration ratio; taking the time difference between each moment and the start moment of its corresponding cardiac cycle as the cardiac phase delay; if the cardiac phase delay is less than or equal to a preset contraction cutoff threshold, then determining the phase weighting factor as a first preset value; if the cardiac phase delay is greater than the preset contraction cutoff threshold, then determining the phase weighting factor as a second preset value; wherein, the first preset value is greater than the second preset value; and taking the product of the relative filling duration ratio and the phase weighting factor at each moment as the cardiac cycle filling index.

[0046] It should be noted that a standard 12-lead electrocardiograph was used, and simulated electrocardiogram signals were acquired in real time during the surgical period via electrode pads attached to the subject's body surface. The signals were then converted to digital waveforms via A / D conversion, with a sampling rate set to 500 Hz. This embodiment uses the Pan-Tompkins algorithm to detect the R-wave timing. The duration of the preceding cardiac cycle at each moment determines the ventricular end-diastolic filling volume at that moment; the average duration of the pre-set number of adjacent cardiac cycles before each moment reflects the subject's baseline cardiac rhythm in that physiological state. If the relative filling duration ratio is greater than 1.0, it means that the filling time of the cardiac cycle at each moment is prolonged, indicating that the subsequent mechanical contraction amplitude will be enhanced; conversely, the cardiac contraction amplitude is weakened. Based on the physiological characteristic that the mechanical motion amplitude during cardiac systole is significantly greater than that during diastole, the response weight of the cardiac cycle filling index to large deformation characteristics during strong systole is strengthened through differential assignment of phase weighting factors, while suppressing the influence of small diastolic fluctuations on the matching tolerance, thereby achieving accurate quantification of the physiological state of the heart's nonlinear motion characteristics. Specifically: if the cardiac phase delay is less than or equal to the preset systolic cutoff threshold, it indicates that the heart is in a strong systolic phase, and a larger phase weighting factor is assigned; conversely, if the heart enters the diastolic or quiescent phase, a smaller phase weighting factor is assigned. Therefore, the first preset value is greater than the second preset value. If the cardiac cycle filling index is larger, it indicates that the preload of the cardiac cycle at each moment of the subject is greater (the more fully filled), and the heart will undergo more significant mechanical contraction and physiological torsional movements.

[0047] In this embodiment of the invention, the preset contraction cutoff threshold typically covers the main time period of left ventricular contraction and passive traction of the left atrium, and is set to 350 milliseconds. The implementer can set it according to the specific situation.

[0048] In this embodiment of the invention, the preset quantity N is set to 10. The first preset value corresponds to the ventricular systole and early filling period, during which myocardial fibers shorten dramatically and are accompanied by significant torsion. To fully reflect the high deformation characteristics of this period, the first preset value is set to 1, so that the cardiac cycle filling index can directly reflect the maximum expected contraction amplitude determined by preload during systole. The second preset value corresponds to the mid-to-late ventricular diastole and atrial systole, during which the cardiac morphology is relatively stable. To avoid excessive adjustment of the matching tolerance by small filling fluctuations, based on empirical statistics, the diastolic deformation amplitude is approximately 20% to 30% of the systolic peak. Therefore, the second preset value is preferably set to 0.25.

[0049] It should be noted that the cardiac cycle filling index at all times within the first N cardiac cycles during the surgical period is directly set as the standard filling state reference value.

[0050] The cardiac cycle filling index reflects the time-varying physiological state of the heart at each moment. Since the amplitude of cardiac mechanical contraction is positively correlated with the ventricular filling volume, a dynamic mapping relationship is established between the normal deviation threshold and the increasing filling index. This allows for a wider directional matching tolerance to capture the true edge during strong cardiac contraction and physiological torsion, while tightening the tolerance during diastole to eliminate artifacts with disordered orientation. In this embodiment of the invention, the formula for calculating the normal deviation threshold is: In the formula, The normal deviation threshold is the value at the k-th moment of the surgical procedure. The cardiac cycle filling index at the k-th moment of the surgical period; This is a reference value for the standard full-fill state; Preset basic normal deviation tolerance; The preset torsional gain coefficient is defined; the max function is the maximum value function. Several cardiac cycles (e.g., 10 to 20) are collected from the subject during quiet breathing. The cardiac cycle filling index is calculated at each moment within these cardiac cycles using the same method as in this protocol. The average cardiac cycle filling index at all moments within these cardiac cycles is used as the reference value for standard filling state. It should be noted that the amplitude of cardiac mechanical contraction exhibits a significant non-linear increasing trend in high filling states; in low filling or diastole, the cardiac morphology is relatively stable, and contrast agent flow artifacts with disordered directions are often mixed in the fluoroscopic images. When... This means the heart is in diastole and must be maintained. To eliminate interference from contrast agent flow artifacts; when When the heart is in a state of high filling, the greater the distance of the heart contraction, the greater the torsional angle, and the greater the normal deviation threshold should be, so as to capture the real anatomical edge that is deflected at a large angle due to violent exercise.

[0051] In this embodiment of the invention, multi-frame matching data of a three-dimensional cardiac model and fluoroscopic images of a subject or a standard anatomical database during end-diastole and with a stable heart rate are collected. The average deviation angle between the normal of the three-dimensional predicted contour points and the gradient direction of the two-dimensional observed edge points, caused by mesh discretization, projection errors, and minor physiological fluctuations, is statistically analyzed. Typically, the average deviation angle is between 15 and 20 degrees. To ensure the robustness of diastolic matching and accommodate a certain amount of noise, a preset basic normal deviation tolerance is set. Slightly higher than the average deviation angle, preferably set to 20 degrees. Data are collected from subjects or databases during the strong systolic phase, with a cardiac cycle filling index of 2 at various times within the cardiac cycle; the maximum physiological torsion angle of the left atrium induced by ventricular contraction at this time (compared to diastole) is measured; statistics show that under extreme strong systole, the maximum torsion angle can reach 15 to 20 degrees. To capture this large torsion characteristic, a preset torsion gain coefficient is preferably used. Setting it to 15 degrees means that for every increase of 1 in the fullness index, the maximum allowable deviation tolerance increases by 15 degrees.

[0052] In one specific implementation of this invention, the direction deviation penalty coefficient is expressed by the formula: In the formula, The directional deviation penalty coefficient between the i-th 3D predicted contour point and the j-th 2D observed edge point at each time step; The angle between the two-dimensional projection normal vector of the i-th three-dimensional predicted contour point at each time step and the unit gradient vector of the j-th two-dimensional observed edge point; is the normal deviation threshold at the k-th moment of the surgical period; exp is an exponential function with the natural constant as the base. This is a preset penalty coefficient. It should be noted that when... This means that the directional consistency between the i-th 3D predicted contour point and the j-th 2D observed edge point is within the physiologically permissible dynamic range, belonging to a potentially correct anatomical match. Therefore, the following setting is made. A value of 0 means no additional penalty is imposed on the matching cost. When... When the directional consistency of two points exceeds the dynamic range allowed by physiology, it is very likely to be an erroneous artifact match. It can impose a great penalty value on erroneous matches that deviate significantly from physiological expectations, forcing algorithms such as the Hungarian algorithm to abandon erroneous matches, thereby significantly improving the robustness and anatomical consistency of matching results in complex dynamic environments.

[0053] In this embodiment of the invention, a preset penalty coefficient is used to control the growth rate of the directional deviation penalty term as the deviation angle exceeds a threshold. The value of this coefficient is set based on an analysis of the robustness requirements of the matching algorithm in dynamic environments. A significant deviation tolerance is set as follows: The value is 10 degrees, and it is stipulated that the penalty coefficient should reach a significant blocking level under this deviation, for example, the directional deviation penalty coefficient is 10. Substituting the above 10 degrees and 10... The formula for calculating the direction deviation penalty coefficient is as follows: The value is set to the preset penalty coefficient.

[0054] The cost analysis unit 133 is used to adjust the distance between the two-dimensional projection normal vector of the three-dimensional predicted contour point and the unit gradient vector of the two-dimensional observed edge point by using the anatomical deformation resistance coefficient of the corresponding vertex of the three-dimensional predicted contour point on the preoperative three-dimensional cardiac anatomy model, so as to obtain the distance cost between the two corresponding points; and to weight and sum the direction deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point with the distance cost to obtain the composite matching cost between the two corresponding points.

[0055] In one specific implementation of this invention, the composite matching cost is expressed by the formula: In the formula, R is the anatomically weighted matching cost between the i-th 3D predicted contour point and the j-th 2D observed edge point at each time step; R is the anatomical deformation resistance coefficient between the 3D predicted contour point and the vertex on the nearest preoperative 3D cardiac anatomy model in the best matching pair set at each time step. The two-dimensional projection normal vector of the i-th three-dimensional predicted contour point at each time step; This is the unit gradient vector of the j-th two-dimensional observation edge point at each time step; The directional deviation penalty coefficient between the i-th 3D predicted contour point and the j-th 2D observed edge point at each time step; The preset resistance weighting coefficient; The preset minimum positive number is used to prevent the predicted point from geometrically coinciding with the observation point; in this embodiment, it is set to 0.1 pixels. This is the preset direction conversion coefficient.

[0056] It should be noted that, Quantifying the positional deviation between 3D predicted contour points and 2D observed edge points on the image is the fundamental matching criterion. Introducing the anatomical deformation resistance coefficient R as a weight aims to address the issue in strongly constrained anchoring regions (such as pulmonary veins). Approximately 1) applying additional value amplification forces the global optimization algorithm to prioritize finding the nearest matching point for these critical regions, thereby ensuring the stability of the navigation benchmark; while for weakly constrained free regions (such as the auricle, ... If the distance is close to 0, it mainly relies on the geometric distance itself, allowing for a larger degree of deformation freedom. Adding... In order to prevent When the value is 0, the dissection weight term becomes invalid, ensuring that the constraint always exists. The directional deviation penalty term aims to use directional information to eliminate incorrect matches that are close in location but have contradictory edge directions (such as intersecting blood vessel shadows). Used to balance the weights of distance and direction dimensions.

[0057] In this embodiment of the invention, a preset resistance weighting coefficient is used for amplification. A strong constraint anchoring region close to 1 relative to The matching cost weight for the weakly constrained free region is close to zero. To ensure that key anatomical structures such as the pulmonary veins have significant priority in global optimization matching, a weight is typically set... This results in the anchoring region costing 2 to 5 times that of the free region. Therefore, it is preferable to set... With a value range of 2, even if the anchor point undergoes the same geometric displacement, the cost is three times that of the free point, forcing the algorithm to prioritize correcting the anchor point deviation. To ensure that the penalty for inconsistency in direction is sufficient to eliminate artifact matches that are geometrically close but in the wrong direction, a preset direction conversion coefficient is usually set. This ensures that when the directional deviation is significant, the directional cost term dominates the total cost. Based on experimental experience, we set... The value ranges from 5 to 10, with 7 being the preferred value. The unit is pixels, which converts the dimensionless directional penalty item into an equivalent pixel cost.

[0058] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the optimal matching pair set includes: constructing a bipartite graph model based on the three-dimensional predicted contour points and two-dimensional observed edge points at each time step, setting the weight of the edge connecting a three-dimensional predicted contour point and a two-dimensional observed edge point in the model as the contour matching cost between the corresponding two points; using the Hungarian algorithm to solve the minimum weight matching of the bipartite graph model to obtain the optimal matching pair set composed of several matching point pairs; wherein, the matching point pair includes a three-dimensional predicted contour point and a two-dimensional observed edge point.

[0059] It should be noted that by using the anatomical matching cost between the 3D predicted contour points and the 2D observed edge points as the weight of the connecting edges, the matching problem between the 3D model and the 2D image can be transformed into a minimum weight matching problem in a bipartite graph. This allows the use of the Hungarian algorithm to find the optimal correspondence among all possible point pair combinations that minimizes the overall anatomical structure difference and directional deviation, i.e., the optimal set of matching pairs with the minimum total matching cost. Since the number of 3D predicted contour points and 2D observed edge points at any given time in a real-world scenario is often unequal, virtual nodes are typically added to the fewer points to make the number of both types of points equal. The weight of the boundary connecting the virtual nodes is set to a large cutoff threshold (e.g., 50 pixels) to filter out high-error outliers and noise interference. For matching point pairs containing virtual nodes, if the matching point pair contains a 3D predicted contour point, or its composite matching cost is greater than or equal to the cutoff threshold, the 3D predicted contour point is marked as unobserved or occluded, and its position remains unchanged in subsequent model update steps without line-of-sight projection updates. If the matching point pair contains a 2D observed edge point, the point pair is not further analyzed.

[0060] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the geometric error cost includes: weighted summing of the distance and direction deviation penalty coefficient between the two-dimensional projection normal vector of the three-dimensional predicted contour point and the unit gradient vector of the two-dimensional observed edge point in each matching point pair in the best matching pair set at each time step to obtain the local error cost between the corresponding two points; and calculating the arithmetic mean of the local error costs of all matching point pairs in the best matching pair set as the geometric error cost at each time step.

[0061] In one specific implementation of this invention, the geometric error cost is expressed by the formula: In the formula, The geometric error cost between the i-th 3D predicted contour point and the j-th 2D observed edge point at each time step; The two-dimensional projection normal vector of the i-th three-dimensional predicted contour point at each time step; This is the unit gradient vector of the j-th two-dimensional observation edge point at each time step; The directional deviation penalty coefficient between the i-th 3D predicted contour point and the j-th 2D observed edge point at each time step; This is a preset direction conversion coefficient. It should be noted that it only includes the geometric distance and direction deviation penalty between the 3D predicted contour points and the 2D observed edge points. This ensures that the geometric error cost can accurately reflect the true degree of geometric deviation between the 3D model projection and the 2D image edge at each moment, and ensures that the circuit breaker mechanism only applies to real image artifacts or large deformation errors.

[0062] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the reconstructed mesh model includes: for each matching point pair in the best matching pair set at each time moment, using camera imaging projection parameters, mapping the two-dimensional observation edge points in the matching point pair to the three-dimensional physical point coordinates on the imaging plane under the surgical space coordinate system; taking the ray pointing from the camera optical center point to the three-dimensional physical point coordinates as the target line-of-sight ray; taking the orthogonal projection point of the three-dimensional predicted contour point in the matching point pair on the target line-of-sight ray as the spatial foot; taking the direction of the three-dimensional predicted contour point in the matching point pair pointing to the spatial foot as the direction of the line-of-sight projection traction vector; and taking the distance between the three-dimensional predicted contour point in the matching point pair and the spatial foot as... The magnitude of the line-of-sight projection traction vector is given; the geometric error cost of the matching point pair is negatively correlated and normalized to obtain the global confidence weight; the product of the constant 1 and the difference between the anatomical deformation resistance coefficient of the corresponding vertex of the 3D predicted contour point in the matching pair on the preoperative 3D cardiac anatomy model and the global confidence weight is calculated, and the product is multiplied by the line-of-sight projection traction vector to obtain the restricted deformation displacement vector; the position coordinates of the 3D predicted contour point in the matching pair are added to the components of the restricted deformation displacement vector in the same dimension to update the position coordinates of the 3D predicted contour point and obtain the reconstructed position; the 3D reconstruction model is constructed from the reconstructed positions of the 3D predicted contour points of the 3D reference model at each time step.

[0063] It should be noted that by back-projecting the two-dimensional observation edge points into a line-of-sight ray in the surgical space coordinate system, the matching error in three-dimensional space can be constrained to a plane perpendicular to the imaging line of sight, thereby constructing a line-of-sight projection traction vector that is effective in the direction perpendicular to the line of sight. This vector ensures that while the three-dimensional predicted contour points are displaced to match the two-dimensional projected contour, the depth information of the original model is preserved to the maximum extent. This avoids non-anatomical stretching or collapse of the model along the line of sight due to the lack of depth constraints in single-plane projection, thus achieving safe correction of the shape of the three-dimensional model under monocular vision conditions.

[0064] In this embodiment of the invention, the formula for calculating the global credibility weight is: In the formula, The global credibility weight is the set of best matching pairs at time k within the surgical period. The geometric error cost is the cost of each matching pair in the best matching pair set at time k within the surgical period. To preset a security cost threshold; This is the preset cutoff slope. It should be noted that when... When less than 0, A value close to 1 indicates a high degree of reliability in the matching of point pairs, allowing for significant model updates; when... When greater than 0, It rapidly decays to 0, thus automatically identifying and blocking erroneous deformations caused by high-error artifacts, playing an adaptive image stabilization filtering role and preventing model topology collapse.

[0065] In this invention, a preset security cost threshold is used. The preset truncation slope is typically set based on the average resolution of the intraoperative fluoroscopic images and the expected anatomical contour matching error. A value of 5 to 10 pixels is generally preferred, with 8 pixels being the optimal setting. This represents the maximum allowable average pixel distance between the actual anatomical edge and the 3D projected contour, excluding severe artifact interference. The sensitivity used to control weight decay is typically set between 0.5 and 1, with larger values ​​being preferable. This means that the error exceeds The lower tolerance for errors allows the global confidence weight to drop more steeply to 0, thereby enhancing the filtering ability for high-error artifacts.

[0066] In one specific implementation of this invention, the reconstructed position vector of the three-dimensional predicted contour points is expressed by the formula: In the formula, For each matching point in the best matching pair set at time k within the surgical period, the reconstructed position vector of the 3D predicted contour point is given. For each of the three-dimensional predicted contour points in the best matching pair set at time k within the surgical period; The global credibility weight is the set of best matching pairs at time k within the surgical period. , is the anatomical deformation resistance coefficient between the three-dimensional predicted contour point and the vertex on the nearest preoperative three-dimensional cardiac anatomy model in the best matching pair set at time k within the surgical period; For each matching pair in the best matching pair set at time k within the surgical period, there is the gaze projection traction vector. This is a constrained deformation displacement vector. In this embodiment, the position coordinates of the predicted 3D contour point are described in vector form to obtain the position coordinate vector. The reconstructed position vector is then expressed in coordinate form to obtain the reconstructed position coordinates of the contour point.

[0067] It should be noted that, It can dynamically adjust the deformation amplitude at the global level based on the current matching quality, effectively suppressing non-anatomical drift caused by artifacts; It can exert strong constraints on high-resistance areas such as the root of the pulmonary vein at the local level, ensuring that the spatial position of key anatomical structures remains stable relative to the preoperative model; It reflects the limited deformation driving force that can be effectively applied to the three-dimensional predicted contour points, reflects the reliability of visual information and the rigidity and stability of the biomechanical model, and ensures that each update of the model conforms to the currently observed real image characteristics and does not violate the physiological anchoring and deformation laws.

[0068] It is important to note that this approach only updates the positions of the predicted 3D contour points of the 3D reference model at each time step; non-contour points are followed through interpolation or rigid transformation. The goal of this approach is "contour alignment" to assist navigation (2D projection alignment is sufficient to meet the doctor's visual needs), rather than "full 3D reconstruction"; as long as the projection matches the X-ray, it has clinical value.

[0069] This invention is now complete.

[0070] Example 2: Figure 3 This is a schematic diagram of a computer device for cardiovascular three-dimensional reconstruction based on multimodal image data fusion, provided as an embodiment of the present invention. Exemplary examples include... Figure 3 As shown, the computer device includes: a memory 201, a processor 202, and a computer program 203 stored in the memory 201 and running on the processor 202, wherein when the processor 202 executes the computer program 203, the computer device can execute any of the aforementioned cardiovascular three-dimensional reconstruction systems based on multimodal image data fusion.

[0071] Furthermore, embodiments of this application also protect an apparatus that may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to execute a cardiovascular three-dimensional reconstruction system based on multimodal image data fusion provided in embodiments of this application.

[0072] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0073] It should be understood that the device provided in this embodiment is used to execute the above-described cardiovascular three-dimensional reconstruction system based on multimodal image data fusion, and therefore can achieve the same effect as the above-described implementation method.

[0074] When using integrated units, the device may include a processing module and a storage module. When applied to a workpiece, the processing module can be used to control and manage the workpiece's operations. The storage module can be used to support the execution of program code by the workpiece.

[0075] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits as disclosed in this application. The processor may also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and microprocessors, etc., and the storage module may be a memory.

[0076] Example 3: This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the cardiovascular three-dimensional reconstruction system based on multimodal image data fusion provided in the above embodiment.

[0077] In this embodiment, the device and computer-readable storage medium are used to execute the corresponding system provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding system provided above, and will not be repeated here.

[0078] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0079] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cardiovascular three-dimensional reconstruction system based on multimodal image data fusion, characterized in that, The system includes: The data acquisition module is used to acquire a preoperative three-dimensional cardiac anatomical model of the subject and fluoroscopic images at various times during the surgical period; the model is composed of different grid units; The initialization module is used to perform initial rigid registration of the preoperative three-dimensional cardiac anatomy model to obtain the three-dimensional reference model at the initial moment of the surgical period, and to use the three-dimensional reconstruction model of the previous moment of each non-initial moment of the surgical period as the three-dimensional reference model. The matching analysis module is used to determine the three-dimensional predicted contour points at each time step based on the projection relationship of the three-dimensional reference model relative to the imaging viewpoint, and to extract the two-dimensional observation edge points of the perspective image. Based on the positional difference between each three-dimensional predicted contour point and each two-dimensional observation edge point at each time step, the spatial constraints of the anatomical structure, and the consistency of directional features, the module determines the composite matching cost between corresponding two points and determines the best matching pair set between the three-dimensional predicted contour points and the two-dimensional observation edge points at each time step. The model update module is used to adjust the projection deviation between the matching point pairs based on the composite matching cost of the best matching pair set after removing the influence of the anatomical structure spatial constraints, and to update the position of the three-dimensional predicted contour points of the three-dimensional reference model, so as to obtain the three-dimensional reconstruction model at each time step.

2. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 1, characterized in that, The determination of the three-dimensional predicted contour points at each time step includes: Obtain the camera's optical center point; for each mesh cell in the 3D reference model at each moment, obtain the unit vector of the direction vector from the camera's optical center point to the center point of the mesh cell, denoted as the unit line-of-sight vector; Obtain the unit normal vector of the mesh cell pointing to the outside of the 3D reference model; take the dot product of the unit view vector and the unit normal vector as the visibility index of the mesh cell; extract the common edge connecting two adjacent mesh cells with opposite visibility indexes from the 3D reference model as the 3D contour line. Sample all three-dimensional contour lines on the three-dimensional reference model at each time step to obtain the three-dimensional predicted contour points.

3. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 2, characterized in that, Determining the composite matching cost between two corresponding points includes: The surface of the preoperative three-dimensional heart model contains several anatomical anchor points and vertices connecting different mesh units; the geodesic distance from each vertex of the preoperative three-dimensional heart model to the nearest anatomical anchor point is obtained, and the geodesic distance is negatively correlated and normalized to obtain the anatomical deformation resistance coefficient of the corresponding vertex. The unit normal vector of the grid cell containing each 3D predicted contour point at each time step is projected onto the 2D imaging plane and normalized to serve as the 2D projection normal vector. For each 3D predicted contour point and each 2D observed edge point at each time step, the directional deviation penalty coefficient between the corresponding two points is determined based on the directional consistency between the 2D projection normal vector of the 3D predicted contour point and the unit gradient vector of the 2D observed edge point. Using the anatomical deformation resistance coefficient of the corresponding vertex of the three-dimensional predicted contour point on the preoperative three-dimensional cardiac anatomy model, the distance between the two-dimensional projection normal vector of the three-dimensional predicted contour point and the unit gradient vector of the two-dimensional observation edge point is adjusted to obtain the distance cost between the two corresponding points. The directional deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point is weighted and summed with the distance cost to obtain the composite matching cost between the two corresponding points.

4. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 3, characterized in that, Determining the directional deviation penalty coefficient between two corresponding points includes: For each 3D predicted contour point and each 2D observed edge point at each time step, the angle between the 2D projection normal vector of the 3D predicted contour point and the unit gradient vector of the 2D observed edge point is used as the directional deviation angle between the two points. The cardiac cycle filling index is determined based on the time-varying physiological state of the heart at each moment, and the normal deviation threshold at each moment is calculated. If the directional deviation angle is less than or equal to the normal deviation threshold, the deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point is set to 1; if the directional deviation angle is greater than the normal deviation threshold, the directional deviation penalty coefficient between the three-dimensional predicted contour point and the two-dimensional observed edge point is determined based on the difference between the directional deviation angle and the normal deviation threshold.

5. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 4, characterized in that, The determination of the cardiac cycle filling index based on the time-varying physiological state of the heart at each moment includes: Acquire surface electrocardiogram signals during the surgical period, and perform R-wave peak detection on the signals to obtain R-wave moments; take the time interval between each two adjacent R-wave moments as a cardiac cycle; Calculate the ratio of the duration of the cardiac cycle preceding each moment to the average duration of the number of adjacent cardiac cycles preceding that moment, and use it as the relative fullness duration ratio. The time difference between each moment and the start moment of its corresponding cardiac cycle is taken as the cardiac phase delay; if the cardiac phase delay is less than or equal to a preset contraction cutoff threshold, the phase weighting factor is determined to be a first preset value; if the cardiac phase delay is greater than the preset contraction cutoff threshold, the phase weighting factor is determined to be a second preset value. The product of the relative filling duration ratio at each moment and the phase weighting factor is used as the cardiac cycle filling index.

6. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 1, characterized in that, The determination of the optimal set of matching pairs between the three-dimensional predicted contour points and the two-dimensional observed edge points at each time step includes: A bipartite graph model is constructed based on the three-dimensional predicted contour points and two-dimensional observed edge points at each time step. The weight of the edge connecting a three-dimensional predicted contour point and a two-dimensional observed edge point in the model is set as the composite matching cost between the two points. The minimum weight matching of the bipartite graph model is solved using the Hungarian algorithm to obtain the optimal matching pair set consisting of several matching point pairs; wherein, the matching point pair includes a three-dimensional predicted contour point and a two-dimensional observed edge point.

7. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 3, characterized in that, The method for obtaining the geometric error cost includes: For each time step, the distance between the two-dimensional projection normal vector of the three-dimensional predicted contour point and the unit gradient vector of the two-dimensional observed edge point in each matching point pair in the best matching pair set is weighted and summed with the direction deviation penalty coefficient to obtain the local error cost between the corresponding two points. Calculate the arithmetic mean of the local error costs of all matching point pairs in the best matching pair set as the geometric error cost at each time step.

8. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 3, characterized in that, The process of obtaining the 3D reconstruction model at each time step includes: For each matching point pair in the best matching pair set at each time moment, the two-dimensional observation edge points in the matching point pair are mapped to the three-dimensional physical point coordinates on the imaging plane under the surgical space coordinate system using the camera imaging projection parameters; the ray pointing from the camera optical center to the three-dimensional physical point coordinates is taken as the target line-of-sight ray; the orthogonal projection point of the three-dimensional predicted contour points in the matching point pair on the target line-of-sight ray is taken as the spatial foot. The direction from the 3D predicted contour point of the matching point centering to the spatial perpendicular foot is taken as the direction of the line of sight projection traction vector, and the distance between the 3D predicted contour point of the matching point centering and the spatial perpendicular foot is taken as the magnitude of the line of sight projection traction vector. The geometric error costs of the matching point pairs are negatively correlated and normalized to obtain the global confidence weight; The product of the difference between constant 1 and the anatomical deformation resistance coefficient of the corresponding vertex of the three-dimensional predicted contour point in the matching pair on the preoperative three-dimensional cardiac anatomy model and the global confidence weight is calculated. The product is then multiplied by the line-of-sight projection traction vector to obtain the restricted deformation displacement vector. The position coordinates of the matched 3D predicted contour point are added to the components of the same dimension of the constrained deformation displacement vector to update the position coordinates of the 3D predicted contour point and obtain the reconstructed position. A 3D reconstruction model is constructed from the reconstructed positions of the 3D predicted contour points of the 3D reference model at each time step.

9. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 1, characterized in that, The extraction of two-dimensional observation edge points from the perspective image includes: Edge detection is performed on the perspective image at each time step, and pixels with gradient magnitudes greater than a preset gradient threshold are selected as edge candidate points; the edge candidate points are then sparsely sampled to obtain the two-dimensional observation edge points at the corresponding time step.

10. The cardiovascular three-dimensional reconstruction system based on multimodal image data fusion according to claim 5, characterized in that, The first preset value is greater than the second preset value.

Images