Three-dimensional modeling device and navigation system

Abstract

The application relates to a three-dimensional modeling device and a navigation system, the device comprising an intraoperative three-dimensional model generation module for generating a current intraoperative three-dimensional model corresponding to each intraoperative phase based on preoperative three-dimensional model data and intraoperative electrical signals; a key frame retrieval module for acquiring a current endoscope image and a corresponding intraoperative phase, determining each target key frame data from each key frame data generated in advance based on the intraoperative phase, and determining matching key frame data corresponding to the current endoscope image; the key frame data comprising a first mapping relationship between each key frame and a current intraoperative three-dimensional model of the same intraoperative phase and a feature point pair; a fusion module for determining a second mapping relationship based on the first mapping relationship and the feature point pair in the matching key frame data, and fusing the current intraoperative three-dimensional model and the current endoscope image based on the second mapping relationship. The device and the system can improve efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of intelligent medical technology, and in particular to a three-dimensional modeling device and navigation system. Background Technology

[0002] Cardiac surgery, due to the unique characteristics of the heart—its continuous rhythmic beating and significant elastic deformation—places far more stringent demands on the precision, real-time performance, and continuity of intraoperative navigation and image-guided techniques than conventional surgery. In complex cardiac surgeries, such as off-pump coronary artery bypass grafting (OPCABG) or interventional treatment of structural heart disease, surgeons not only need to rely on preoperative images (such as CT / MRI) to understand the three-dimensional spatial relationships and anatomical structures of the lesion, but also urgently need to utilize augmented reality (AR) or mixed reality (MR) navigation technology during surgery to dynamically fuse preoperative three-dimensional anatomical models with intraoperative endoscopic images with high precision and high fidelity. This allows for real-time localization of deep target tissues (such as coronary artery branches, papillary muscles, chordae tendineae, and adjacent vessels) and avoidance of important physiological functional areas.

[0003] In traditional techniques, preoperative CT / MRI image data is used to construct a three-dimensional model and then fused with the intraoperative endoscopic view. Specifically, through image registration and feature matching techniques, the preoperative three-dimensional model is superimposed on the endoscopic image to provide the surgeon with a spatial reference for the anatomical structure.

[0004] However, the heart is in a state of continuous deformation throughout the cardiac cycle, and there are significant differences in the surface morphology, vascular course, and relative spatial position of the heart at end-systole and end-diastole. The three-dimensional model constructed preoperatively in traditional techniques is a static model, which cannot accurately reflect the true deformation state of the heart at different cardiac phases, resulting in inaccurate models. Summary of the Invention

[0005] Therefore, it is necessary to provide a 3D modeling device and navigation system that can improve efficiency and accuracy in addressing the aforementioned technical problems.

[0006] In a first aspect, this application provides a three-dimensional modeling apparatus, the apparatus comprising:

[0007] The intraoperative 3D model generation module is used to generate the current intraoperative 3D model corresponding to each intraoperative cardiac motion phase based on the preoperative 3D model data and the intraoperative electrocardiographic signal.

[0008] The keyframe retrieval module is used to acquire the current endoscopic image and the intraoperative motion phase corresponding to the current endoscopic image, determine target keyframe data from pre-generated keyframe data based on the intraoperative motion phase corresponding to the current endoscopic image, and determine matching keyframe data corresponding to the current endoscopic image from each target keyframe data; the keyframe data includes a first mapping relationship between each keyframe and the current intraoperative 3D model of the same intraoperative motion phase, as well as feature point pairs;

[0009] The fusion module is used to determine a second mapping relationship between the current endoscopic image and the corresponding current intraoperative 3D model based on a first mapping relationship in the matching keyframe data and the feature point pairs, and to fuse the current intraoperative 3D model with the current endoscopic image based on the second mapping relationship.

[0010] In some optional embodiments, the intraoperative 3D model generation module is specifically used to acquire preoperative 3D model data and intraoperative electrocardiogram (ECG) signals. The preoperative 3D model data includes a baseline 3D model and displacement vectors corresponding to each cardiac phase. Based on the intraoperative cardiac phase corresponding to the intraoperative ECG signals, a target displacement vector is determined from the preoperative 3D model data, and the baseline 3D model is deformed based on the target displacement vector to obtain the current intraoperative 3D model corresponding to each intraoperative cardiac phase.

[0011] In some alternative embodiments, the device further includes:

[0012] The preoperative 3D model generation module is used to acquire preoperative medical images and preoperative electrocardiogram (ECG) signals, generate a preoperative 3D model of the cardiac phase corresponding to the preoperative ECG signals based on the preoperative medical images, and generate a baseline 3D model and displacement vectors corresponding to each cardiac phase based on the preoperative 3D model of the cardiac phase corresponding to the preoperative ECG signals, as preoperative 3D model data.

[0013] In some optional embodiments, the preoperative 3D model generation module includes:

[0014] The preoperative data acquisition unit is used to acquire preoperative medical images and various preoperative electrocardiogram signals;

[0015] A segmentation unit is used to segment the preoperative medical images to obtain the cardiac region;

[0016] The multi-phase model construction unit is used to perform three-dimensional reconstruction of the segmented heart region based on the preoperative cardiac phase corresponding to the preoperative electrocardiogram signal, so as to obtain the preoperative three-dimensional model corresponding to each preoperative cardiac phase.

[0017] The baseline model determination unit is used to determine the preoperative three-dimensional model corresponding to the target cardiac phase from the preoperative three-dimensional models corresponding to each of the preoperative cardiac phases as the baseline three-dimensional model.

[0018] The displacement vector determination unit is used to calculate the displacement vector of each of the preoperative three-dimensional models relative to the reference three-dimensional model based on the differential homeomorphism mapping algorithm.

[0019] In some alternative embodiments, the device includes a parallel GPU; the preoperative 3D model data is pre-stored in the GPU, and the intraoperative 3D model generation module is located in the GPU.

[0020] In some alternative embodiments, the device further includes:

[0021] The keyframe construction module is used to acquire the initial endoscopic image corresponding to each intraoperative cardiac phase within a complete cardiac cycle as a keyframe; and to establish a first mapping relationship between each keyframe and the current intraoperative three-dimensional model corresponding to the intraoperative cardiac phase, and generate keyframe data based on the first mapping relationship.

[0022] In some alternative embodiments, the keyframe building module includes:

[0023] The initial matching unit is used to receive a keyframe addition instruction, obtain an initial endoscopic image corresponding to each intraoperative cardiac phase within a complete cardiac cycle as a keyframe based on the keyframe addition instruction, and establish a first mapping relationship between the keyframe and the corresponding intraoperative cardiac phase of the current intraoperative three-dimensional model.

[0024] The first feature extraction unit is used to perform feature point detection on the initial endoscopic image in parallel to obtain initial feature points and to perform heart region segmentation on the initial endoscopic image to obtain a first target heart region, and to filter the initial feature points to obtain a first target feature point located in the first target heart region.

[0025] The keyframe data unit is used to determine the second target feature point corresponding to the first target feature point from the current intraoperative 3D model corresponding to the intraoperative motion phase based on the first mapping relationship, and to generate keyframe data based on the first mapping relationship, feature point pairs including the first target feature point and the second target feature point, and the intraoperative motion phase.

[0026] In some alternative embodiments, the keyframe building module further includes:

[0027] The integrity judgment unit is used to detect whether the generated keyframe data covers a complete cardiac cycle during the execution of the keyframe addition instruction; if the generated keyframe data does not cover a complete cardiac cycle, the unit controls the initial matching unit to continue acquiring the initial endoscopic image of the next cardiac phase during the operation until it is determined that a complete cardiac cycle has been covered, and then stops acquiring the initial endoscopic image of the next cardiac phase during the operation.

[0028] In some alternative embodiments, the first feature extraction unit is used to perform feature point detection on the initial endoscopic image to obtain anatomical feature points and / or general image texture feature points.

[0029] In some optional embodiments, the keyframe retrieval module includes:

[0030] The second feature extraction unit is used to acquire the current endoscope image, perform feature point detection on the current endoscope image in parallel to obtain the feature points to be matched, and perform heart region segmentation on the current endoscope image to obtain the second target heart region, and filter the feature points to be matched to obtain the third target feature points located in the second target heart region.

[0031] The keyframe retrieval unit is used to determine each target keyframe data from each pre-generated keyframe data based on the intraoperative motion phase corresponding to the current endoscopic image.

[0032] The matching unit is used to match the third target feature point with the first target feature point of the feature point pair in the target keyframe data to obtain the matching keyframe data corresponding to the current endoscopic image.

[0033] In some optional embodiments, the keyframe retrieval unit is further configured to adjust the phase deviation threshold when the rate of change of the heart rate of the intraoperative electrocardiogram exceeds the rate of change threshold; and to determine the difference between the intraoperative cardiac motion phase corresponding to the current endoscopic image and the intraoperative cardiac motion phase in each pre-generated keyframe data, and to acquire keyframe data with a difference less than the phase deviation threshold as target keyframe data when the difference is less than the phase deviation threshold.

[0034] In some alternative embodiments, the fusion module includes:

[0035] The feature point pair generation unit is used to generate target feature point pairs based on the first mapping relationship, the feature point pairs in the matching keyframe data, and the third target feature point in the current endoscopic image. Each target feature point pair includes the second target feature point of the feature point pair in the matching keyframe data and the third target feature point in the current endoscopic image.

[0036] The second mapping relationship generation unit is used to generate a second mapping relationship based on the target feature point pair;

[0037] The fusion unit is used to fuse the current intraoperative 3D model with the current endoscopic image based on the second mapping relationship.

[0038] In some alternative embodiments, the device further includes:

[0039] The fusion quality monitoring module is used to acquire the mapping feature points of the current intraoperative 3D model to the current endoscopic image, and generate a fusion error based on the distance between the mapping feature points and the fourth target feature point in the current endoscopic image; when the fusion error corresponding to a consecutive preset number of the current endoscopic images is greater than the error threshold, an alarm message is output.

[0040] In some alternative embodiments, the device further includes:

[0041] The display module is used to display the fused medical image and adjust the display parameters of the current intraoperative 3D model according to the rendering instructions corresponding to the current intraoperative 3D model. The display parameters include at least one of rendering transparency, magnification, section cutting, and rotation viewpoint.

[0042] Secondly, this application also provides a navigation system, which includes the three-dimensional modeling device described in any of the above embodiments.

[0043] The aforementioned 3D modeling device and navigation system include: an intraoperative 3D model generation module, used to generate a current intraoperative 3D model corresponding to each intraoperative cardiac motion phase based on preoperative 3D model data and intraoperative cardiac electrocardiogram signals; and a keyframe retrieval module, used to acquire a current endoscopic image and the intraoperative cardiac motion phase corresponding to the current endoscopic image, determine target keyframe data from pre-generated keyframe data based on the intraoperative cardiac motion phase corresponding to the current endoscopic image, and determine matching keyframe data corresponding to the current endoscopic image from each target keyframe data; the keyframe data includes each keyframe and the current intraoperative cardiac motion phase of the same intraoperative cardiac motion phase. The first mapping relationship and feature point pairs of the three-dimensional model; the fusion module is used to determine the second mapping relationship between the current endoscopic image and the corresponding current intraoperative three-dimensional model based on the first mapping relationship and the feature point pairs in the matching keyframe data, and to fuse the current intraoperative three-dimensional model with the current endoscopic image based on the second mapping relationship. In this way, the keyframe retrieval module determines each target keyframe data from each pre-generated keyframe data based on the cardiac phase during the operation, and then performs matching, which improves processing efficiency. Moreover, different cardiac phases correspond to different keyframe data, so there will be no errors in matching, which also improves accuracy. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of a coronary artery bypass surgery scenario in one embodiment;

[0046] Figure 2 This is a schematic diagram of a three-dimensional modeling device in one embodiment;

[0047] Figure 3 This is a schematic diagram of the fusion algorithm flow in one embodiment;

[0048] Figure 4 This is a rendering diagram of a three-dimensional model during the operation in one embodiment;

[0049] Figure 5 This is a flowchart illustrating the intraoperative 3D model generation module in one embodiment;

[0050] Figure 6 This is a flowchart of the preoperative 3D model generation module in one embodiment;

[0051] Figure 7 This is a schematic diagram of the keyframe construction module process in one embodiment;

[0052] Figure 8 This is a schematic diagram illustrating the mapping relationship between image point coordinates and three-dimensional point coordinates in one embodiment;

[0053] Figure 9 This is a schematic diagram of feature point detection in one embodiment;

[0054] Figure 10 This is a flowchart illustrating the intraoperative keyframe retrieval module in one embodiment;

[0055] Figure 11 This is a schematic diagram of feature point pair matching in one embodiment;

[0056] Figure 12 This is a flowchart illustrating the fusion module in one embodiment;

[0057] Figure 13 This is a functional diagram illustrating a coronary artery bypass surgery scenario in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0060] The three-dimensional modeling device provided in this application embodiment can be applied to... Figure 1 The navigation system shown is illustrated in this application, which uses the example of a navigation system used in coronary artery bypass surgery. However, those skilled in the art will recognize that this navigation system can also be used for other surgical procedures.

[0061] Combination Figure 1 As shown, Figure 1This is a schematic diagram of a coronary artery bypass surgery scenario in one embodiment, wherein the scenario includes a surgical robot, a patient, an image carriage, and the three-dimensional modeling device of this application; the image carriage can display endoscopic images (two-dimensional images) in real time, as well as fused images that have been synchronously corrected and registered by ECG signals (Electrocardiogram, electrocardiogram signal), the fused images including the heart and the current intraoperative three-dimensional model, the current intraoperative three-dimensional model can be a three-dimensional rendering model of blood vessels.

[0062] In one exemplary embodiment, such as Figure 2 As shown, this application provides a three-dimensional modeling device, which includes an intraoperative three-dimensional model generation module, a keyframe retrieval module, and a fusion module. The intraoperative three-dimensional model generation module generates a current intraoperative three-dimensional model corresponding to each intraoperative cardiac motion phase based on preoperative three-dimensional model data and intraoperative electrocardiographic signals. The keyframe retrieval module acquires the current endoscopic image and the corresponding intraoperative cardiac motion phase, determines target keyframe data from pre-generated keyframe data based on the intraoperative cardiac motion phase, and determines matching keyframe data corresponding to the current endoscopic image from the target keyframe data. The fusion module determines a second mapping relationship between the current endoscopic image and the corresponding current intraoperative three-dimensional model based on a first mapping relationship and feature point pairs in the matching keyframe data, and fuses the current intraoperative three-dimensional model with the current endoscopic image based on the second mapping relationship.

[0063] Cardiac phase refers to a specific moment or state of the heart during its beating cycle. A complete cardiac cycle includes multiple phases such as systole and diastole. Keyframe data consists of pre-constructed data nodes, which contain at least one endoscopic image (i.e., a keyframe), a first mapping relationship between the keyframe and the current intraoperative 3D model at the same cardiac phase, and feature point pairs located on the keyframe and the current intraoperative 3D model. The first mapping relationship refers to the initial spatial geometric correspondence between two-dimensional pixels in the keyframe image and three-dimensional points on the 3D model. A feature point pair is a set of points located on the endoscopic image and the current intraoperative 3D model, respectively, corresponding to the same anatomical location, where one point is a two-dimensional image point and the other is a 3D model point.

[0064] Among them, combined Figure 3 As shown, Figure 3 The diagram below illustrates the fusion algorithm flow in one embodiment. The fusion algorithm in this application includes a preoperative stage and an intraoperative stage. The preoperative stage is used to generate preoperative three-dimensional model data. Specifically, it is used to decouple the patient's preoperative image sequence to generate a baseline three-dimensional model and displacement vectors corresponding to each cardiac phase.

[0065] The intraoperative phase comprises two parallel processing chains. The first chain, a deformation rendering chain, continuously drives the baseline model for deformation rendering based on real-time intraoperative ECG signals. Specifically, this chain uses an intraoperative 3D model generation module to generate the current intraoperative 3D model corresponding to each intraoperative cardiac phase based on preoperative 3D model data and intraoperative ECG signals. The preoperative 3D model data includes the baseline 3D model and displacement vectors corresponding to each cardiac phase. This allows for the acquisition of real-time intraoperative cardiac phases during the operation, and the corresponding displacement vectors are obtained based on these phases. The baseline 3D model is then processed using these displacement vectors to obtain the intraoperative 3D model.

[0066] The second link is the tracking link, in which the intraoperative phase includes the intraoperative preparation phase and the intraoperative execution phase. The intraoperative preparation phase is used to generate keyframe data, which can be manually completed by the doctor. The keyframe data includes the first mapping relationship between each keyframe and the current intraoperative 3D model of the same intraoperative motion phase, as well as feature point pairs. That is, the keyframe is the endoscopic image, the first mapping relationship is the initial matching relationship between the endoscopic image and the current intraoperative 3D model of the same intraoperative motion phase, and the feature point pair is a pair of points located on the endoscopic image and the current intraoperative 3D model of the same intraoperative motion phase, with one point on the endoscopic image and the other point on the current intraoperative 3D model.

[0067] The intraoperative execution phase is used to sequentially retrieve and match keyframe data through phase constraint during the operation, and finally solve the camera pose through the fusion module to achieve real-time dynamic fusion display of the 3D model and the real organ. Specifically, the intraoperative execution phase obtains the current endoscopic image and the intraoperative motion phase corresponding to the current endoscopic image through the keyframe retrieval module. Based on the intraoperative motion phase corresponding to the current endoscopic image, it determines each target keyframe data from the pre-generated keyframe data, and determines the matching keyframe data corresponding to the current endoscopic image from each target keyframe data. Then, the fusion module determines the second mapping relationship between the current endoscopic image and the corresponding current intraoperative 3D model based on the first mapping relationship and feature point pairs in the matching keyframe data, and fuses the current intraoperative 3D model with the current endoscopic image based on the second mapping relationship.

[0068] The keyframe retrieval module can acquire real-time intraoperative cardiac phase based on ECG signals. Since the keyframe data includes the first mapping relationship between each keyframe and the current intraoperative 3D model of the same intraoperative cardiac phase, as well as feature point pairs, it can acquire corresponding target keyframe data based on the real-time intraoperative cardiac phase. Only these target keyframe data need to be matched, rather than all keyframe data, reducing the amount of matching data and improving matching efficiency. After the target keyframe data is determined, the current endoscopic image is matched with each target keyframe data, for example, with the keyframes within each target keyframe data, to determine the matching keyframe data. This establishes a matching relationship between the current endoscopic image and the matching keyframe data. Given the first mapping relationship and feature point pairs, a second mapping relationship between the current endoscopic image and the corresponding current intraoperative 3D model can be obtained based on the feature points in the matching keyframes, the feature points in the current endoscopic image, and the feature points of the current intraoperative 3D model. This involves camera pose calculation, and finally, based on the calculation result (the second mapping relationship), the current intraoperative 3D model and the current endoscopic image are fused. Optionally, this application reprojects the current intraoperative 3D model onto the current endoscopic image to achieve real-time dynamic enhancement display of the 3D model and the actual organ movement.

[0069] In the above embodiments, the keyframe retrieval module determines each target keyframe data from the pre-generated keyframe data based on the cardiac phase during surgery, and then performs matching, which improves processing efficiency. Furthermore, different cardiac phases correspond to different keyframe data, so there will be no errors during matching, which also improves accuracy.

[0070] In some optional embodiments, the intraoperative 3D model generation module is specifically used to acquire preoperative 3D model data and intraoperative electrocardiogram (ECG) signals. The preoperative 3D model data includes a baseline 3D model and displacement vectors corresponding to each cardiac phase. Based on the intraoperative cardiac phase corresponding to the intraoperative ECG signals, the target displacement vector is determined from the preoperative 3D model data, and the baseline 3D model is deformed based on the target displacement vector to obtain the current intraoperative 3D model corresponding to each intraoperative cardiac phase.

[0071] Preoperative 3D model data is a dataset pre-calculated and stored before surgery, including at least a static baseline 3D model and displacement vectors corresponding to each cardiac phase. The baseline 3D model refers to the 3D model of the heart at a specific phase (such as end-diastole) selected from the cardiac cycle, serving as the basis for subsequent deformation. The displacement vector refers to the geometric deformation of the heart's shape at a certain cardiac phase relative to the baseline 3D model. It can be represented as a vector field (i.e., a deformation field) containing 3D spatial coordinate offsets, used to characterize the non-rigid motion of various points on the heart wall.

[0072] Combination Figure 4 As shown, Figure 4 This is a rendering diagram of an intraoperative 3D model in one embodiment, wherein the input terminal of the 3D modeling device receives static data and dynamic signals. The static data is the preoperative 3D cardiac model (i.e., the baseline model) obtained from preoperative reconstruction; the dynamic signals are the intraoperative ECG signals (i.e., electrocardiogram signals) collected in real time by an intraoperative electrocardiogram monitoring device.

[0073] The 3D modeling device analyzes the current intraoperative cardiac motion phase based on the intraoperative ECG signal. This intraoperative cardiac motion phase is then used as an index to search within a pre-set preoperative 3D model dataset. This preoperative 3D model dataset includes a continuous set of spatiotemporal displacement vectors consisting of deformation fields from ventricular systole to ventricular diastole, i.e., the displacement vectors corresponding to each preoperative cardiac motion phase.

[0074] After extracting the target displacement vector corresponding to the intraoperative cardiac motion phase, the 3D modeling device superimposes this target displacement vector onto the baseline model. By driving the spatial non-rigid displacement of the model vertices, the morphology of the heart at different physiological cycles (such as systolic morphology, diastolic morphology and their intermediate transitional states) that match the time axis is dynamically reconstructed, which is the current intraoperative 3D model.

[0075] Through the aforementioned decoupling mapping mechanism, the morphological evolution of the intraoperative three-dimensional cardiac model in the time dimension can be strictly and in real time adapted to the patient's own physiological rhythm with high-fidelity non-rigid follow-up changes.

[0076] Combination Figure 5 As shown, Figure 5 This is a flowchart of the intraoperative 3D model generation module in one embodiment. In this embodiment, during the operation, the 3D modeling device receives intraoperative ECG signals from the patient's ECG monitoring device in real time and continuously, and analyzes the intraoperative ECG signals in real time to obtain the current intraoperative cardiac phase (a phase in the cardiac cycle, which may include diastole and systole).

[0077] Then, the phase-corresponding deformation field (i.e., the target displacement vector) is mapped and calculated. Specifically, this includes: based on the current intraoperative motion phase that has been resolved, retrieving and extracting the target elastic deformation field (i.e., the corresponding target displacement vector) corresponding to the current intraoperative motion phase from the preoperative 3D model data.

[0078] Third, the target displacement vector is applied to the baseline 3D model: the extracted target displacement vector is input into the GPU vertex shader, and the vertices of the baseline 3D model are driven to move in real time through hardware acceleration, thereby generating the current intraoperative 3D model corresponding to the current intraoperative cardiac motion phase, that is, the 3D point cloud model of the heart after deformation.

[0079] Fourth, real-time rendering and display: The current intraoperative 3D model is rendered in real time and reprojected into the surgical endoscope field of view for AR fusion display to ensure that the model shape is synchronized with the actual physiological beating state of the patient's heart.

[0080] Finally, the loop iterates, that is, after the rendering and display of the current frame is completed, it continues to receive the real-time ECG signal of the next time sequence, thereby realizing dynamic closed-loop follow-up rendering of the entire surgical cycle.

[0081] In the above embodiments, during the operation, the three-dimensional modeling device accurately analyzes the cardiac phase in the operating room based on the real-time electrocardiogram signal acquired in real time, and directly drives the three-dimensional cardiac model to deform through phase indexing. This achieves physiological synchronization between the three-dimensional model morphology and the patient's real heart rhythm. This physiological-level dynamic matching eliminates sensory deviations caused by model morphology lag or mismatch, greatly improves the intuitiveness and reliability of clinical navigation, and provides surgeons with continuous dynamic visual feedback that is highly consistent with the heart rhythm in the surgical field.

[0082] In some optional embodiments, the three-dimensional modeling device further includes: a preoperative three-dimensional model generation module, used to acquire preoperative medical images and preoperative electrocardiogram signals, generate a preoperative three-dimensional model of the cardiac phase corresponding to the preoperative electrocardiogram signals based on the preoperative medical images, and generate a reference three-dimensional model and displacement vectors corresponding to each cardiac phase based on the preoperative three-dimensional model of the cardiac phase corresponding to the preoperative electrocardiogram signals, as preoperative three-dimensional model data.

[0083] The preoperative 3D model data includes a baseline 3D model and displacement vectors corresponding to each cardiac phase. In this embodiment, the patient's MRI images (preoperative medical images) are acquired preoperatively and combined with ECG signals (preoperative electrocardiogram signals). A 3D model sequence of each cardiac phase is generated using an image segmentation algorithm. Using non-rigid registration technology, the 4D cardiac image sequence is decoupled into a baseline model and a spatiotemporal deformation field, where the deformation field records the displacement vectors of each phase relative to the baseline model.

[0084] In some optional embodiments, the preoperative 3D model generation module includes a preoperative data acquisition unit, a segmentation unit, a multi-phase model construction unit, a baseline model determination unit, and a displacement vector determination unit. The preoperative data acquisition unit acquires preoperative medical images and various preoperative electrocardiogram (ECG) signals; the segmentation unit segments the preoperative medical images to obtain the cardiac region; the multi-phase model construction unit performs 3D reconstruction of the segmented cardiac region based on the preoperative cardiac phase corresponding to the preoperative ECG signal, obtaining a preoperative 3D model corresponding to each preoperative cardiac phase; the baseline model determination unit determines the preoperative 3D model corresponding to the target cardiac phase from the preoperative 3D models corresponding to each preoperative cardiac phase as the baseline 3D model; and the displacement vector determination unit calculates the displacement vector of each preoperative 3D model relative to the baseline 3D model based on a differential homeomorphic mapping algorithm.

[0085] Among them, combined Figure 6 As shown, Figure 6 The flowchart is a preoperative three-dimensional model generation module in one embodiment. In this embodiment, the preoperative data acquisition unit is used to acquire multimodal preoperative data, that is, to simultaneously acquire the patient's preoperative MRI image sequence (i.e., preoperative medical images) and the corresponding electrocardiogram signal (ECG signal, i.e., preoperative electrocardiogram signal).

[0086] The segmentation unit is used to segment the heart region. For example, the acquired MRI image sequence is input into a preset heart segmentation network to perform anatomical structure recognition and semantic segmentation to extract the heart region.

[0087] The multi-phase model construction unit is used to construct multi-phase three-dimensional point clouds. For example, by combining the cardiac phase corresponding to the preoperative electrocardiogram signal corresponding to the ECG signal, three-dimensional reconstruction is performed on the segmented MRI image, and dynamic cardiac three-dimensional point cloud data covering each preoperative cardiac phase within the cardiac cycle is extracted to obtain the preoperative three-dimensional model corresponding to each preoperative cardiac phase.

[0088] The benchmark model determination unit is used to establish a benchmark model. For example, in the three-dimensional point cloud of the heart at each preoperative cardiac phase, a specific phase (such as end-diastole or end-systole) is selected as the benchmark model.

[0089] The displacement vector determination unit is used to calculate the elastic deformation field. For example, using a non-rigid registration algorithm, the displacement vectors of the cardiac 3D point cloud relative to the static reference model for each of the remaining phases are calculated, thereby constructing an elastic deformation field characterizing the non-rigid motion of the heart wall. Finally, spatiotemporal decoupled data storage is performed, that is, a mapping relationship is established between the elastic deformation field (displacement vector) and the preoperative cardiac phase. The reference model and the displacement vector including the preoperative cardiac phase are stored together as preoperative 3D model data.

[0090] Furthermore, the differential homeomorphism mapping algorithm in this embodiment is a non-rigid registration algorithm that guarantees the continuity of the topological structure. By simulating fluid dynamics or elasticity models, it calculates a smooth and reversible coordinate transformation from one shape to another, which can ensure that the topological structure of the heart model mesh remains continuous during deformation and will not produce non-physical mesh intersections or damage.

[0091] In this embodiment, by using the differential homeomorphism mapping algorithm to accurately calculate the displacement vector between each phase and the reference model in the preoperative stage, a high-precision, topologically consistent spatiotemporal deformation field can be constructed, providing a reliable data foundation for real-time, high-quality deformation rendering during the operation.

[0092] In some alternative embodiments, the 3D modeling apparatus includes a parallel GPU; preoperative 3D model data is pre-stored in the GPU, and the intraoperative 3D model generation module is located in the GPU.

[0093] The 3D modeling device includes a parallel GPU. Preoperative 3D model data is stored in the GPU, and the intraoperative 3D model generation module is located on the GPU. By storing the preoperative 3D model data in the GPU and utilizing the GPU's parallel computing capabilities to perform vertex displacement calculations and rendering, the computational overhead of real-time rendering can be significantly reduced, improving the response speed and smoothness of the 3D modeling device.

[0094] In some optional embodiments, the 3D modeling device further includes: a keyframe construction module, used to acquire the initial endoscopic image corresponding to each intraoperative cardiac phase within a complete cardiac cycle as a keyframe; and to establish a first mapping relationship between each keyframe and the current intraoperative 3D model of the corresponding intraoperative cardiac phase, and to generate keyframe data based on the first mapping relationship.

[0095] The keyframe construction module generates keyframe data during the intraoperative preparation phase. A keyframe is an endoscopic image acquired at a specific cardiac phase, serving as a reference frame for subsequent matching and retrieval. Keyframe data consists of pre-constructed data nodes, each containing at least one endoscopic image (i.e., the keyframe), a first mapping relationship between the keyframe and the current intraoperative 3D model at the same cardiac phase, and feature point pairs located on the keyframe and the current intraoperative 3D model. The first mapping relationship refers to the initial spatial geometric correspondence between two-dimensional pixels in the keyframe and three-dimensional points on the current intraoperative 3D model.

[0096] Specifically, during the intraoperative preparation phase, when the keyframe addition command is triggered, the 3D modeling device begins acquiring initial endoscopic images. The surgeon manually registers the 3D heart model, after real-time ECG phase-driven deformation, with the initial endoscopic image (keyframe) in the current surgical endoscopic field of view, establishing a first mapping relationship between the keyframe and the current intraoperative 3D model corresponding to the cardiac motion phase during surgery. Subsequently, based on this first mapping relationship, the 3D modeling device combines the feature points extracted from the keyframe and their corresponding 3D points on the 3D model to generate a complete keyframe data node, which is then stored in the keyframe library.

[0097] In addition, it should be noted that each time a keyframe addition command is added, the 3D modeling device will continuously acquire each initial endoscopic image and the corresponding first mapping relationship throughout the entire cardiac cycle; and the number of keyframe addition commands is greater than or equal to one, so each cardiac phase during the operation has at least one initial endoscopic image, and the final number of keyframes is the product of the number of keyframe addition commands and the number of initial endoscopic images acquired in each cardiac cycle.

[0098] In the above embodiments, by constructing keyframe data covering each cardiac phase, a rich reference benchmark is provided for real-time intraoperative retrieval and matching, making subsequent phase-constrained retrieval possible.

[0099] In some optional embodiments, the keyframe construction module includes an initial matching unit, a first feature extraction unit, and a keyframe data unit.

[0100] Combination Figure 7 As shown, Figure 7 The diagram below illustrates the keyframe construction module process in one embodiment. In this embodiment, the initial matching unit is used to receive a keyframe addition instruction, obtain the initial endoscopic image corresponding to each intraoperative cardiac phase within a complete cardiac cycle as a keyframe based on the keyframe addition instruction, and establish a first mapping relationship between the keyframe and the corresponding intraoperative cardiac phase current intraoperative three-dimensional model.

[0101] Specifically, when a keyframe addition command is triggered, the 3D modeling device continuously acquires the initial endoscopic image corresponding to each intraoperative cardiac phase within a complete cardiac cycle as a keyframe. The doctor then manually performs initial registration between the 3D heart model, which has undergone real-time ECG phase-driven deformation, and the initial endoscopic image (in other embodiments, the 3D modeling device can also call an algorithm to automatically perform registration, which is not specifically limited here) to establish the first mapping relationship between each keyframe and the current intraoperative 3D model of the corresponding intraoperative cardiac phase. It should be noted that for each keyframe addition command, the doctor's manual initial registration (which can be replaced by an algorithm) is only performed once, but the initial matching unit will continue to acquire the initial endoscopic image corresponding to each intraoperative cardiac phase throughout the entire cardiac cycle, thereby constructing multiple keyframes.

[0102] Among them, combined Figure 8 As shown, Figure 8 This is a schematic diagram of the mapping relationship between image point coordinates and three-dimensional point coordinates in one embodiment. The entire geometric mapping architecture includes two core spatial layers: one is the three-dimensional model space under the three-dimensional camera coordinate system or world coordinate system (i.e., the preoperative three-dimensional model after deformation driven by preoperative decoupling and real-time ECG signals during surgery); the other is the two-dimensional image space under the two-dimensional camera imaging plane (i.e., the endoscopic image acquired in real time by the surgical endoscope).

[0103] Figure 8 In this model, a one-to-one mapping is established between the spatial 3D feature points on the surface of the 3D model and the corresponding 2D feature points in the 2D image at the locations of the anatomical structures. This mapping relationship is mathematically determined by the camera's intrinsic parameter matrix and the rigid transformation matrix (the first mapping relationship) representing the relative pose of the camera (endoscope) with respect to the organ.

[0104] The first feature extraction unit is used to perform feature point detection on the initial endoscopic image in parallel to obtain initial feature points, and to perform cardiac region segmentation on the initial endoscopic image to obtain the first target cardiac region. The initial feature points are then filtered to obtain the first target feature points located within the first target cardiac region. Here, the first target cardiac region refers to the cardiac anatomical region mask segmented from the initial endoscopic image by the cardiac region segmentation network. The first target feature point refers to the effective feature point located within this cardiac region mask.

[0105] The process involves extracting features from the initial endoscopic image using a dual-path parallel network: The current endoscopic frame image is acquired in real-time and input into a parallel feature point detection network and a heart region segmentation network. The feature point detection network detects initial feature points in the initial endoscopic image; the heart region segmentation network segments the initial endoscopic image to obtain the first target heart region.

[0106] Combination Figure 9 , Figure 9 This is a schematic diagram of feature point detection in one embodiment, wherein, Figure 9 The image presents two-dimensional endoscopic frame images captured by the endoscope during actual surgery, which include the target heart organ in dynamic motion, as well as several surgical instruments performing the surgical procedure.

[0107] During real-time processing, the 3D modeling device performs pixel-level analysis on the input endoscopic image through a heart region segmentation network, accurately identifying and outlining the anatomical contour (mask area) of the heart organ. This allows for dynamic spatial separation of the target organ from surrounding surgical instruments, reflective backgrounds, or bodily fluids and smoke, preventing instrument movement from interfering with feature extraction.

[0108] Combining a feature point detection network, the 3D modeling device specifically extracts local texture and geometric features from the surface of the heart organ after it has been constrained by the segmentation network, thereby locating and acquiring several highly robust organ feature points (such as...). Figure 9 The space-specific flag indicated by the middle circle.

[0109] In some optional embodiments, the first feature extraction unit is used to perform feature point detection on the initial endoscopic image to obtain anatomical feature points and / or general image texture feature points as organ feature points.

[0110] Anatomical feature points refer to feature points located at specific locations with clear anatomical significance, such as heart valves, vascular openings, or myocardial texture. They are highly robust and not easily affected by changes in lighting and viewing angle. General image texture feature points refer to local texture abrupt changes in an image extracted using traditional feature detection algorithms (such as SIFT, ORB, etc.).

[0111] The 3D modeling device uses a deep learning feature detection network (such as SuperPoint) to detect feature points in the initial endoscopic image. This network can simultaneously output general image texture feature points as well as anatomically significant feature points. Optionally, the 3D modeling device can also run dedicated anatomical feature point detection networks and general feature point detection networks separately, and then fuse the feature points output by both. By combining anatomical feature points and general texture feature points, a richer and more discriminative feature description can be constructed. Even under conditions where the texture of the heart surface is not obvious or the lighting conditions are poor, sufficient effective feature points can be guaranteed for matching, thereby further improving matching accuracy and system robustness.

[0112] Subsequently, by combining the initial feature points output by the feature point detection network and the anatomical region mask (i.e., the first target heart region) output by the heart region segmentation network, the first target feature points belonging only to the first target heart region are extracted and locked.

[0113] The keyframe data unit is used to determine the second target feature point corresponding to the first target feature point from the current intraoperative 3D model of the corresponding intraoperative motion phase based on the first mapping relationship, and to generate keyframe data based on the first mapping relationship, the feature point pair including the first target feature point and the second target feature point, and the intraoperative motion phase.

[0114] The second target feature point refers to the three-dimensional spatial point on the surface of the current intraoperative three-dimensional model that corresponds to the first target feature point (2D point).

[0115] In this embodiment, the spatial geometric relationship obtained from the initial registration calculation is combined to associate the first target feature point of the 2D heart region on the initial endoscopic image with the 3D model point on the surface of the three-dimensional heart model corresponding to the current phase, and establish a mapping pair between 2D image points and 3D model points as feature point pairs.

[0116] Finally, the key related information such as the feature points of the current keyframe, 2D-3D feature point pairs, manual registration matrix T (i.e., the first mapping relationship), and the motion phase of the surgical center are combined and packaged, recorded as independent nodes (i.e., keyframe data), and stored in the keyframe library of the 3D modeling device.

[0117] In the above embodiments, by performing feature point detection and heart region segmentation in parallel and filtering feature points based on a heart region mask, interfering feature points located in non-target areas such as surgical instruments and background are effectively eliminated, improving the accuracy and robustness of feature point pairs in keyframe data. Furthermore, by introducing ECG phase as a retrieval constraint, the search space for feature matching is reduced from a global full-cycle approach to a real-time phase window, effectively avoiding mismatches caused by similar features during systole and diastole, significantly improving the tracking robustness of the navigation algorithm in scenarios with rapidly beating hearts. In addition, by supporting the dynamic addition of keyframe data during surgery, the 3D modeling device can respond instantly to perspective changes or scene changes during the surgical process, eliminating the need for cumbersome preoperative full-coverage registration. The surgeon can trigger calibration at any time according to actual fusion needs, greatly simplifying the intraoperative preparation process, reducing the technical burden on doctors, and allowing them to focus more on core clinical operations.

[0118] In some optional embodiments, the keyframe construction module further includes: an integrity judgment unit, used to detect whether the generated keyframe data covers a complete cardiac cycle during the execution of the keyframe addition instruction; if the generated keyframe data does not cover a complete cardiac cycle, control the initial matching unit to continue acquiring the initial endoscopic image of the next cardiac phase in the operation until it is determined that a complete cardiac cycle has been covered, and then stop acquiring the initial endoscopic image of the next cardiac phase in the operation.

[0119] A complete cardiac cycle refers to the entire process from the start of one contraction to the start of the next, including the systolic and diastolic phases.

[0120] Specifically, such as Figure 7 As shown, during the execution of each keyframe addition instruction, after packaging and storing the keyframe data, the 3D modeling device automatically determines whether the keyframe data generated during this keyframe addition instruction execution has covered a complete physiological cardiac cycle. If the determination result is "no," the 3D modeling device control flow returns to the step of acquiring the next initial endoscopic image, and continues to cyclically acquire and construct new keyframe nodes at other intraoperative cardiac phase points within the cardiac cycle; if the determination result is "yes" (i.e., keyframe data covering the complete cardiac cycle has been acquired), the keyframe construction module process ends, that is, the keyframe addition instruction processing is completed. Other keyframe addition instructions can be received subsequently to construct keyframe data. In other words, the integrity judgment unit takes effect during each keyframe addition instruction. For each keyframe addition instruction, it can determine whether the keyframe data generated during its execution covers a complete physiological cardiac cycle. If it does, the keyframe addition instruction is completed, meaning that the initial endoscopic image corresponding to each cardiac phase within a complete cardiac cycle is acquired as a keyframe based on the keyframe addition instruction. If it does not cover a complete physiological cardiac cycle, the initial endoscopic image corresponding to the next cardiac phase is acquired as a keyframe, until the generated keyframe data covers a complete physiological cardiac cycle.

[0121] In this embodiment, by ensuring that the keyframe library covers the complete cardiac cycle, it is possible to guarantee that there are available candidate keyframes at any phase during the operation, thereby ensuring the reliability and continuity of phase-constrained retrieval and avoiding the problem of mismatch due to missing keyframes.

[0122] In some optional embodiments, the keyframe retrieval module includes a second feature extraction unit, a keyframe retrieval unit, and a matching unit.

[0123] The second feature extraction unit is used to acquire the current endoscope image, perform feature point detection on the current endoscope image in parallel to obtain the feature points to be matched, and perform heart region segmentation on the current endoscope image to obtain the second target heart region. The feature points to be matched are then filtered to obtain the third target feature points located in the second target heart region.

[0124] The third target feature point refers to the effective feature point located in the heart region of the current endoscopic image, which will be used as the feature to be matched in the current frame in subsequent matching.

[0125] The keyframe retrieval unit is used to determine each target keyframe data from pre-generated keyframe data based on the intraoperative motion phase corresponding to the current endoscopic image.

[0126] Among them, target keyframe data refers to all keyframe data in the keyframe library whose corresponding intraoperative cardiac phase and the current real-time cardiac phase difference are within the preset phase deviation threshold range.

[0127] The matching unit is used to match the third target feature point with the first target feature point in the feature point pair in the target keyframe data to obtain the matching keyframe data corresponding to the current endoscopic image.

[0128] Among them, the matching keyframe data refers to the keyframe data with the highest similarity to the feature points of the current frame.

[0129] Combination Figure 10 As shown, Figure 10 This is a flowchart of the intraoperative keyframe retrieval module in one embodiment; in this embodiment, the intraoperative keyframe retrieval module is implemented through dual-path parallel processing of image feature detection and physiological phase retrieval.

[0130] The first parallel branch is for feature point detection in the current endoscopic image: The current frame of the endoscopic image is acquired and input into the parallel feature point detection network and the heart region segmentation network, respectively. Noise points in non-target regions are filtered out, and joint analysis of global features and specific organ morphology is performed. Valid feature points (i.e., third target feature points) belonging only to the second target heart region in the current endoscopic image are extracted and locked. This, combined with deep learning-based automatic surgical instrument segmentation technology, allows the 3D modeling device to automatically switch to general scene features within the surgical field of view for auxiliary matching when key cardiac feature points are missing due to instrument occlusion. This mechanism significantly reduces the risk of navigation interruption caused by complex operations, making the 3D modeling device highly reliable in long-duration, high-intensity surgeries.

[0131] The second parallel branch is phase-constrained retrieval: the current intraoperative cardiac phase of the patient is obtained in real time from the ECG monitoring device at the current time step, a retrieval window is constructed based on the current intraoperative cardiac phase, preliminary screening is performed in the preset keyframe library, and a subset of candidate keyframes (i.e. target keyframe data) that matches the current intraoperative cardiac phase is filtered and locked.

[0132] In the intersection matching stage, the third target feature points obtained by the first branch are matched with the target keyframe data selected by the second branch across frames. The similarity between the current endoscope image and each target keyframe data is calculated, and the target keyframe data with the highest feature similarity is locked (i.e., the matching keyframe data).

[0133] Among them, combined Figure 11 As shown, Figure 11 This is a schematic diagram of feature point pair matching in one embodiment, where the entire matching architecture comprises two endoscopic image planes at different time sequences: Figure 11 The left side shows keyframe images that have been pre-built and stored in the keyframe library (which contain known anatomical structures with 3D model point mapping pairs). Figure 11 The right side shows the current endoscopic image captured in real time during the surgery. Due to the translation, rotation, or perspective stretching of the endoscopic camera during the surgery, the heart organs and surrounding surgical instruments in the two images exhibit different spatial poses and perspective shapes.

[0134] Figure 11 The horizontal dashed line connects local structures with the same anatomical significance in two frames, representing the 2D-2D feature point pair matching relationship established by the 3D modeling device between the two frames.

[0135] In actual operation, the 3D modeling device extracts feature points of the cardiac region from the current endoscopic image and performs feature point matching based on descriptor similarity among the target keyframe data filtered by ECG phase constraints. By measuring the topological consistency of feature point pairs, the most matching keyframe with the highest similarity is identified to obtain the matched keyframe data.

[0136] In the above embodiments, by introducing ECG phase constraints during the retrieval process, the candidate space for feature matching is locked in real time within the current physiological phase window, effectively filtering out interference frames under irrelevant phases and solving the problem of phase mismatch in high-speed dynamic scenes.

[0137] In some optional embodiments, the keyframe retrieval unit is further configured to adjust the phase deviation threshold when the rate of change of the heart rate of the intraoperative electrocardiogram exceeds the rate of change threshold; and to determine the difference between the intraoperative cardiac motion phase corresponding to the current endoscopic image and the intraoperative cardiac motion phase in each pre-generated keyframe data, and to acquire keyframe data with a difference less than the phase deviation threshold as target keyframe data when the difference is less than the phase deviation threshold.

[0138] The rate of change of heart rate refers to the magnitude of change in heart rate per unit time, used to characterize whether a sudden change has occurred in the patient's heart rate. The rate of change threshold is a preset empirical value; when the rate of change of heart rate exceeds this threshold, a significant change in heart rhythm is considered to have occurred. The phase deviation threshold refers to the allowable range of phase deviation, which determines the size of the filtering window for the target keyframe data.

[0139] Specifically, during the surgery, the 3D modeling device monitors the intraoperative electrocardiogram (ECG) signal in real time and calculates the rate of change of heart rate. When a sudden change in heart rate is detected (i.e., the rate of change of heart rate exceeds a threshold), the 3D modeling device dynamically expands or shifts the phase deviation threshold range to ensure that a sufficient number of target keyframe data can still be retrieved after the heart rate change. For example, when the heart rate suddenly increases, the 3D modeling device can appropriately expand the phase deviation threshold to cover the phase shift caused by the heart rate change. Subsequently, the 3D modeling device calculates the difference between the current cardiac phase and the cardiac phase recorded in each keyframe data in the keyframe library, and extracts all keyframe data whose difference is less than the adjusted phase deviation threshold as the target keyframe data.

[0140] In the above embodiments, by introducing a heart rate variability compensation mechanism, the phase deviation threshold is dynamically adjusted when a sudden change in heart rate is detected, which can maintain the matching accuracy and real-time response speed of the retrieval process and avoid matching failure due to heart rate changes.

[0141] In some optional embodiments, the fusion module includes a feature point pair generation unit, a second mapping relationship generation unit, and a fusion unit.

[0142] The feature point pair generation unit is used to generate target feature point pairs based on a first mapping relationship, matching feature point pairs in the keyframe data, and a third target feature point in the current endoscopic image. Each target feature point pair includes a second target feature point matching the feature point pair in the keyframe data and a third target feature point in the current endoscopic image. Here, the target feature point pair refers to a 2D-3D point pair used for PnP pose solving.

[0143] The second mapping relationship generation unit is used to generate a second mapping relationship based on the target feature point pairs. The second mapping relationship refers to the real-time spatial geometric correspondence between the current endoscopic image and the current intraoperative 3D model, and its mathematical representation is the real-time pose matrix of the camera (including rotation and translation).

[0144] The fusion unit is used to fuse the current intraoperative 3D model with the current endoscopic image based on the second mapping relationship.

[0145] Combination Figure 12 As shown, Figure 12 This is a flowchart illustrating the fusion module in one embodiment. In this embodiment, the matching keyframe data stores the first target feature point (2D point) and the corresponding second target feature point (3D point) on the matching keyframe. Since the third target feature point in the current endoscopic image and the first target feature point in the matching keyframe are established as a 2D-2D correspondence based on a feature matching algorithm (such as descriptor similarity calculation), the correspondence between the third target feature point (current frame 2D point) and the second target feature point (keyframe 3D point) can be indirectly established, i.e., target feature point pairs. Subsequently, these 2D-3D point pairs are input into the fluoroscopic n-point (PnP) solver, and robust pose calculation is performed using the Random Sample Consensus Algorithm (RANSAC) to solve for the real-time pose matrix of the surgical endoscope camera at the current time step (i.e., the second mapping relationship). Finally, using this pose matrix, the current 3D heart model, which has undergone deformation via real-time ECG phase driving, is subjected to perspective transformation and reprojection, and then superimposed onto the current endoscopic video stream to achieve dynamic follow-up enhancement display of the 3D virtual-real fusion model.

[0146] In this embodiment, by combining the first mapping relationship pre-stored in the matching keyframe, the 2D feature points of the current frame are linked with the 3D points on the 3D model, thereby realizing the accurate calculation of the camera pose and ultimately driving the dynamic 3D model and the endoscope field of view to perform precise virtual-real fusion.

[0147] In some optional embodiments, the 3D modeling device further includes: a fusion quality monitoring module, used to acquire the mapping feature points of the current intraoperative 3D model to the current endoscopic image, and generate a fusion error based on the distance between the mapping feature points and the fourth target feature point in the current endoscopic image; and output alarm information when the fusion error corresponding to a consecutive preset number of current endoscopic images is greater than the error threshold.

[0148] Here, the mapped feature point refers to the two-dimensional point obtained by projecting the feature points on the current intraoperative 3D model onto the endoscopic image plane according to the currently calculated second mapping relationship (camera pose matrix). The fourth target feature point refers to the actual detected feature point in the current endoscopic image that corresponds to the above mapped feature point. The fusion error can be the reprojection error, that is, the pixel distance between the mapped feature point and the fourth target feature point.

[0149] Specifically, after each frame is fused and displayed, the 3D modeling device projects a set of known feature points on the current intraoperative 3D model onto the endoscopic image plane according to the second mapping relationship, obtaining a set of mapped feature points. These mapped feature points are then compared with the fourth target feature points actually detected in the current endoscopic image, and the pixel distance between them is calculated as the fusion error. When the fusion error of multiple consecutive frames (e.g., 5 consecutive frames) exceeds a preset error threshold (e.g., 10 pixels), the 3D modeling device outputs an alarm message via an image trolley or sound, indicating to the doctor that the current fusion quality has deteriorated and manual recalibration may be necessary.

[0150] In this embodiment, by monitoring the reprojection error in real time and issuing an alarm when the error accumulates beyond the limit, the doctor can be promptly prompted to intervene (such as recalibration), which effectively ensures the long-term operational reliability of the 3D modeling device.

[0151] In some optional embodiments, the 3D modeling apparatus further includes: a display module for displaying the fused medical image and adjusting the display parameters of the current intraoperative 3D model according to the rendering instructions corresponding to the current intraoperative 3D model. The display parameters include at least one of rendering transparency, magnification, cross-sectional cutting, and rotational viewpoint.

[0152] Combination Figure 12 As shown, Figure 12 This is a functional diagram illustrating a coronary artery bypass surgery scenario in one embodiment. The display module is connected to an image carriage for real-time display of fused medical images. Doctors can issue rendering commands through interaction with the image carriage (e.g., touchscreen operation, keyboard input, or voice commands). For example, throughout the entire coronary artery bypass surgery, doctors can see both the actual blood vessels and instruments through the endoscope, and simultaneously view a high-precision, highly synchronized (rigid camera pose tracking and non-rigid heart rhythm deformation tracking) three-dimensional anatomical structure projection in real time. This function effectively assists doctors in establishing high-precision "virtual spatial perspective navigation" on the dynamically beating surface of the heart, significantly improving the accuracy and safety of coronary artery target localization and vascular anastomosis procedures.

[0153] Furthermore, when a doctor needs to observe tissue obscured beneath the model, they can issue a command to adjust the transparency. The display module responds to this command by reducing the rendering transparency of the current intraoperative 3D model (e.g., adjusting the transparency from 0% to 50%), making the actual tissue visible in the original endoscopic image. For another example, doctors can use the rotation viewpoint command to observe the alignment between the 3D model and the real organ from different angles. Yet another example is the use of the section cutting command to virtually cut the 3D model to observe the direction of internal blood vessels.

[0154] In this embodiment, by providing interactive visualization functions, doctors can flexibly adjust the display form of the model according to clinical needs, which greatly improves the clinical practicality and ease of operation of the 3D modeling device.

[0155] In the above embodiments, by acquiring ECG signals in real time during surgery and introducing physiological phase prior constraints during keyframe retrieval, the search space for feature matching is limited to a physiological window consistent with the current cardiac cycle. This significantly reduces the probability of mismatches in multi-phase and complex backgrounds, ensuring that the 3D modeling device can still quickly and accurately lock the optimal reference benchmark even under conditions of intense cardiac activity and frequent changes in the surgical field, greatly enhancing the robustness of intraoperative augmented reality fusion. Furthermore, the real-time 3D scene reconstruction and point cloud fusion function upgrades single model overlay to stereoscopic scene fusion. This enhanced stereoscopic information representation not only assists the surgeon in precise positioning but also provides surgical assistants with a more intuitive spatial reference, helping to reduce information errors in team collaboration.

[0156] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0157] In some optional embodiments, this application also provides a navigation system that includes the three-dimensional modeling device in any of the above embodiments.

[0158] Combination Figure 13As shown, Figure 13 This is a functional diagram illustrating a coronary artery bypass surgery scenario in one embodiment. The display module is connected to an image carriage for real-time display of fused medical images. Doctors can issue rendering commands through interaction with the image carriage (e.g., touchscreen operation, keyboard input, or voice commands). For example, throughout the entire coronary artery bypass surgery, doctors can see both the actual blood vessels and instruments through the endoscope, and simultaneously view a high-precision, highly synchronized (rigid camera pose tracking and non-rigid heart rhythm deformation tracking) three-dimensional anatomical structure projection in real time. This function effectively assists doctors in establishing high-precision "virtual spatial perspective navigation" on the dynamically beating surface of the heart, significantly improving the accuracy and safety of coronary artery target localization and vascular anastomosis procedures.

[0159] Furthermore, when a doctor needs to observe tissue obscured beneath the model, they can issue a command to adjust the transparency. The display module responds to this command by reducing the rendering transparency of the current intraoperative 3D model (e.g., adjusting the transparency from 0% to 50%), making the actual tissue visible in the original endoscopic image. For another example, doctors can use the rotation viewpoint command to observe the alignment between the 3D model and the real organ from different angles. Yet another example is the use of the section cutting command to virtually cut the 3D model to observe the direction of internal blood vessels.

[0160] In this embodiment, by providing interactive visualization functions, doctors can flexibly adjust the display form of the model according to clinical needs, which greatly improves the clinical practicality and ease of operation of the 3D modeling device.

[0161] In the above embodiments, by acquiring ECG signals in real time during surgery and introducing physiological phase prior constraints during keyframe retrieval, the search space for feature matching is limited to a physiological window consistent with the current cardiac cycle. This significantly reduces the probability of mismatches in multi-phase and complex backgrounds, ensuring that the 3D modeling device can still quickly and accurately lock the optimal reference benchmark even under conditions of intense cardiac activity and frequent changes in the surgical field, greatly enhancing the robustness of intraoperative augmented reality fusion. Furthermore, the real-time 3D scene reconstruction and point cloud fusion function upgrades single model overlay to stereoscopic scene fusion. This enhanced stereoscopic information representation not only assists the surgeon in precise positioning but also provides surgical assistants with a more intuitive spatial reference, helping to reduce information errors in team collaboration.

[0162] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0163] In some optional embodiments, this application also provides a surgical system that includes the electrocardiogram signal-based medical image fusion system of any of the above embodiments.

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0166] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A three-dimensional modeling device, characterized in that, The device includes: The intraoperative 3D model generation module is used to generate the current intraoperative 3D model corresponding to each intraoperative cardiac motion phase based on the preoperative 3D model data and the intraoperative electrocardiographic signal. The keyframe retrieval module is used to acquire the current endoscopic image and the intraoperative motion phase corresponding to the current endoscopic image, determine target keyframe data from pre-generated keyframe data based on the intraoperative motion phase corresponding to the current endoscopic image, and determine matching keyframe data corresponding to the current endoscopic image from each target keyframe data; the keyframe data includes a first mapping relationship between each keyframe and the current intraoperative 3D model of the same intraoperative motion phase, as well as feature point pairs; The fusion module is used to determine a second mapping relationship between the current endoscopic image and the corresponding current intraoperative 3D model based on a first mapping relationship in the matching keyframe data and the feature point pairs, and to fuse the current intraoperative 3D model with the current endoscopic image based on the second mapping relationship.

2. The three-dimensional modeling device according to claim 1, characterized in that, The intraoperative 3D model generation module is specifically used to acquire preoperative 3D model data and intraoperative electrocardiogram (ECG) signals. The preoperative 3D model data includes a baseline 3D model and displacement vectors corresponding to each cardiac phase. Based on the intraoperative cardiac phase corresponding to the intraoperative ECG signals, the target displacement vector is determined from the preoperative 3D model data, and the baseline 3D model is deformed based on the target displacement vector to obtain the current intraoperative 3D model corresponding to each intraoperative cardiac phase.

3. The three-dimensional modeling device according to claim 2, characterized in that, The device further includes: The preoperative 3D model generation module is used to acquire preoperative medical images and preoperative electrocardiogram (ECG) signals, generate a preoperative 3D model of the cardiac phase corresponding to the preoperative ECG signals based on the preoperative medical images, and generate a baseline 3D model and displacement vectors corresponding to each cardiac phase based on the preoperative 3D model of the cardiac phase corresponding to the preoperative ECG signals, as preoperative 3D model data.

4. The three-dimensional modeling device according to claim 3, characterized in that, The preoperative 3D model generation module includes: The preoperative data acquisition unit is used to acquire preoperative medical images and various preoperative electrocardiogram signals; A segmentation unit is used to segment the preoperative medical images to obtain the cardiac region; The multi-phase model construction unit is used to perform three-dimensional reconstruction of the segmented heart region based on the preoperative cardiac phase corresponding to the preoperative electrocardiogram signal, so as to obtain the preoperative three-dimensional model corresponding to each preoperative cardiac phase. The baseline model determination unit is used to determine the preoperative three-dimensional model corresponding to the target cardiac phase from the preoperative three-dimensional models corresponding to each of the preoperative cardiac phases as the baseline three-dimensional model. The displacement vector determination unit is used to calculate the displacement vector of each of the preoperative three-dimensional models relative to the reference three-dimensional model based on the differential homeomorphism mapping algorithm.

5. The three-dimensional modeling apparatus according to any one of claims 1 to 4, characterized in that, The device includes a parallel GPU; the preoperative 3D model data is pre-stored in the GPU, and the intraoperative 3D model generation module is located in the GPU.

6. The three-dimensional modeling apparatus according to any one of claims 1 to 4, characterized in that, The device further includes: The keyframe construction module is used to acquire the initial endoscopic image corresponding to each intraoperative cardiac phase within a complete cardiac cycle as a keyframe; and to establish a first mapping relationship between each keyframe and the current intraoperative three-dimensional model corresponding to the intraoperative cardiac phase, and generate keyframe data based on the first mapping relationship.

7. The three-dimensional modeling device according to claim 6, characterized in that, The keyframe construction module includes: The initial matching unit is used to receive a keyframe addition instruction, obtain an initial endoscopic image corresponding to each intraoperative cardiac phase within a complete cardiac cycle as a keyframe based on the keyframe addition instruction, and establish a first mapping relationship between the keyframe and the corresponding intraoperative cardiac phase of the current intraoperative three-dimensional model. The first feature extraction unit is used to perform feature point detection on the initial endoscopic image in parallel to obtain initial feature points and to perform heart region segmentation on the initial endoscopic image to obtain a first target heart region, and to filter the initial feature points to obtain a first target feature point located in the first target heart region. The keyframe data unit is used to determine the second target feature point corresponding to the first target feature point from the current intraoperative 3D model corresponding to the intraoperative motion phase based on the first mapping relationship, and to generate keyframe data based on the first mapping relationship, feature point pairs including the first target feature point and the second target feature point, and the intraoperative motion phase.

8. The three-dimensional modeling device according to claim 7, characterized in that, The keyframe construction module also includes: The integrity judgment unit is used to detect whether the generated keyframe data covers a complete cardiac cycle during the execution of the keyframe addition instruction; if the generated keyframe data does not cover a complete cardiac cycle, the unit controls the initial matching unit to continue acquiring the initial endoscopic image of the next cardiac phase during the operation until it is determined that a complete cardiac cycle has been covered, and then stops acquiring the initial endoscopic image of the next cardiac phase during the operation.

9. The three-dimensional modeling device according to claim 7, characterized in that, The first feature extraction unit is used to perform feature point detection on the initial endoscopic image to obtain anatomical feature points and / or general image texture feature points.

10. The three-dimensional modeling apparatus according to any one of claims 1 to 4, characterized in that, The keyframe retrieval module includes: The second feature extraction unit is used to acquire the current endoscope image, perform feature point detection on the current endoscope image in parallel to obtain the feature points to be matched, and perform heart region segmentation on the current endoscope image to obtain the second target heart region, and filter the feature points to be matched to obtain the third target feature points located in the second target heart region. The keyframe retrieval unit is used to determine each target keyframe data from each pre-generated keyframe data based on the intraoperative motion phase corresponding to the current endoscopic image. The matching unit is used to match the third target feature point with the first target feature point of the feature point pair in the target keyframe data to obtain the matching keyframe data corresponding to the current endoscopic image.

11. The three-dimensional modeling apparatus according to claim 10, characterized in that, The keyframe retrieval unit is also used to adjust the phase deviation threshold when the rate of change of the heart rate of the intraoperative electrocardiogram exceeds the rate of change threshold; and to determine the difference between the intraoperative cardiac motion phase corresponding to the current endoscopic image and the intraoperative cardiac motion phase in each pre-generated keyframe data, and to acquire keyframe data with a difference less than the phase deviation threshold as target keyframe data when the difference is less than the phase deviation threshold.

12. The three-dimensional modeling apparatus according to any one of claims 1 to 4, characterized in that, The fusion module includes: The feature point pair generation unit is used to generate target feature point pairs based on the first mapping relationship, the feature point pairs in the matching keyframe data, and the third target feature point in the current endoscopic image. Each target feature point pair includes the second target feature point of the feature point pair in the matching keyframe data and the third target feature point in the current endoscopic image. The second mapping relationship generation unit is used to generate a second mapping relationship based on the target feature point pair; The fusion unit is used to fuse the current intraoperative 3D model with the current endoscopic image based on the second mapping relationship.

13. The three-dimensional modeling device according to claim 12, characterized in that, The device further includes: The fusion quality monitoring module is used to acquire the mapping feature points of the current intraoperative 3D model to the current endoscopic image, and generate a fusion error based on the distance between the mapping feature points and the fourth target feature point in the current endoscopic image; when the fusion error corresponding to a consecutive preset number of the current endoscopic images is greater than the error threshold, an alarm message is output.

14. The three-dimensional modeling apparatus according to any one of claims 1 to 4, characterized in that, The device further includes: The display module is used to display the fused medical image and adjust the display parameters of the current intraoperative 3D model according to the rendering instructions corresponding to the current intraoperative 3D model. The display parameters include at least one of rendering transparency, magnification, cross-sectional cutting, and rotation viewpoint.

15. A navigation system, characterized in that, The navigation system includes the three-dimensional modeling device according to any one of claims 1 to 14.

Images