Augmented Reality-Based Navigation Method and Device Based on Cross-Modal Identification and Temporal Phase

Abstract

This invention provides an augmented reality interventional navigation method and device based on cross-modal identification and temporal phase. The method includes: S10, deploying several surface markers on the patient's body surface at a set location, acquiring preoperative CT image data of the surface markers, and constructing a three-dimensional anatomical model based on the preoperative CT image data; S20, tracking the interventional device to obtain its spatial pose; S30, establishing a cross-modal unified coordinate mapping relationship between the three-dimensional position of the surface markers in the intraoperative optical camera coordinate system and the spatial pose data of the interventional device, thereby achieving spatial registration of the two in a unified coordinate system; S40, extracting a respiratory curve using the displacement data of the real-time position information of the surface markers to determine the stable phase within the respiratory cycle. This invention solves the problems of respiratory motion interference and insufficient image registration accuracy in percutaneous interventional surgery.

Description

Technical Field

[0001] This invention relates to the field of medical image navigation and augmented reality technology, and in particular to an augmented reality interventional navigation method and device based on cross-modal identification and temporal phase. Background Technology

[0002] Current interventional surgical navigation methods primarily rely on CT or ultrasound imaging. While CT provides clear anatomical information, it lacks real-time capabilities and poses radiation risks; ultrasound, although real-time, has limited penetration depth and is difficult to apply in gaseous areas such as the lungs. Augmented reality (AR) technology has been applied to surgical navigation, but it still has shortcomings in terms of respiratory motion interference and registration accuracy.

[0003] Currently, most respiratory gating systems use external chest and abdominal sensors, which are not deeply integrated with the navigation system. This makes operation complex and introduces synchronization errors, leading to deviations between the puncture trajectory and the target lesion, thus increasing surgical risks. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an augmented reality interventional navigation method and device based on cross-modal identification and temporal phase, so as to solve the problems of respiratory motion interference and insufficient image registration accuracy in percutaneous interventional surgery.

[0005] In a first aspect, the present invention provides an augmented reality-guided navigation method based on cross-modal identification and temporal phase, comprising the following steps: S10. Place several surface markers at predetermined locations on the patient's body surface, collect the three-dimensional position of the surface markers in the preoperative CT coordinate system, and construct a preoperative three-dimensional anatomical model based on the three-dimensional position in the preoperative CT coordinate system. S20. Track the interventional device to obtain its spatial pose: S21. Introduce a two-dimensional coding structure on the surface of interventional devices; S22. Capture the two-dimensional projection coordinates of the two-dimensional coding structure using an optical vision device; S23. Solve the spatial pose of the interventional device using the two-dimensional projection coordinates; S24. Apply a temporal filtering algorithm to the spatial pose data and output the spatial pose data. S30. The three-dimensional position of the surface marker in the intraoperative optical camera coordinate system is captured by the optical vision device. A cross-modal unified coordinate mapping relationship is established between the three-dimensional position of the surface marker in the intraoperative optical camera coordinate system and the spatial pose data of the interventional instrument. The spatial pose data of the interventional instrument is converted to the coordinate system of the preoperative three-dimensional anatomical model through the mapping relationship, thereby realizing the spatial registration of the two in the unified coordinate system. S40. Based on the displacement data of the three-dimensional position of the surface marker captured in S30 under a unified coordinate system, extract the respiratory curve and determine the stable phase within the respiratory cycle: S41. Extract the displacement sequence of the surface marker in the direction perpendicular to the patient's chest cavity; S42. Bandpass filter is applied to the displacement sequence of the marker body to obtain the respiratory signal; S43. Perform phase extraction on the respiratory signal to obtain the instantaneous phase; S44. By using the derivative information and instantaneous phase of the respiratory signal, determine the stable phase in the respiratory cycle to guide the doctor on the timing of puncture.

[0006] Furthermore, in step S20, tracking the interventional device to obtain its spatial pose is specifically as follows: S21. A two-dimensional coding structure is introduced on the surface of the interventional device, wherein the two-dimensional coding structure has a highly redundant feature point distribution; let the three-dimensional coordinate set of the coding points of the two-dimensional coding structure be: ; M is the three-dimensional coordinate set of the coding points of the two-bit coding structure. The three-dimensional coordinates of the coding point in the two-bit coding structure, the It is a set of three-dimensional real numbers; S22. The two-dimensional projection coordinates of the two-dimensional coding structure on the imaging plane are captured in real time using an optical vision device. The two-dimensional projection coordinates are: ; The Let i be the two-dimensional coordinates of the i-th encoded point projected onto the image plane. Represents a perspective projection model, the Describe the orientation of the interventional instrument relative to the coordinate system of the optical vision device. Describe the position of the interventional device in the camera coordinate system; S23. Establish the geometric constraint relationship between the three-dimensional point set and the two-dimensional projection through the camera imaging model, and solve for the spatial pose of the interventional device: Minimize the projection error function: ; E(R,T) is the projection error function, and (R,T) is the spatial pose of the interventional device. The projection error describes the squared error of a single feature point, which can be used to achieve the optimal estimation of the spatial pose (R,T) of the interventional device. When some feature points of the interventional device are occluded or the lighting changes, the RANSAC algorithm is introduced; S24. Apply a temporal filtering algorithm to the pose sequence of the spatial pose to compensate for motion interference: The filtering algorithm is as follows: ; The This represents the pose estimation state at time t, the stated For optical observation input, F is the state transition, H is the observation matrix, and the... This is the filter gain; The optimization constraint for the total projection error function is: ; The For two-dimensional observation coordinates, the... Let R be a two-dimensional projected coordinate system, and let R be a rotation matrix. The coordinates of the encoding points in the two-bit encoding structure are 3D coordinates, and T is the translation matrix.

[0007] Furthermore, in step S30, the real-time position information of the surface marker is captured by an optical vision device, and a cross-modal unified coordinate mapping relationship is established between the real-time position information of the surface marker and the spatial pose of the interventional device; specifically: Let the position of the surface landmark in the preoperative CT coordinate system be: ; Let the position of the surface marker in the coordinate system of the intraoperative optical vision device be: ; The registration process can then be abstracted as a rigid body transformation: ; The optimal mapping of surface markers in the preoperative CT coordinate system and the intraoperative optical vision device coordinate system can be obtained by minimizing the following registration error function: ; The The set of three-dimensional location points of the surface marker in the preoperative CT coordinate system, the The coordinates of a three-dimensional point in the preoperative CT coordinate system, the The set of real numbers in three-dimensional space, the The set of three-dimensional points in the coordinate system of the intraoperative optical vision device, the These are the coordinates of a three-dimensional point in the coordinate system of the intraoperative optical vision device.

[0008] Furthermore, in step S40, the displacement data of the real-time position information of the surface marker is used to extract the respiratory curve and determine the stable phase within the respiratory cycle, specifically as follows: S41. Extract the displacement sequence of the surface marker in the direction perpendicular to the patient's chest cavity. : ; The This refers to the displacement of the surface marker in the direction perpendicular to the thoracic cavity; S42. Bandpass filtering is applied to the displacement sequence of the marker body. The bandpass filtering consists of low-frequency motion filtering and high-frequency noise suppression to obtain the respiratory signal. : ; S43. Perform phase extraction on the respiratory signal and obtain the instantaneous phase using Hilbert transform or wavelet packet decomposition: ; H[·] represents the Hilbert transform. It is the instantaneous phase; S44. Based on the derivative information and instantaneous phase of the respiratory signal, determine the stable phase in the respiratory cycle and generate a safety window to guide the timing of puncture. Specifically, determining the stable phase in the respiratory cycle involves: If satisfied If so, this interval is determined to be a stable respiratory phase; If not satisfied If so, this interval is determined to be a non-breathing stable phase.

[0009] Furthermore, it also includes the following steps: S50. In the augmented reality display device, the three-dimensional anatomical model, the trajectory and timing prompts of the interventional instruments, as well as the safety window and prompt window are provided in a multi-view manner. The safety window is defined as follows: when the respiratory phase is determined to be stable, a safety window pops up in the augmented reality display device, indicating that the doctor has permission to perform the surgical procedure at this moment; The prompt window is a pop-up window that appears on the augmented reality display device when the patient is determined to be in a non-respiratory phase, prompting the doctor that surgical procedures are not allowed at this time.

[0010] Secondly, the present invention provides an augmented reality-guided navigation device based on cross-modal identification and temporal phase, comprising the following modules: Preoperative data acquisition module: Several surface markers are placed at predetermined locations on the patient's body surface, and the three-dimensional positions of the surface markers in the preoperative CT coordinate system are acquired. Based on the three-dimensional positions in the preoperative CT coordinate system, a preoperative three-dimensional anatomical model is constructed. Intraoperative instrument data acquisition module: Tracks interventional instruments and obtains their spatial pose. The intraoperative instrument data acquisition module includes the following units: The device identification unit introduces a two-dimensional coding structure on the surface of the interventional device; The coordinate acquisition unit captures the two-dimensional projected coordinates of the two-dimensional encoded structure through an optical vision device; The pose unit is used to solve the spatial pose of the interventional device through the two-dimensional projected coordinates. The temporal filtering unit applies a temporal filtering algorithm to the pose sequence of the spatial pose and outputs spatial pose data. Cross-modal coordinate mapping module: The three-dimensional position of the surface marker in the intraoperative optical camera coordinate system is captured by the optical vision device. A cross-modal unified coordinate mapping relationship is established between the three-dimensional position of the surface marker in the intraoperative optical camera coordinate system and the spatial pose data of the interventional instrument. Through this mapping relationship, the spatial pose data of the interventional instrument is transformed into the coordinate system of the preoperative three-dimensional anatomical model, thereby realizing the spatial registration of the two in a unified coordinate system. The temporal phase analysis module extracts the respiratory curve using displacement data from the real-time position information of the surface markers and determines the stable phase within the respiratory cycle. Specifically, it includes the following units: The sequence extraction unit extracts the displacement sequence of the surface markers in a direction perpendicular to the patient's thoracic cavity; The respiratory signal unit performs bandpass filtering on the marker displacement sequence to obtain a respiratory signal; The phase extraction unit extracts the phase of the respiratory signal to obtain the instantaneous phase; The determination window unit determines the stable phase in the respiratory cycle by using the derivative information and instantaneous phase of the respiratory signal, thus guiding the doctor on the timing of puncture.

[0011] Furthermore, the intraoperative instrument data acquisition module specifically comprises: The device identification unit introduces a two-dimensional coding structure on the surface of the interventional device. This two-dimensional coding structure has a highly redundant feature point distribution. Let the three-dimensional coordinate set of the coding points of the two-dimensional coding structure be: ; M is the three-dimensional coordinate set of the coding points of the two-bit coding structure. The three-dimensional coordinates of the coding point in the two-bit coding structure, the It is a set of three-dimensional real numbers; The coordinate acquisition unit captures the two-dimensional projection coordinates of the two-dimensional coded structure on the imaging plane in real time through an optical vision device. The two-dimensional projection coordinates are: ; The Let i be the two-dimensional coordinates of the i-th encoded point projected onto the image plane. Represents a perspective projection model, the Describe the orientation of the interventional instrument relative to the coordinate system of the optical vision device. Describe the position of the interventional device in the camera coordinate system; The pose unit is solved by establishing the geometric constraint relationship between the 3D point set and the 2D projection through the camera imaging model, and then solving for the spatial pose of the interventional device. Minimize the projection error function: ; E(R,T) is the projection error function, and (R,T) is the spatial pose of the interventional device. The projection error describes the squared error of a single feature point, which can be used to achieve the optimal estimation of the spatial pose (R,T) of the interventional device. When some feature points of the interventional device are occluded or the lighting changes, the RANSAC algorithm is introduced; The temporal filtering unit applies a temporal filtering algorithm to the pose sequence of the spatial pose to compensate for motion interference: The filtering algorithm is as follows: ; The This represents the pose estimation state at time t, the stated For optical observation input, F is the state transition, H is the observation matrix, and the... This is the filter gain; The optimization constraint for the total projection error function is: ; The Let R be the two-dimensional observation coordinates, and let R be the rotation matrix. The coordinates of the encoding points in the two-bit encoding structure are 3D coordinates, and T is the translation matrix.

[0012] An augmented reality-guided navigation device based on cross-modal identification and temporal phase, wherein the cross-modal coordinate mapping module specifically comprises: Let the position of the surface landmark in the preoperative CT coordinate system be: ; Let the position of the surface marker in the coordinate system of the intraoperative optical vision device be: ; The registration process can then be abstracted as a rigid body transformation: ; The optimal mapping of surface markers in the preoperative CT coordinate system and the intraoperative optical vision device coordinate system can be obtained by minimizing the following registration error function: ; The The set of three-dimensional location points of the surface marker in the preoperative CT coordinate system, the The coordinates of a three-dimensional point in the preoperative CT coordinate system, the The set of real numbers in three-dimensional space, the The set of three-dimensional points in the coordinate system of the intraoperative optical vision device, the These are the coordinates of a three-dimensional point in the coordinate system of the intraoperative optical vision device.

[0013] Furthermore, the time-series phase analysis module: extracts the respiratory curve using the displacement data of the real-time position information of the surface marker, and determines the stable phase within the respiratory cycle, specifically: The sequence extraction unit extracts the displacement sequence of the surface markers in a direction perpendicular to the patient's thoracic cavity. : ; The This refers to the displacement of the surface marker in the direction perpendicular to the thoracic cavity; The respiratory signal unit performs bandpass filtering on the displacement sequence of the marker body. The bandpass filtering consists of low-frequency motion filtering and high-frequency noise suppression to obtain the respiratory signal. : ; The phase extraction unit extracts the phase of the respiratory signal and obtains the instantaneous phase using Hilbert transform or wavelet packet decomposition. ; H[·] represents the Hilbert transform. It is the instantaneous phase; The determination window unit, through the derivative information and instantaneous phase of the respiratory signal, determines the stable phase in the respiratory cycle and generates a safety window to guide the timing of puncture. Specifically, determining the stable phase in the respiratory cycle involves: If satisfied If so, this interval is determined to be a stable respiratory phase; If not satisfied If so, this interval is determined to be a non-breathing stable phase.

[0014] Furthermore, it also includes a human-computer interaction and prompting module: in the augmented reality display device, it provides the three-dimensional anatomical model, the trajectory and timing prompts of the interventional instruments, as well as safety windows and prompt windows in a multi-view manner; The safety window is defined as follows: when the respiratory phase is determined to be stable, a safety window pops up in the augmented reality display device, indicating that the doctor has permission to perform the surgical procedure at this moment; The prompt window is a pop-up window that appears on the augmented reality display device when the patient is determined to be in a non-respiratory phase, prompting the doctor that surgical procedures are not allowed at this time.

[0015] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: This invention achieves a unified mapping between anatomical models and real-time operational scenarios by introducing cross-domain perceptible surface markers between preoperative image modeling and intraoperative visual tracking. Meanwhile, this invention utilizes the spatial micro-displacement of the marker under respiratory drive to construct a time-series signal model. Through phase decoupling and dynamic reconstruction algorithms, the respiratory cycle is extracted and a safe operation window is generated to remind and guide the doctor's operation. This invention significantly improves the precision and safety of interventional surgery by deeply integrating cross-modal registration with respiratory dynamics gating and presenting it in the form of interactive prompts in an augmented reality interface.

[0016] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the device in Embodiment 2 of the present invention. Detailed Implementation

[0019] This application provides an augmented reality-guided navigation method and apparatus based on cross-modal identification and temporal phase.

[0020] Example 1 This embodiment provides an augmented reality-guided navigation method based on cross-modal identification and temporal phase, such as... Figure 1 As shown, it includes the following steps: S10. Several surface markers are placed on the patient's body surface, and preoperative CT image data of the surface markers are collected. Based on the preoperative CT image data, a three-dimensional anatomical model that can be displayed and calculated by the AR system is constructed. S20. Tracking of interventional devices: A two-dimensional coding structure with highly redundant feature point distribution is introduced on the surface of the interventional device. The geometric constraint relationship between the three-dimensional point set and the two-dimensional projection is established through the camera imaging model. The virtual navigation model is superimposed on the real scene. The system recognizes and uses PnP optimization and least squares estimation to achieve robust solution of device pose. The pose sequence is dynamically compensated by combining a time-series filtering algorithm, thereby ensuring the consistency and stability of the device trajectory and the virtual navigation model at sub-pixel accuracy. Specifically, it includes the following sub-steps: S21. A two-dimensional coding structure with high robustness and redundancy distribution is deployed on the surface of the interventional device; let the three-dimensional coordinate set of the coding points of the two-dimensional coding structure be: ; M is the three-dimensional coordinate set of the coding points in a two-bit coding structure. The above refers to the three-dimensional coordinates of the coding points in a two-bit coding structure. The above refers to a set of three-dimensional real numbers. S22. The intraoperative optical vision device captures the two-dimensional coded structure in real time to obtain its two-dimensional projection coordinates on the imaging plane, wherein the two-dimensional projection coordinates are: ; The coordinates of the i-th encoding point projected onto the image plane are two-dimensional coordinates. Represents a perspective projection model, the Describe the orientation of the interventional instrument relative to the coordinate system of the optical vision device. Describe the position of the interventional device in the camera coordinate system; S23. Establish the geometric constraint relationship between the three-dimensional point set and the two-dimensional projection through the camera imaging model, and solve for the spatial pose of the interventional device: Minimize the projection error function: ; E(R,T) is the projection error function, which measures the deviation between the predicted projected coordinates and the actual observed coordinates, and is the optimization objective for solving (R,T); (R,T) is the spatial pose of the interventional device. The projection error describes the squared error of a single feature point, which can be used to achieve the optimal estimation of the spatial pose (R,T) of the interventional device. This process can be solved using PnP (Perspective-n-Point) optimization or the least squares method. To improve the robustness of tracking, when some feature points of the interventional device are occluded or the illumination changes, the RANSAC algorithm is introduced to ensure the stability of the pose calculation. S24. Considering the interference of breathing and hand movement during the operation, a temporal filtering algorithm is used on the pose sequence of the spatial pose to compensate for motion interference: The filtering algorithm is as follows: ; The This represents the pose estimation state at time t, the stated For optical observation input, F is the state transition, H is the observation matrix, and the... This is the filter gain; The total projection error function reflects the deviation between the predicted position and the actual observed position. The optimization constraint of the total projection error function is as follows: ; Ui is the two-dimensional observation coordinate, R is the rotation matrix, and... The coordinates of the encoding points in the two-bit encoding structure are 3D coordinates, and T is the translation matrix.

[0021] S30. A cross-modal recognition surface marker system is deployed on the patient's body surface. This system exhibits stable grayscale contrast characteristics during tomographic image modeling and still has significant recognizability under the intraoperative optical perception environment. Through the surface marker, unified spatial registration between preoperative three-dimensional image data and the actual intraoperative anatomical scene can be achieved, and a mapping relationship between multi-domain data can be established. The real-time position information of the surface marker is captured by an optical vision device, and a cross-modal unified coordinate mapping relationship is established between the real-time position information of the surface marker and the spatial pose of the interventional device, specifically as follows: Let the position of the surface landmark in the preoperative CT coordinate system be: ; Let the position of the surface marker in the coordinate system of the intraoperative optical vision device be: ; The registration process can then be abstracted as a rigid body transformation: ; The optimal mapping of surface markers in the preoperative CT coordinate system and the intraoperative optical vision device coordinate system can be obtained by minimizing the following registration error function: ; The The set of three-dimensional location points of the surface marker in the preoperative CT coordinate system, the The coordinates of a three-dimensional point in the preoperative CT coordinate system, the The set of real numbers in three-dimensional space, the The set of three-dimensional points in the coordinate system of the intraoperative optical vision device, the These are the coordinates of a three-dimensional point in the coordinate system of the intraoperative optical vision device.

[0022] S40. Further, a respiratory dynamics analysis method based on the spatial micro-displacement of the marker is proposed: using the periodic displacement trajectory of the surface marker under the drive of respiratory motion, after temporal filtering and phase reconstruction, the respiratory signal is extracted and a dynamic respiratory phase model is generated. Combined with this model, the system can determine the low motion range within the respiratory cycle and form a temporal safety window, which is presented in the augmented reality interactive interface as a prompt, thereby significantly improving the timing accuracy and overall safety of percutaneous interventional surgery. Using the displacement data of the real-time position information of the surface markers, a respiratory curve is extracted to determine the safe window within the respiratory cycle: S41. Extract the displacement sequence of the surface marker in the direction perpendicular to the patient's chest cavity. : ; The This refers to the displacement of the surface marker in the direction perpendicular to the thoracic cavity; S42. Bandpass filtering is applied to the displacement sequence of the marker body. The bandpass filtering consists of low-frequency motion filtering and high-frequency noise suppression to obtain the respiratory signal. : ; S43. Perform phase extraction on the respiratory signal and obtain the instantaneous phase using Hilbert transform or wavelet packet decomposition: ; H[·] represents the Hilbert transform. It is the instantaneous phase; S44. Based on the derivative information and instantaneous phase of the respiratory signal, determine the stable phase in the respiratory cycle and generate a safety window to guide the timing of puncture. Specifically, determining the stable phase in the respiratory cycle involves: If satisfied If this interval is determined to be a stable respiratory phase, a safety window is generated and displayed in green; If not satisfied If this interval is not determined to be a stable respiratory phase, a prompt window will be generated, displaying a red color, indicating that the doctor is not allowed to perform surgical procedures at this time.

[0023] S50. In the augmented reality display device, the three-dimensional anatomical model, the trajectory and timing information of the interventional instruments, as well as the safety window and the prompt window are provided in a multi-view manner.

[0024] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0025] Example 2 This embodiment provides an augmented reality-guided navigation device based on cross-modal identification and temporal phase, such as... Figure 2 As shown, it includes the following modules: Preoperative data acquisition module: Several surface markers are placed on the patient's body surface at a set location, preoperative CT image data of the surface markers are acquired, and a three-dimensional anatomical model is constructed based on the preoperative CT image data; Intraoperative instrument data acquisition module: Tracks interventional instruments and obtains their spatial pose. The intraoperative instrument data acquisition module includes the following units: The device identification unit introduces a two-dimensional coding structure on the surface of the interventional device; The coordinate acquisition unit captures the two-dimensional projected coordinates of the two-dimensional encoded structure through an optical vision device; The pose unit is used to solve the spatial pose of the interventional device through the two-dimensional projected coordinates. The temporal filtering unit applies a temporal filtering algorithm to the pose sequence of the spatial pose to compensate for motion interference; Cross-modal coordinate mapping module: The three-dimensional position of the surface marker in the intraoperative optical camera coordinate system is captured by the optical vision device. A cross-modal unified coordinate mapping relationship is established between the three-dimensional position of the surface marker in the intraoperative optical camera coordinate system and the spatial pose data of the interventional instrument. Through this mapping relationship, the spatial pose data of the interventional instrument is transformed into the coordinate system of the preoperative three-dimensional anatomical model, thereby realizing the spatial registration of the two in a unified coordinate system. The temporal phase analysis module extracts the respiratory curve using displacement data from the real-time position information of the surface markers and determines the safe window within the respiratory cycle. Specifically, it includes the following units: The sequence extraction unit extracts the displacement sequence of the surface markers in a direction perpendicular to the patient's thoracic cavity; The respiratory signal unit performs bandpass filtering on the marker displacement sequence to obtain a respiratory signal; The phase extraction unit extracts the phase of the respiratory signal to obtain the instantaneous phase; The determination window unit determines the stable phase in the respiratory cycle based on the derivative information and instantaneous phase of the respiratory signal, and generates a safety window to guide the timing of puncture. Human-computer interaction and prompting module: In the augmented reality display device, the three-dimensional anatomical model, the trajectory and timing prompts of the interventional instruments, as well as the safety window and prompt window are provided in a multi-view manner.

[0026] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0027] Example 3 Based on the same inventive concept, this application also provides a system corresponding to the method in Embodiment 1, including the following modules: 1. Data Acquisition Module: Acquires preoperative tomographic images and intraoperative optical visual flow as the data foundation for cross-domain modeling and real-time navigation.

[0028] 2. Coordinate Unification Module: Based on surface markers, a unified coordinate system is constructed for patients, image models, and intraoperative scenes to achieve consistent association across multiple spatial domains.

[0029] 3. Virtual-Real Fusion Module: Dynamically fuses the 3D anatomical model with the real-time visual flow, and overlays operation paths and respiratory phase prompts.

[0030] 4. Respiratory Gating Module: Extracts respiratory signals through the periodic displacement of the marker, determines the low-motion zone within the respiratory cycle through phase decoupling and dynamic reconstruction, and generates a puncture safety window; The respiratory gating module also includes a respiratory signal extraction unit, a safety window determination unit, and a synchronization prompting unit, which are used to determine the optimal puncture time during the respiratory cycle and prompt the doctor.

[0031] 5. User Interaction Module: Presents target points, instrument trajectories, and timing prompts on augmented reality devices from multiple perspectives to assist operators in real-time operation and correction; the user interaction module supports multi-view observation and overlays target point positions, needle trajectories, and respiratory gating prompts on the augmented reality interface.

[0032] The system implementation process is as follows: S10. Affix dual-mode markers to the patient's body surface and acquire preoperative CT images to delineate the target point and puncture path, and reconstruct a three-dimensional anatomical model. The dual-mode markers serve as position-aware markers, and their material is radiopaque under CT scanning. The information of the marking codes attached to them can be simultaneously recognized by CT imaging and visible light cameras. S20. Tracking of interventional devices is achieved through the clamps of the interventional devices, which are marked with a two-digit code. S30. During the procedure, the position information of dual-mode markers and interventional instruments is captured in real time using an industrial camera to construct a unified coordinate system and perform virtual-real fusion. S40. Use dual-mode marker displacement to extract respiratory curves, determine the safe window within the respiratory cycle, and guide doctors to perform puncture at the optimal time. The S50 displays the target point, needle trajectory, and respiratory gating prompts intuitively on the AR headset or screen, allowing doctors to observe and correct operations in real time through a multi-view interface.

[0033] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0034] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0035] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0036] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0037] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An augmented reality-guided navigation method based on cross-modal identification and temporal phase, characterized in that, Includes the following steps: S10. Place several surface markers at predetermined locations on the patient's body surface, collect the three-dimensional position of the surface markers in the preoperative CT coordinate system, and construct a preoperative three-dimensional anatomical model based on the three-dimensional position in the preoperative CT coordinate system. S20. Track the interventional device to obtain its spatial pose: S21. Introduce a two-dimensional coding structure on the surface of interventional devices; S22. Capture the two-dimensional projection coordinates of the two-dimensional coding structure using an optical vision device; S23. Solve the spatial pose of the interventional device using the two-dimensional projection coordinates; S24. Apply a temporal filtering algorithm to the spatial pose data and output the spatial pose data. S30. The three-dimensional position of the surface marker in the intraoperative optical camera coordinate system is captured by the optical vision device. A cross-modal unified coordinate mapping relationship is established between the three-dimensional position of the surface marker in the intraoperative optical camera coordinate system and the spatial pose data of the interventional instrument. The spatial pose data of the interventional instrument is converted to the coordinate system of the preoperative three-dimensional anatomical model through the mapping relationship, thereby realizing the spatial registration of the two in the unified coordinate system. S40. Based on the displacement data of the three-dimensional position of the surface marker captured in S30 under a unified coordinate system, extract the respiratory curve and determine the stable phase within the respiratory cycle: S41. Extract the displacement sequence of the surface marker in the direction perpendicular to the patient's chest cavity; S42. Bandpass filter is applied to the displacement sequence of the marker body to obtain the respiratory signal; S43. Perform phase extraction on the respiratory signal to obtain the instantaneous phase; S44. By using the derivative information and instantaneous phase of the respiratory signal, determine the stable phase in the respiratory cycle to guide the doctor on the timing of puncture.

2. The augmented reality-guided navigation method based on cross-modal identification and temporal phase as described in claim 1, characterized in that, S20 involves tracking the interventional device to obtain its spatial pose, specifically as follows: S21. A two-dimensional coding structure is introduced on the surface of the interventional device, wherein the two-dimensional coding structure has a highly redundant feature point distribution; let the three-dimensional coordinate set of the coding points of the two-dimensional coding structure be: ; M is the three-dimensional coordinate set of the coding points of the two-bit coding structure. The three-dimensional coordinates of the coding point in the two-bit coding structure, the It is a set of three-dimensional real numbers; S22. The two-dimensional projection coordinates of the two-dimensional coding structure on the imaging plane are captured in real time using an optical vision device. The two-dimensional projection coordinates are: ; The Let i be the two-dimensional coordinates of the i-th encoded point projected onto the image plane. Represents a perspective projection model, the Describe the orientation of the interventional instrument relative to the coordinate system of the optical vision device. Describe the position of the interventional device in the camera coordinate system; S23. Establish the geometric constraint relationship between the three-dimensional point set and the two-dimensional projection through the camera imaging model, and solve for the spatial pose of the interventional device: Minimize the projection error function: ; E(R,T) is the projection error function, and (R,T) is the spatial pose of the interventional device. The projection error describes the squared error of a single feature point, which can be used to achieve the optimal estimation of the spatial pose (R,T) of the interventional device. When a portion of the two-bit coding structure of the interventional device is obscured or the illumination changes, the RANSAC algorithm is introduced. S24. Apply a temporal filtering algorithm to the pose sequence of the spatial pose to compensate for motion interference: The filtering algorithm is as follows: ; The This represents the pose estimation state at time t, the stated For optical observation input, F is the state transition, H is the observation matrix, and the... This is the filter gain; The optimization constraint for the total projection error function is: ; The Let R be the two-dimensional observation coordinates, and let R be the rotation matrix. The coordinates of the encoding point are 3D coordinates, and T is the translation matrix.

3. The augmented reality-guided navigation method based on cross-modal identification and temporal phase as described in claim 1, characterized in that, S30 involves capturing the three-dimensional position of the surface marker in the intraoperative optical camera coordinate system using an optical vision device, establishing a cross-modal unified coordinate mapping relationship between the three-dimensional position of the surface marker in the intraoperative optical camera coordinate system and the spatial pose data of the interventional instrument, and converting the spatial pose data of the interventional instrument to the coordinate system of the preoperative three-dimensional anatomical model through this mapping relationship, thereby achieving spatial registration of the two in a unified coordinate system; specifically: Let the position of the surface marker in the coordinate system of the preoperative three-dimensional anatomical model be: ; Let the position of the surface marker in the coordinate system of the intraoperative optical vision device be: ; The registration process can then be abstracted as a rigid body transformation: ; The optimal mapping of surface markers in the preoperative three-dimensional anatomical model coordinate system and the intraoperative optical vision device coordinate system can be obtained by minimizing the following registration error function: ; The The set of three-dimensional position points of the surface marker in the preoperative three-dimensional anatomical model coordinate system, the The coordinates of a three-dimensional point in the coordinate system of the preceding three-dimensional anatomical model, the The set of real numbers in three-dimensional space, the The set of three-dimensional points in the coordinate system of the intraoperative optical vision device, the These are the coordinates of a three-dimensional point in the coordinate system of the intraoperative optical vision device.

4. The augmented reality-guided navigation method based on cross-modal identification and temporal phase as described in claim 1, characterized in that, In step S40, based on the displacement data of the three-dimensional position of the surface marker captured in step S30 under a unified coordinate system, a respiratory curve is extracted to determine the stable phase within the respiratory cycle. Specifically: S41. Extract the displacement sequence of the surface marker in the direction perpendicular to the patient's chest cavity. : ; The This refers to the displacement of the surface marker in the direction perpendicular to the thoracic cavity; S42. Bandpass filtering is applied to the displacement sequence of the marker body. The bandpass filtering consists of low-frequency motion filtering and high-frequency noise suppression to obtain the respiratory signal. : ; S43. Perform phase extraction on the respiratory signal and obtain the instantaneous phase using Hilbert transform or wavelet packet decomposition: ; H[·] represents the Hilbert transform. It is the instantaneous phase; S44. Based on the derivative information and instantaneous phase of the respiratory signal, determine the stable phase in the respiratory cycle and generate a safety window to guide the timing of puncture. Specifically, determining the stable phase in the respiratory cycle involves: If satisfied If so, this interval is determined to be a stable respiratory phase; If not satisfied If so, this interval is determined to be a non-breathing stable phase.

5. The augmented reality-guided navigation method based on cross-modal identification and temporal phase according to claim 4, characterized in that, It also includes the following steps: S50. In the augmented reality display device, the three-dimensional anatomical model, the trajectory and timing prompts of the interventional instruments, as well as the safety window and prompt window are provided in a multi-view manner. The safety window is defined as follows: when the respiratory phase is determined to be stable, a safety window pops up in the augmented reality display device, indicating that the doctor has permission to perform the surgical procedure at this moment; The prompt window is a pop-up window that appears on the augmented reality display device when the patient is determined to be in a non-respiratory phase, prompting the doctor that surgical procedures are not allowed at this time.

6. An augmented reality-guided navigation device based on cross-modal identification and temporal phase, characterized in that, Includes the following modules: Preoperative data acquisition module: Several surface markers are placed at predetermined locations on the patient's body surface, and the three-dimensional positions of the surface markers in the preoperative CT coordinate system are acquired. Based on the three-dimensional positions in the preoperative CT coordinate system, a preoperative three-dimensional anatomical model is constructed. Intraoperative instrument data acquisition module: Tracks interventional instruments and obtains their spatial pose. The intraoperative instrument data acquisition module includes the following units: The device identification unit introduces a two-dimensional coding structure on the surface of the interventional device; The coordinate acquisition unit captures the two-dimensional projected coordinates of the two-dimensional encoded structure through an optical vision device; The pose unit is used to solve the spatial pose of the interventional device through the two-dimensional projected coordinates. The temporal filtering unit applies a temporal filtering algorithm to the pose sequence of the spatial pose and outputs spatial pose data. Cross-modal coordinate mapping module: The three-dimensional position of the surface marker in the intraoperative optical camera coordinate system is captured by the optical vision device. A cross-modal unified coordinate mapping relationship is established between the three-dimensional position of the surface marker in the intraoperative optical camera coordinate system and the spatial pose data of the interventional instrument. Through this mapping relationship, the spatial pose data of the interventional instrument is transformed into the coordinate system of the preoperative three-dimensional anatomical model, thereby realizing the spatial registration of the two in a unified coordinate system. The temporal phase analysis module extracts the respiratory curve using displacement data from the real-time position information of the surface markers and determines the stable phase within the respiratory cycle. Specifically, it includes the following units: The sequence extraction unit extracts the displacement sequence of the surface markers in a direction perpendicular to the patient's thoracic cavity; The respiratory signal unit performs bandpass filtering on the marker displacement sequence to obtain a respiratory signal; The phase extraction unit extracts the phase of the respiratory signal to obtain the instantaneous phase; The determination window unit determines the stable phase in the respiratory cycle by using the derivative information and instantaneous phase of the respiratory signal, thus guiding the doctor on the timing of puncture.

7. The augmented reality-guided navigation device based on cross-modal identification and temporal phase according to claim 6, characterized in that, The intraoperative instrument data acquisition module specifically comprises: The device identification unit introduces a two-dimensional coding structure on the surface of the interventional device. This two-dimensional coding structure has a highly redundant feature point distribution. Let the three-dimensional coordinate set of the coding points of the two-dimensional coding structure be: ; M is the three-dimensional coordinate set of the coding points of the two-bit coding structure. The three-dimensional coordinates of the coding points in the two-bit coding structure are given. It is a set of three-dimensional real numbers; The coordinate acquisition unit captures the two-dimensional projection coordinates of the two-dimensional coded structure on the imaging plane in real time through an optical vision device. The two-dimensional projection coordinates are: ; The Let i be the two-dimensional coordinates of the i-th encoded point projected onto the image plane. Represents a perspective projection model, the Describe the orientation of the interventional instrument relative to the coordinate system of the optical vision device. Describe the position of the interventional device in the camera coordinate system; The pose unit is solved by establishing the geometric constraint relationship between the 3D point set and the 2D projection through the camera imaging model, and then solving for the spatial pose of the interventional device. Minimize the projection error function: ; E(R,T) is the projection error function, and (R,T) is the spatial pose of the interventional device. The projection error describes the squared error of a single feature point, which can be used to achieve the optimal estimation of the spatial pose (R,T) of the interventional device. When a portion of the two-bit coding structure of the interventional device is obscured or the illumination changes, the RANSAC algorithm is introduced. The temporal filtering unit applies a temporal filtering algorithm to the pose sequence of the spatial pose to compensate for motion interference: The filtering algorithm is as follows: ; The This represents the pose estimation state at time t, the stated For optical observation input, F is the state transition, H is the observation matrix, and the... This is the filter gain; The optimization constraint for the total projection error function is: ; The Let R be the two-dimensional observation coordinates, and let R be the rotation matrix. Let T be the three-dimensional coordinates of the encoding point of the two-bit encoding structure, and let T be the translation matrix.

8. The augmented reality-guided navigation device based on cross-modal identification and temporal phase according to claim 6, characterized in that, The cross-modal coordinate mapping module is specifically as follows: Let the position of the surface landmark in the preoperative CT coordinate system be: ; Let the position of the surface marker in the coordinate system of the intraoperative optical vision device be: ; The registration process can then be abstracted as a rigid body transformation: ; The optimal mapping of surface markers in the preoperative CT coordinate system and the intraoperative optical vision device coordinate system can be obtained by minimizing the following registration error function: ; The The set of three-dimensional location points of the surface marker in the preoperative CT coordinate system, the The coordinates of a three-dimensional point in the preoperative CT coordinate system, the The set of real numbers in three-dimensional space, the The set of three-dimensional points in the coordinate system of the intraoperative optical vision device, the These are the coordinates of a three-dimensional point in the coordinate system of the intraoperative optical vision device.

9. An augmented reality-guided navigation device based on cross-modal identification and temporal phase according to claim 6, characterized in that, The time-series phase analysis module: extracts the respiratory curve using displacement data from the real-time position information of the surface markers, and determines the stable phase within the respiratory cycle, specifically: The sequence extraction unit extracts the displacement sequence of the surface markers in a direction perpendicular to the patient's thoracic cavity. : ; The This refers to the displacement of the surface marker in the direction perpendicular to the thoracic cavity; The respiratory signal unit performs bandpass filtering on the displacement sequence of the marker body. The bandpass filtering consists of low-frequency motion filtering and high-frequency noise suppression to obtain the respiratory signal. : ; The phase extraction unit extracts the phase of the respiratory signal and obtains the instantaneous phase using Hilbert transform or wavelet packet decomposition. ; H[·] represents the Hilbert transform. It is the instantaneous phase; The determination window unit, through the derivative information and instantaneous phase of the respiratory signal, determines the stable phase in the respiratory cycle and generates a safety window to guide the timing of puncture. Specifically, determining the stable phase in the respiratory cycle involves: If satisfied If so, this interval is determined to be a stable respiratory phase; If not satisfied If so, this interval is determined to be a non-breathing stable phase.

10. An augmented reality-guided navigation device based on cross-modal identification and temporal phase according to claim 9, characterized in that, It also includes a human-computer interaction and prompting module: in the augmented reality display device, it provides the three-dimensional anatomical model, the trajectory and timing prompts of the interventional instruments, as well as safety windows and prompt windows in a multi-view manner; The safety window is defined as follows: when the respiratory phase is determined to be stable, a safety window pops up in the augmented reality display device, indicating that the doctor has permission to perform the surgical procedure at this moment; The prompt window is a pop-up window that appears on the augmented reality display device when the patient is determined to be in a non-respiratory phase, prompting the doctor that surgical procedures are not allowed at this time.

Images