Accuracy correction method for augmented reality system with three-dimensional mark field and one-surface and one-point positioning

Abstract

The invention provides an augmented reality system precision correction method based on three-dimensional marking field and one-side one-point positioning, which is used for preoperative closed-loop convergence of CT image-patient registration errors and instrument pose errors, and adopts an augmented reality system precision closed-loop correction system based on three-dimensional marking field and one-side one-point positioning. Comprising a data acquisition module, a coordinate unification module, a virtual-real fusion module, a closed-loop correction module and a user interaction module which are connected through a data bus, and the correction method comprises the following steps: firstly, establishing a unified coordinate system by using a three-dimensional marking field, and then forming six-degree-of-freedom full constraint of a rigid body through a one-side and one-point positioning strategy; a registration error is monitored and fed back in real time through a closed-loop correction mechanism, and surgical navigation information is presented on an augmented reality interface; through establishment of a unified coordinate system, spatial registration, closed-loop error correction and multi-view visual interaction, high precision, real-time performance and visualization of the whole process of surgical navigation can be realized, and the safety and precision of interventional surgery are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of medical navigation and augmented reality (AR) technology, and in particular to a method for accuracy correction of augmented reality systems using three-dimensional marked fields and one-point localization on one surface. Background Technology

[0002] In existing technologies, the coordinate registration between virtual models and real patient scenarios is mostly an open-loop process, and errors cannot be corrected in real time before and during surgery, resulting in insufficient surgical navigation accuracy and potential safety risks.

[0003] In existing percutaneous interventional surgical navigation methods, the registration accuracy between CT images and patient position is insufficient, and the instrument position error cannot be corrected in a closed loop before surgery. This leads to deviations between the navigation information and the actual anatomical structure during surgery, affecting puncture accuracy and surgical safety.

[0004] This invention aims to solve the problem that in existing percutaneous interventional surgical navigation, the registration errors between CT images and patient positioning, as well as the errors in instrument pose, cannot be closed-loop converged before surgery.

[0005] This invention proposes an augmented reality system accuracy closed-loop correction method based on three-dimensional marked field and one-point localization on one surface. By establishing a unified coordinate system, spatial registration, closed-loop error correction, and multi-view visualization interaction, it achieves high precision, real-time operation, and visualization of the entire surgical navigation process, significantly improving the safety and accuracy of interventional surgery. Summary of the Invention

[0006] This invention proposes an accuracy correction method for augmented reality systems based on three-dimensional labeled fields and one-face-point localization. This method is a closed-loop accuracy correction method for augmented reality systems based on three-dimensional labeled fields and one-face-point localization, which can solve the problem that the registration error between CT images and patients and the instrument pose error in existing interventional surgical navigation cannot be closed-loop converged before surgery. This invention uses a three-dimensional labeled field to establish a unified coordinate system, and combines the one-face-point localization strategy to achieve full constraint of the six degrees of freedom of rigid body. Through the closed-loop correction mechanism, the registration error is monitored and fed back in real time, and finally the surgical navigation information is presented intuitively in the augmented reality interface.

[0007] The present invention adopts the following technical solution.

[0008] A method for accuracy correction in augmented reality systems using a three-dimensional marker field and one-face-one-point localization is proposed. This method is used for preoperative closed-loop convergence of CT image-patient registration errors and instrument pose errors in interventional surgical navigation. It includes a data acquisition module, a coordinate unification module, a virtual-real fusion module, a closed-loop correction module, and a user interaction module connected via a data bus. First, a unified coordinate system is established using a three-dimensional marker field. Then, a rigid body with six degrees of freedom is fully constrained through a one-face-one-point localization strategy. Finally, the registration error is monitored and fed back in real time through a closed-loop correction mechanism, and the surgical navigation information is presented on the augmented reality interface.

[0009] In the method, the data acquisition module is responsible for acquiring preoperative CT image data and intraoperative real-time images from an industrial camera; the CT image data is used to reconstruct or generate a three-dimensional anatomical model of the patient, and the real-time images from the industrial camera are used to capture the spatial position information of intraoperative markers and surgical instruments in real time, providing a data basis for coordinate unification and spatial registration;

[0010] The coordinate unification module is based on a three-dimensional labeled field and establishes a unified spatial coordinate system among patients, CT images, surgical instruments, markers and cameras to ensure that multi-source data are processed in the same coordinate system and avoid cumulative errors caused by coordinate inconsistencies.

[0011] The virtual-real fusion module is used to achieve precise spatial registration between the CT reconstruction model and the real patient scene during surgery, ensuring that the virtual model and the actual anatomical structure of the patient remain highly consistent in space.

[0012] The closed-loop correction module adopts a single-face single-point positioning strategy, that is, a one-face-one-point positioning strategy, to detect registration error and instrument pose deviation in real time, and to achieve closed-loop convergence of error through iterative feedback, ensuring that the navigation accuracy throughout the operation meets the sub-millimeter level requirements.

[0013] The user interaction module includes a multi-view augmented reality interface that displays surgical navigation information, registration status, and deviation prompts in real time, providing surgeons with intuitive operational feedback and decision-making support.

[0014] The markers are three-dimensional markers placed on the patient's body surface and surgical instruments. In the method, the coordinate unification module uses a calibration plate to calibrate the camera's intrinsic and extrinsic parameters, obtains the transformation matrix from the camera coordinate system to the world coordinate system, and constructs a unified three-dimensional coordinate system to ensure that all data are calculated in the same coordinate system and reduce the cumulative error caused by multiple coordinate transformations.

[0015] The coordinate unification module establishes a unified spatial coordinate system among the patient, CT images, surgical instruments, markers, and camera using the following method: It establishes five coordinate systems: patient {pat}, CT {CT}, surgical instruments {Nee}, markers {Mar}, and camera {Cam}, and achieves unification through the following transformation matrix:

[0016]

[0017] in, Used to map patient coordinates to camera coordinates Used to align preoperative CT scans with the patient's position during surgery. Define the spatial relationship between the needle tip and the calibration object. This enables real-time positioning.

[0018] Three-dimensional markers such as Figure 3 As shown, the three-dimensional markers of the surgical instruments are affixed to the rigid calibrator connected to the puncture guide needle handle. The placement principles are as follows: avoid the operating part at the front end of the instrument to ensure that the puncture accuracy and surgical area disinfection are not affected; select parts of the instrument with rigid and planar features to ensure that the markers and instruments maintain a fixed spatial rigid body relationship; and maintain an unobstructed field of view relationship between the markers and the industrial camera for real-time tracking.

[0019] The three-dimensional markers attached to the patient's body surface are mainly placed in the planar area of ​​the patient's chest and abdomen;

[0020] The reasons for choosing this area are as follows:

[0021] 1. The abdominal region is usually the main surgical operating surface in percutaneous interventional procedures (such as liver puncture, lung puncture, etc.), providing a stable and visible plane; 2. The abdominal skin surface is relatively flat, which facilitates the application of planar markers and camera recognition; 3. This region is relatively free of bony obstructions, which helps to improve spatial coordinate stability.

[0022] The method for fixing the marker is as follows: a PVC waterproof label made of medical-grade non-reflective flexible material is used and fixed to the skin surface with medical double-sided tape or hypoallergenic dressing to ensure stable adhesion and non-invasiveness during the operation;

[0023] The markers are made using 3D printing, and the marker materials do not use metal objects.

[0024] When the data acquisition module is working, CT image data is imported in the DICOM standard format and a 3D slicer three-dimensional reconstruction algorithm is used to generate a patient anatomical model.

[0025] The three-dimensional coordinates of the markers are obtained from the images of the markers captured by industrial cameras using a computer vision method based on feature point detection.

[0026] When the virtual-real fusion module is working, it first uses the three-dimensional coordinates of the markers to determine the patient's reference coordinate system. Then, it calculates the optimal rigid body transformation matrix from the CT model to the patient's coordinate system using a rigid body registration algorithm (such as SVD singular value decomposition algorithm), completing the initial registration between the virtual model and the real anatomical structure. After registration, the resulting virtual model can maintain high-precision consistency with the patient's surface and internal anatomical structure in the augmented reality interface; specifically:

[0027] Coordinate transformation employs a point cloud registration method based on Singular Value Decomposition (SVD): a corresponding point set {p} is extracted between the CT model and patient markers. i},{q i}, by minimizing

[0028]

[0029] The rotation matrix R and translation vector t are obtained, and finally the rigid body transformation matrix [R|t] is obtained, realizing the unification of CT model and patient anatomical coordinates;

[0030] The intrinsic and extrinsic parameters of the industrial camera are calibrated using a calibration board, and a transformation matrix T is established from the camera coordinate system to the world coordinate system. CW The coordinates of markers, puncture site, and target point are read from the CT model. Combined with the coordinates of marker points on the patient's body surface, a transformation matrix T is constructed from CT coordinates to patient coordinates. CP Ultimately, the coordinate relationships of all objects in the system are as follows:

[0031] X W =T CW ·X C ,X P =T CP ·X CT

[0032] Where X C X represents the point in the camera coordinate system. CT This represents the lower point of the CT model; through this unified coordinate system, the virtual CT model and the real patient's position remain highly consistent in the AR interface.

[0033] The closed-loop correction module uses a one-point positioning strategy to monitor and correct the registration accuracy in real time. Planar markers are selected in the three-dimensional marker field to constrain multiple degrees of freedom of translation and rotation, and single-point markers are selected on the plane to constrain the remaining degrees of freedom, so as to achieve full constraint of the six degrees of freedom of the rigid body.

[0034] "One side" corresponds to the upper surface of the marker, and "one point" is the intersection of two straight lines on the side of the rigid body;

[0035] The specific method is as follows: First, select the planar marker pasted on the patient's abdomen in the three-dimensional marker field and constrain the five degrees of freedom of translation and rotation in the three-dimensional space; then select a single point marker on the plane and constrain the remaining one degree of freedom to achieve full constraint positioning of six degrees of freedom.

[0036] The one-point positioning strategy includes:

[0037] To avoid the accumulation of errors in traditional open-loop registration, a "one-face-one-point" positioning strategy is introduced, specifically: the planar marker provides translation (t) x ,t y ) and rotation (θ) x ,θ y ,θ z The intersection markers provide depth (t) constraints. z Constraints, combined with other constraints, achieve full constraint of six degrees of freedom:

[0038] C′marker ∪C′ point ={t x ,t y ,t z ,θ x ,θ y ,θ z}

[0039] This allows for the construction of a closed-loop feedback loop, enabling self-calibration of accuracy before and during surgery.

[0040] The user interaction module uses augmented reality display devices to overlay surgical navigation information, registration status, and deviation prompts onto the patient's real-world scene in real time, allowing the surgeon to intuitively understand the registration accuracy and instrument position. In addition, the user interaction module supports simultaneous multi-view observation, allowing the surgeon to freely choose between frontal, top-down, or perspective modes to comprehensively observe the surgical area and the real-time position of the instruments and the patient from multiple angles, reducing cognitive burden.

[0041] The method includes the following steps;

[0042] Step 1: In the preoperative stage, data collection is carried out, specifically: acquiring the patient's preoperative CT image data, and capturing the spatial position information of intraoperative markers and surgical instruments in real time using an industrial camera;

[0043] Step 2: Coordinate unification is performed in the preoperative stage, specifically: based on the three-dimensional marker field, a unified spatial coordinate system is established among the patient, CT images, surgical instruments, markers and cameras;

[0044] Step 3: Perform virtual-real fusion in the preoperative stage, specifically: spatially register the CT 3D model with the patient's actual intraoperative scene under a unified coordinate system;

[0045] Step 4: During the intraoperative phase, the spatial position of instruments and markers is captured in real time and closed-loop correction is performed. Specifically, a face-to-point positioning strategy is used to monitor the registration accuracy, and closed-loop correction of errors is achieved through iterative feedback.

[0046] Step 5: During the intraoperative phase, user interaction and visualization are performed, specifically by displaying surgical navigation information, registration status, and deviation prompts in real time on the augmented reality interface.

[0047] The method for percutaneous interventional surgery navigation also includes: verifying the accuracy of the augmented reality system's guidance based on CT results during the postoperative stage.

[0048] This invention solves the problem that the registration errors between CT images and patient position, as well as the instrument pose errors, cannot be closed and converged before surgery in existing percutaneous interventional surgical navigation. By establishing a unified coordinate system, spatial registration, closed-loop error correction, and multi-view visualization interaction, this invention achieves high precision, real-time operation, and visualization of the entire surgical navigation process, significantly improving the safety and accuracy of interventional surgery.

[0049] This invention comprises five functional modules: a data acquisition module, a coordinate unification module, a virtual-real fusion module, a closed-loop correction module, and a user interaction module. These modules work collaboratively to establish a unified coordinate system using a three-dimensional marker field. Combined with a face-point positioning strategy, it achieves full constraint of the six degrees of freedom of a rigid body. Through a closed-loop correction mechanism, it monitors and provides feedback on registration errors in real time. Finally, it presents surgical navigation information intuitively in an augmented reality interface, ensuring that the system can provide high-precision and visualized navigation information both before and during surgery. Attached Figure Description

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0051] Appendix Figure 1 This is a schematic diagram of the structure of an augmented reality system accuracy closed-loop correction method based on three-dimensional marker field and one-point localization on one surface, according to an embodiment of the present invention.

[0052] Appendix Figure 2 A schematic diagram of a system architecture for establishing a unified spatial coordinate system among the patient, CT images, surgical instruments, markers, and camera, according to an embodiment of the present invention;

[0053] Appendix Figure 3 This is a schematic diagram of the full constraint of a rigid body with six degrees of freedom, achieved by combining a one-sided-one-point positioning strategy according to an embodiment of the present invention (one side refers to the upper surface of the marker above the marker, and one point refers to the intersection point obtained by the intersection of the side of the rigid body through a straight line (wire));

[0054] Appendix Figure 4 This is a schematic diagram of a multi-view augmented reality interface according to an embodiment of the present invention (the left side is a superimposed image of virtual and real objects from the camera's perspective, and the right side is a model and equipment model in the virtual twin space). Detailed Implementation

[0055] As shown in the figure, an augmented reality system accuracy correction method based on a three-dimensional marker field and one-face-one-point localization is described. This method is used for preoperative closed-loop convergence of CT image-patient registration errors and instrument pose errors in interventional surgical navigation. It includes a data acquisition module, a coordinate unification module, a virtual-real fusion module, a closed-loop correction module, and a user interaction module connected via a data bus. First, a unified coordinate system is established using a three-dimensional marker field. Then, a rigid body with six degrees of freedom is fully constrained through a one-face-one-point localization strategy. Finally, the registration error is monitored and fed back in real time through a closed-loop correction mechanism, and the surgical navigation information is presented on the augmented reality interface.

[0056] In the method, the data acquisition module is responsible for acquiring preoperative CT image data and intraoperative real-time images from an industrial camera; the CT image data is used to reconstruct or generate a three-dimensional anatomical model of the patient, and the real-time images from the industrial camera are used to capture the spatial position information of intraoperative markers and surgical instruments in real time, providing a data basis for coordinate unification and spatial registration;

[0057] The coordinate unification module is based on a three-dimensional labeled field and establishes a unified spatial coordinate system among patients, CT images, surgical instruments, markers and cameras to ensure that multi-source data are processed in the same coordinate system and avoid cumulative errors caused by coordinate inconsistencies.

[0058] The virtual-real fusion module is used to achieve precise spatial registration between the CT reconstruction model and the real patient scene during surgery, ensuring that the virtual model and the actual anatomical structure of the patient remain highly consistent in space.

[0059] The closed-loop correction module adopts a single-face single-point positioning strategy, that is, a one-face-one-point positioning strategy, to detect registration error and instrument pose deviation in real time, and to achieve closed-loop convergence of error through iterative feedback, ensuring that the navigation accuracy throughout the operation meets the sub-millimeter level requirements.

[0060] The user interaction module includes a multi-view augmented reality interface that displays surgical navigation information, registration status, and deviation prompts in real time, providing surgeons with intuitive operational feedback and decision-making support.

[0061] The markers are three-dimensional markers placed on the patient's body surface and surgical instruments. In the method, the coordinate unification module uses a calibration plate to calibrate the camera's intrinsic and extrinsic parameters, obtains the transformation matrix from the camera coordinate system to the world coordinate system, and constructs a unified three-dimensional coordinate system to ensure that all data are calculated in the same coordinate system and reduce the cumulative error caused by multiple coordinate transformations.

[0062] The coordinate unification module establishes a unified spatial coordinate system among the patient, CT images, surgical instruments, markers, and camera using the following method: It establishes five coordinate systems: patient {Pat}, CT {CT}, surgical instruments {Nee}, markers {Mar}, and camera {Cam}, and achieves unification through the following transformation matrix:

[0063]

[0064] in, Used to map patient coordinates to camera coordinates Used to align preoperative CT scans with the patient's position during surgery. Define the spatial relationship between the needle tip and the calibration object. This enables real-time positioning.

[0065] Three-dimensional markers such as Figure 3 As shown, the three-dimensional markers of the surgical instruments are affixed to the rigid calibrator connected to the puncture guide needle handle. The placement principles are as follows: avoid the operating part at the front end of the instrument to ensure that the puncture accuracy and surgical area disinfection are not affected; select parts of the instrument with rigid and planar features to ensure that the markers and instruments maintain a fixed spatial rigid body relationship; and maintain an unobstructed field of view relationship between the markers and the industrial camera for real-time tracking.

[0066] The three-dimensional markers attached to the patient's body surface are mainly placed in the planar area of ​​the patient's chest and abdomen;

[0067] The reasons for choosing this area are as follows:

[0068] 1. The abdominal region is usually the main surgical operating surface in percutaneous interventional procedures (such as liver puncture, lung puncture, etc.), providing a stable and visible plane; 2. The abdominal skin surface is relatively flat, which facilitates the application of planar markers and camera recognition; 3. This region is relatively free of bony obstructions, which helps to improve spatial coordinate stability.

[0069] The method for fixing the marker is as follows: a PVC waterproof label made of medical-grade non-reflective flexible material is used and fixed to the skin surface with medical double-sided tape or hypoallergenic dressing to ensure stable adhesion and non-invasiveness during the operation;

[0070] The markers are made using 3D printing, and the marker materials do not use metal objects.

[0071] When the data acquisition module is working, CT image data is imported in the DICOM standard format and a 3D slicer three-dimensional reconstruction algorithm is used to generate a patient anatomical model.

[0072] The three-dimensional coordinates of the markers are obtained from the images of the markers captured by industrial cameras using a computer vision method based on feature point detection.

[0073] When the virtual-real fusion module is working, it first uses the three-dimensional coordinates of the markers to determine the patient's reference coordinate system. Then, it calculates the optimal rigid body transformation matrix from the CT model to the patient's coordinate system using a rigid body registration algorithm (such as SVD singular value decomposition algorithm), completing the initial registration between the virtual model and the real anatomical structure. After registration, the resulting virtual model can maintain high-precision consistency with the patient's surface and internal anatomical structure in the augmented reality interface; specifically:

[0074] Coordinate transformation employs a point cloud registration method based on Singular Value Decomposition (SVD): a corresponding point set {p} is extracted between the CT model and patient markers. i},{q i}, by minimizing

[0075]

[0076] The rotation matrix R and translation vector t are obtained, and finally the rigid body transformation matrix [R|t] is obtained, realizing the unification of CT model and patient anatomical coordinates;

[0077] The intrinsic and extrinsic parameters of the industrial camera are calibrated using a calibration board, and a transformation matrix T is established from the camera coordinate system to the world coordinate system. CW The coordinates of markers, puncture site, and target point are read from the CT model. Combined with the coordinates of marker points on the patient's body surface, a transformation matrix T is constructed from CT coordinates to patient coordinates. CP Ultimately, the coordinate relationships of all objects in the system are as follows:

[0078] X W =T CW ·X C ,X P =T CP ·X CT

[0079] Where X C X represents the point in the camera coordinate system. CT This represents the lower point of the CT model; through this unified coordinate system, the virtual CT model and the real patient's position remain highly consistent in the AR interface.

[0080] The closed-loop correction module uses a one-point positioning strategy to monitor and correct the registration accuracy in real time. Planar markers are selected in the three-dimensional marker field to constrain multiple degrees of freedom of translation and rotation, and single-point markers are selected on the plane to constrain the remaining degrees of freedom, so as to achieve full constraint of the six degrees of freedom of the rigid body.

[0081] "One side" corresponds to the upper surface of the marker, and "one point" is the intersection of two straight lines on the side of the rigid body;

[0082] The specific method is as follows: First, select the planar marker pasted on the patient's abdomen in the three-dimensional marker field and constrain the five degrees of freedom of translation and rotation in the three-dimensional space; then select a single point marker on the plane and constrain the remaining one degree of freedom to achieve full constraint positioning of six degrees of freedom.

[0083] The one-point positioning strategy includes:

[0084] To avoid the accumulation of errors in traditional open-loop registration, a "one-face-one-point" positioning strategy is introduced, specifically: the planar marker provides translation (t) x ,t y ) and rotation (θ) x ,θ y ,θ z The intersection markers provide depth (t) constraints. z Constraints, combined with other constraints, achieve full constraint of six degrees of freedom:

[0085] C′ marker ∪C′ point ={t x ,t y ,t z ,θ x ,θ y ,θ z}

[0086] This allows for the construction of a closed-loop feedback loop, enabling self-calibration of accuracy before and during surgery.

[0087] The user interaction module uses augmented reality display devices to overlay surgical navigation information, registration status, and deviation prompts onto the patient's real-world scene in real time, allowing the surgeon to intuitively understand the registration accuracy and instrument position. In addition, the user interaction module supports simultaneous multi-view observation, allowing the surgeon to freely choose between frontal, top-down, or perspective modes to comprehensively observe the surgical area and the real-time position of the instruments and the patient from multiple angles, reducing cognitive burden.

[0088] The method includes the following steps;

[0089] Step 1: In the preoperative stage, data collection is carried out, specifically: acquiring the patient's preoperative CT image data, and capturing the spatial position information of intraoperative markers and surgical instruments in real time using an industrial camera;

[0090] Step 2: Coordinate unification is performed in the preoperative stage, specifically: based on the three-dimensional marker field, a unified spatial coordinate system is established among the patient, CT images, surgical instruments, markers and cameras;

[0091] Step 3: Perform virtual-real fusion in the preoperative stage, specifically: spatially register the CT 3D model with the patient's actual intraoperative scene under a unified coordinate system;

[0092] Step 4: During the intraoperative phase, the spatial position of instruments and markers is captured in real time and closed-loop correction is performed. Specifically, a face-to-point positioning strategy is used to monitor the registration accuracy, and closed-loop correction of errors is achieved through iterative feedback.

[0093] Step 5: During the intraoperative phase, user interaction and visualization are performed, specifically by displaying surgical navigation information, registration status, and deviation prompts in real time on the augmented reality interface.

[0094] The method for percutaneous interventional surgery navigation also includes: verifying the accuracy of the augmented reality system's guidance based on CT results during the postoperative stage.

[0095] Example:

[0096] This paper presents a closed-loop accuracy correction method for augmented reality systems based on a three-dimensional marked field and one-face-one-point localization. Applicable to percutaneous interventional surgical navigation, it aims to address the problem of preoperative closed-loop convergence of CT image-patient registration errors and instrument pose errors in existing interventional surgical navigation systems. The method first establishes a three-dimensional marked field on the patient's body surface, constructing a unified coordinate system for CT images, the patient, surgical instruments, and the camera. Then, it combines a six-point localization method to constrain the six degrees of freedom of the rigid body and employs a "single-face-one-point" calibration strategy to form a closed-loop feedback, achieving high-precision registration between the virtual model and the real patient's position. This method can significantly improve the accuracy and stability of percutaneous interventional surgical navigation, reduce surgical risks, and has broad clinical application prospects and promotional value.

[0097] A closed-loop accuracy correction method for augmented reality systems based on 3D marked fields and point localization on a surface includes the following steps:

[0098] (1) Data acquisition: Acquire preoperative CT image data of patients and capture the spatial position information of intraoperative markers and surgical instruments in real time using an industrial camera;

[0099] (2) Coordinate unification: Based on the three-dimensional marker field, a unified spatial coordinate system is established among the patient, CT images, surgical instruments, markers and cameras;

[0100] (3) Virtual-real fusion: Under a unified coordinate system, the CT 3D model is spatially registered with the patient's real intraoperative scene;

[0101] (4) Closed-loop correction: The registration accuracy is monitored by a one-point positioning strategy and the error is corrected in a closed loop through iterative feedback.

[0102] (5) User interaction and visualization: Display surgical navigation information, registration status and deviation prompts in real time in the augmented reality interface.

[0103] The one-point positioning strategy includes:

[0104] In a three-dimensional marking field, planar markers are selected to constrain multiple degrees of freedom of translation and rotation, and single-point markers are selected on the same plane to constrain the remaining degrees of freedom, so as to achieve full constraint of the six degrees of freedom of the rigid body.

[0105] "One side" refers to the upper surface of the marker, and "one point" is the intersection of the two wires on the side of the rigid body.

[0106] The coordinate system is unified by calibrating the camera's intrinsic and extrinsic parameters using a calibration plate to obtain the transformation matrix from the camera coordinate system to the world coordinate system, thereby constructing a unified three-dimensional coordinate system.

[0107] The virtual-real fusion uses a rigid body registration algorithm to calculate the transformation relationship between the CT model and the patient coordinate system in order to complete the initial registration between the model and the real anatomical structure.

[0108] The closed-loop correction achieves error closed-loop convergence by detecting and feeding back the registration error and the device pose deviation, and iteratively updating the registration relationship.

[0109] An augmented reality system accuracy closed-loop correction system based on three-dimensional marked fields and one-point localization on a surface includes:

[0110] (1) Data acquisition module, used to acquire preoperative CT image data and spatial location information of intraoperative markers and surgical instruments;

[0111] (2) Coordinate unification module, used to establish a unified spatial coordinate system among patients, CT images, surgical instruments, markers and cameras;

[0112] (3) Virtual-real fusion module, used to realize spatial registration between CT reconstruction model and intraoperative patient scene;

[0113] (4) Closed-loop correction module, used to monitor and correct registration error based on one-face-one-point positioning strategy;

[0114] (5) User interaction module, used to display surgical navigation information, registration status and deviation prompts through augmented reality interface.

[0115] The user interaction module supports multi-view observation and overlays surgical navigation information, registration status, and deviation prompts onto the real scene on the augmented reality display device.

Claims

1. A method for accuracy correction of a three-dimensional marker field and one-point positioning enhanced reality system, characterized in that: The correction method is used for preoperative closed loop convergence of CT image-patient registration error and instrument pose error in interventional surgery navigation, and the correction method uses a precision closed loop correction system based on a three-dimensional marker field and one-face-one-point positioning, and the precision closed loop correction system comprises a data acquisition module, a coordinate unification module, a virtual-real fusion module, a closed loop correction module and a user interaction module which are connected through a data bus; the correction method firstly establishes a unified coordinate system by using the three-dimensional marker field, then forms full constraint of rigid body six degrees of freedom through one-face-one-point positioning strategy, and then monitors and feeds back the registration error in real time through the closed loop correction mechanism, and presents the surgery navigation information on the augmented reality interface.

2. The accuracy correction method for augmented reality systems using three-dimensional marked fields and point localization on a surface, as described in claim 1, is characterized in that: In the method, the data acquisition module is responsible for collecting preoperative CT image data and intraoperative industrial camera real-time images; the CT image data is used to reconstruct or generate a three-dimensional anatomical model of the patient, and the industrial camera real-time image is used to capture the spatial position information of the intraoperative marker and the surgical instrument in real time, thereby providing a data basis for coordinate unification and spatial registration; The coordinate unification module establishes a unified spatial coordinate system among the patient, the CT image, the surgical instrument, the marker and the camera based on the three-dimensional marker field, so as to ensure that the multi-source data is processed under the same coordinate system and to avoid the cumulative error caused by the non-unified coordinates; The virtual-real fusion module is used to realize the spatial accurate registration of the CT reconstruction model and the intraoperative real patient scene, so as to ensure that the virtual model and the actual anatomical structure of the patient are consistent in space; The closed loop correction module adopts a single-face-single-point positioning strategy, i.e., one-face-one-point positioning strategy, to detect the registration error and the instrument pose deviation in real time, and to realize the closed loop convergence of the error through iterative feedback, thereby ensuring that the navigation accuracy of the whole surgery meets the requirements; The user interaction module comprises a multi-view augmented reality interface, which displays the surgery navigation information, the registration state and the deviation prompt in real time, thereby providing intuitive operation feedback and decision basis for the operator.

3. The method of claim 2, wherein the method further comprises: determining a distance between the first point and the second point; and determining a third point on the surface of the object based on the distance and the first point. The marker is a three-dimensional marker arranged on the patient's body surface and the surgical instrument; in the method, the coordinate unification module calibrates the internal and external parameters of the camera by using the calibration board, obtains the conversion matrix from the camera coordinate system to the world coordinate system, and constructs a unified three-dimensional coordinate system, so as to ensure that all data is operated in the same coordinate system and to reduce the cumulative error caused by multiple coordinate conversions; The method for establishing a unified spatial coordinate system among the patient, the CT image, the surgical instrument, the marker and the camera by the coordinate unification module is as follows: five coordinate systems of the patient {Pat}, the CT {CT}, the surgical instrument {Nee}, the marker {Mar} and the camera {Cam} are established, and the unification is completed through the following transformation matrix: wherein, for mapping patient coordinates to camera coordinates, for aligning pre-operative CT with intra-operative patient position, defining the spatial relationship between the needle tip and the calibration object, real-time positioning is achieved.

4. The method of claim 3, wherein the method is used for a three-dimensional marker field and one-plane one-point positioning enhanced reality system. The three-dimensional marker of the surgical instrument is pasted on the rigid calibration device connected with the handle of the puncture guide needle, and the arrangement principle is as follows: avoiding the operating part at the front end of the instrument to ensure that the puncture accuracy and the sterilization of the operation area are not affected; selecting the part with rigid and planar features on the instrument to ensure that the marker and the instrument maintain a fixed spatial rigid body relationship; and ensuring that the marker and the industrial camera maintain a non-shielded field of view relationship so as to be tracked in real time. The three-dimensional marker pasted on the patient's body surface is arranged on the planar area of the chest and abdomen of the patient. The fixing method of the marker is as follows: a PVC waterproof label made of a medical-grade non-reflective flexible material is used, and the label is fixed on the skin surface through medical double-sided adhesive or low-sensitivity paste, so as to ensure stable adhesion and non-invasion during the operation; The marker is realized by 3D printing, and the marker material does not use metal objects.

5. The method of claim 2, wherein the method is used for a three-dimensional marker field and one-plane one-point positioning enhanced reality system. When the data acquisition module works, CT image data is imported through the DICOM standard format, and a 3D slicer three-dimensional reconstruction algorithm is used to generate a patient anatomical structure model; The marker image collected by the industrial camera is obtained through a computer vision method of feature point detection to obtain the three-dimensional coordinates of the marker.

6. The method of claim 2, wherein the method is used for a three-dimensional marker field and one-plane one-point positioning enhanced reality system. When the virtual-real fusion module is working, it first uses the 3D coordinates of the markers to determine the patient's reference coordinate system. Then, it calculates the optimal rigid body transformation matrix from the CT model to the patient's coordinate system using a rigid body registration algorithm, completing the initial registration between the virtual model and the real anatomical structure. After registration, the resulting virtual model can maintain high-precision consistency with the patient's surface and internal anatomical structure in the augmented reality interface. Specifically, the coordinate transformation uses a point cloud registration method based on singular value decomposition (SVD): extracting the corresponding point set {p} between the CT model and the patient markers. i },{q i }, by minimizing The rotation matrix R and the translation vector t are obtained, and finally the rigid body transformation matrix [R|t] is obtained, so as to realize the unification of the CT model and the patient anatomical coordinates; Calibration of industrial camera internal and external parameters by using calibration board, and establishment of conversion matrix T from camera coordinate system to world coordinate system CW ; reading the coordinates of the marker, the puncture entry point and the target point in the CT model, and combining the coordinates of the patient surface marker points to construct the conversion matrix T from the CT coordinates to the patient coordinates CP ; finally, the coordinate relationship of each object in the system is: X W = T CW · X C , X P = T CP · X CT where X C represents the point in the camera coordinate system, X CT represents the point in the CT model; through the unified coordinate system, the virtual CT model and the real patient body position are highly consistent in the AR interface.

7. The method of claim 2, wherein the method is used for a three-dimensional marker field and one-plane one-point positioning enhanced reality system. The closed-loop correction module adopts a one-plane-one-point positioning strategy to monitor and correct the registration accuracy in real time; a planar marker is selected in the three-dimensional marker field to constrain multiple degrees of freedom of translation and rotation, and a single-point marker is selected on the plane to constrain the remaining degrees of freedom, so as to realize the full constraint of six degrees of freedom of the rigid body; The "one plane" corresponds to the upper surface of the marker, and the "one point" is the intersection of the two straight lines on the side surface of the rigid body; The specific method is as follows: first, select the planar marker pasted on the patient's abdomen in the three-dimensional marker field to constrain five degrees of freedom of translation and rotation in the three-dimensional space; then select a single-point marker on the plane to constrain the remaining one degree of freedom, so as to realize the full constraint positioning of six degrees of freedom. The one-plane-one-point positioning strategy comprises: Introduce "one side one point" positioning strategy, which is: plane marker provides translation (t x ,t y ) and rotation (θ x ,θ y ,θ z ) constraints, intersection marker provides depth (t z ) constraint, and the combination of the two realizes six degrees of freedom full constraint: C' marker ∪C' point ={t x ,t y ,t z ,θ x ,θ y ,θ z} Thus, a closed-loop feedback is constructed to realize the accuracy self-calibration before and during the operation.

8. The method of claim 2, wherein the method is used for a three-dimensional marker field and one-plane-one-point positioning enhanced reality system. The user interaction module adopts an augmented reality display device to superimpose the surgical navigation information, registration state and deviation prompt in real time in the display of the patient's real scene, supports simultaneous multi-view observation, and the operator can freely select the orthoview, top view or perspective mode to comprehensively observe the surgical area and observe the real-time pose of the instrument and the patient from multiple angles.

9. The method for accuracy correction in augmented reality systems using three-dimensional marked fields and point localization on a surface, as described in claim 1, is characterized in that: The method comprises the following steps: Step 1, in the preoperative stage, data acquisition is performed, specifically: obtaining the CT image data of the patient before the operation, and capturing the spatial position information of the intraoperative marker and surgical instrument in real time through an industrial camera; Step 2, in the preoperative stage, coordinate unification is performed, specifically: based on the three-dimensional marker field, a unified spatial coordinate system is established among the patient, CT image, surgical instrument, marker and camera; Step 3, in the preoperative stage, virtual-real fusion is performed, specifically: under the unified coordinate system, the CT three-dimensional model is spatially registered with the real scene of the patient during the operation; Step 4, in the intraoperative stage, the spatial positions of the instrument and the marker are captured in real time, and closed-loop correction is performed, specifically: the one-plane-one-point positioning strategy is used to monitor the registration accuracy, and the error is corrected through iterative feedback; Step 5, in the intraoperative stage, user interaction and visualization are performed, specifically: the surgical navigation information, registration state and deviation prompt are displayed in real time in the augmented reality interface.

10. The method of claim 9, wherein the method is used for a three-dimensional marker field and one-plane-one-point augmented reality system accuracy correction. The method is used for percutaneous interventional surgical navigation, and further comprises: in the postoperative stage, verifying the guidance accuracy of the augmented reality system according to the CT result.

Images