Endoscope augmented reality error compensation method and device

Abstract

The invention relates to an endoscope augmented reality error compensation method and device. The accuracy of AR fusion is significantly improved. The camera posture error caused by the rotation of the endoscope tube can be accurately compensated, so that the AR fusion error of the system is maintained at a low level. The application of the compensation model avoids the recalibration of the endoscope camera and is highly sensitive to the system positioning noise and the corner point recognition noise in the image. The method includes: (1) obtaining the observation value of the endoscope camera posture through endoscope calibration and external tracking; (2) calculating the optimal camera posture compensation matrix in the current operation space by combining the observation value and the estimated value of the endoscope camera posture using the least squares method.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to an endoscope augmented reality error compensation method and an endoscope augmented reality error compensation device. Background Art

[0002] Over the past decade, surgical navigation has become an important auxiliary tool in minimally invasive surgery to ensure the accuracy and safety of the surgical process. In recent years, with the development of augmented reality (AR) technology, this technology has been increasingly combined with surgical navigation. AR-based surgical navigation provides doctors with shape and position information of important tissue structures such as tumors, nerves, and blood vessels. By superimposing AR information of specific tissues, organs, or areas of interest on the endoscope or microscope view without the need for additional displays, it provides doctors with richer navigation information while avoiding distracting the doctor's vision and attention.

[0003] AR-based surgical navigation usually requires two steps: image registration and camera calibration (including distortion coefficients, internal and external parameters). These two steps realize the unification of the three main coordinate systems involved in the AR imaging process, namely the real-world coordinate system of the person, the virtual-world coordinate system of the image, and the camera coordinate system. Several studies on endoscopic AR surgical navigation have shown that the hardware configurations they use are roughly the same, including optical or electromagnetic external trackers, position sensors, and endoscopes. By fixing the position sensor on the endoscope and using an external tracker to obtain the rigid transformation from the endoscope camera coordinate system to the real-world coordinate system in real time, the virtual object can be aligned with the endoscope imaging plane to achieve AR fusion and superimposed display. This requires three invariant parameters to achieve: 1) Calculate the rigid transformation from the virtual world coordinate system to the real world coordinate system through real-image registration; 2) Calculate the camera imaging model parameters through camera calibration for mapping 3D world coordinates to 2D imaging planes; 3) Calculate the rigid transformation from the camera coordinate system to the world coordinate system through hand-eye calibration. These parameters are updated as the endoscope moves during AR navigation, and the accuracy of these parameters will significantly affect the accuracy of the AR fusion display.

[0004] Real-image registration in surgical navigation has evolved from fiducial-based to surface-based registration methods, and can now provide sub-millimeter navigation accuracy. Camera calibration, as one of the basic tasks in the field of computer vision, has always been a hot topic in this field. By calibrating the distortion coefficients, internal parameters, and external parameters of a 2D camera, a mapping relationship between the three-dimensional world and the two-dimensional imaging plane of the camera is established, thereby guiding the computer to recognize the entire real world. In an AR-based endoscopic surgical navigation system, the camera's external parameter calibration problem is often regarded as a hand-eye calibration problem to be solved. Through hand-eye calibration, the relative rigid transformation of the endoscopic camera relative to the optical marker is calculated, thereby achieving real-time tracking and positioning of the endoscopic camera. The concept of hand-eye calibration comes from the robot vision system, in which the "hand" and "eye" refer to the robot manipulator and the camera fixed to it, respectively. Depending on the system hardware configuration, the hand-eye calibration problem in the endoscopic AR system can be described by the equation AX=XB or AX=YB. The methods for solving these two equations have always been a research field that has attracted much attention.

[0005] Although it is obvious that higher-precision registration and calibration can lead to higher AR navigation accuracy. However, due to the presence of noise in the system, there are still residual spatial transformation errors in the registration and calibration results. And the residual errors in each spatial transformation will accumulate, thus affecting the accuracy of AR fusion. According to our investigation, traditional endoscopic AR navigation systems use a combination of the above two independent calibration steps to achieve AR navigation, while ignoring the impact of accumulated errors in multiple spatial transformations on AR imaging performance. The difference is that Lee et al. proposed a one-step calibration method for AR navigation systems in 2020. They used special markers to achieve simultaneous optimization of patient-image registration and endoscopic hand-eye calibration. Although this can improve the accuracy of AR fusion to a certain extent, it does not really solve the problem of decreased navigation accuracy due to error accumulation. At the same time, this method requires patients to stick special markers for image scanning, which undoubtedly increases the workload of the operator before surgery and is not friendly to the clinical environment. Summary of the invention

[0006] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide an endoscope augmented reality error compensation method, which has significantly improved the accuracy of AR fusion and can accurately compensate for the camera posture error caused by the rotation of the endoscope tube, so that the system's AR fusion error is maintained at a low level. The application of this compensation model avoids the recalibration of the endoscope camera and is highly sensitive to the system positioning noise and the corner point recognition noise in the image.

[0007] The technical solution of the present invention is: this endoscope augmented reality error compensation method comprises the following steps:

[0008] (1) Obtain the observation value of the endoscope camera posture through endoscope calibration and external tracking;

[0009] (2) By combining the observed and estimated values ​​of the endoscope camera pose, the least squares method is used to calculate the optimal camera pose compensation matrix in the current operation space.

[0010] The present invention obtains the observation value of the endoscope camera pose through endoscope calibration and external tracking, and then proposes an interactive endoscope camera pose estimation method based on 2D marking. By combining the observation value and the estimated value of the endoscope camera pose, the least squares method is used to calculate the optimal camera pose compensation matrix in the current operation space. Therefore, the accuracy of AR fusion is significantly improved, and the camera pose error caused by the rotation of the endoscope tube can be accurately compensated, so that the AR fusion error of the system is maintained at a low level. The application of the compensation model avoids the recalibration of the endoscope camera and is highly sensitive to the system positioning noise and the corner point recognition noise in the image.

[0011] Also provided is an endoscope augmented reality error compensation device, comprising:

[0012] An observation value acquisition module configured to obtain observation values ​​of the endoscope camera posture obtained through endoscope calibration and external tracking;

[0013] The camera pose compensation module is configured to calculate the optimal camera pose compensation matrix in the current operation space by combining the observed value and the estimated value of the endoscope camera pose using the least squares method. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the endoscope augmented reality error compensation method according to the present invention. DETAILED DESCRIPTION

[0015] like Figure 1 As shown, the endoscope augmented reality error compensation method comprises the following steps:

[0016] (1) Obtain the observation value of the endoscope camera posture through endoscope calibration and external tracking;

[0017] (2) By combining the observed and estimated values ​​of the endoscope camera pose, the least squares method is used to calculate the optimal camera pose compensation matrix in the current operation space.

[0018] The present invention obtains the observation value of the endoscope camera pose through endoscope calibration and external tracking, and then proposes an interactive endoscope camera pose estimation method based on 2D marking. By combining the observation value and the estimated value of the endoscope camera pose, the least squares method is used to calculate the optimal camera pose compensation matrix in the current operation space. Therefore, the accuracy of AR fusion is significantly improved, and the camera pose error caused by the rotation of the endoscope tube can be accurately compensated, so that the AR fusion error of the system is maintained at a low level. The application of the compensation model avoids the recalibration of the endoscope camera and is highly sensitive to the system positioning noise and the corner point recognition noise in the image.

[0019] Preferably, in step (1), define represents the transformation from coordinate system A to coordinate system B, and the related coordinate systems include optical tracking system {O}, real tracking marker {PM}, virtual object {V}, endoscope tracking marker {EM}, endoscope camera {EC} and endoscope image {EI}, where the virtual object is a 3D region of interest extracted from CT medical images;

[0020] By using the image registration method based on the identification points, at least three non-collinear point pairs are found in the image virtual space and the patient physical space, respectively, denoted as P V and P PM , use SVD decomposition to minimize the following formula (1) and find the optimal transformation relationship between the two coordinate systems

[0021]

[0022] where N is the number of point pairs used for patient-image registration, ‖·‖ 2 represents the L2 norm;

[0023] The corresponding camera calibration task includes the calibration of the camera's external parameters, intrinsic parameters, and distortion coefficients. The camera imaging model is simplified to a pinhole model, which is represented by the camera's intrinsic parameter matrix K:

[0024]

[0025] Among them, u0 and v0 are the coordinates of the main points, f x and f y are the focal lengths in the directions of the two coordinate axes respectively;

[0026] The intrinsic parameters and distortion coefficients of the camera are calculated by taking at least three images containing a two-dimensional chessboard at different orientations;

[0027] The marker {EM} is fixed on the endoscope tube, and a checkerboard {CB} with markers {CM} is designed. During the calibration process, the endoscope is used to capture data at N (N ≥ 2) positions, and the transformation from the endoscope camera to the endoscope marker is calculated by minimizing the following formula (3):

[0028]

[0029] in They are respectively the matrices X and Y to be solved in the hand-eye calibration equation.

[0030] Preferably, in step (2), assuming that {EC-V} and {EC-R} represent the observed value and the true value of the camera pose, respectively, then due to the residual error of the spatial transformation in the system, there will be a certain offset between {EC-V} and {EC-R}, expressed as T err ;

[0031] The observation posture of the endoscope camera T ob It is expressed as:

[0032]

[0033] in Obtained through offline hand-eye calibration, Real-time acquisition via optical tracker;

[0034] Find the best T err So that it satisfies formula (5):

[0035] min‖T ob T err -T est ‖ 2 (5).

[0036] Preferably, in step (2), T est The solution problem is transformed into two homography transformation estimation problems, namely: the homography transformation from the chessboard to the camera coordinate system Homography transformation from chessboard to optical tracking system OTS coordinate system

[0037] Preferably, in step (2), the problem of estimating the pose relationship between the camera and the 2D marker using the projection model of the camera is transformed into minimizing the following formula (6):

[0038]

[0039] Among them, P CB and P CAMdenote the point coordinates of the feature point in the 2D marker coordinate system and the camera image coordinate system, respectively. The subscript i is the corner point index, N is the number of corner points, N ≥ 3. The operator Q(·) is used to project the points of the chessboard to the camera image plane. The camera's intrinsic parameters K and distortion coefficient D are calculated through an offline calibration process. The LM algorithm is used to iteratively obtain the optimal solution of formula (6):

[0040] Preferably, in step (2), the homography transformation between the 2D marker and the OTS coordinate system is completed by a fast homography estimation method between noisy planes, comprising: first, using the tracked probe to select corner points in the 2D marker for homography matrix calculation, and selecting four corner points of the corners, denoted as P O , and draw on the 2D marker to collect the point set data of the plane; then, the RANSAC algorithm is used to quickly estimate the plane equation l of the obtained points P , and P O The projection point on the plane is taken as its approximate real coordinate; the homography estimation problem between the 2D marker and the OTS is transformed into minimizing the following formula (7):

[0041]

[0042] Among them, Q(·) is used to convert point P O Projection to l P plane, and use the LM algorithm to iteratively obtain the optimal solution of formula (7) The estimated value of the endoscope camera pose is:

[0043]

[0044] Preferably, in step (2), the part in brackets of formula (5) is set to 0, and the optimal error matrix T that makes formula (5) valid is solved. err :

[0045] T err =(T ob ) -1 T est (9)

[0046] The compensated camera pose is It is expressed as:

[0047]

[0048] in is the fixed transformation from the endoscope camera {EC} to the endoscope landmark {EM} obtained through the hand-eye calibration process.

[0049] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an endoscope augmented reality error compensation device, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0050] An observation value acquisition module configured to obtain observation values ​​of the endoscope camera posture obtained through endoscope calibration and external tracking;

[0051] The camera pose compensation module is configured to calculate the optimal camera pose compensation matrix in the current operation space by combining the observed value and the estimated value of the endoscope camera pose using the least squares method.

[0052] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. An endoscope augmented reality error compensation method, characterized in that: It includes the following steps: (1) Obtain the observation value of the endoscope camera posture through endoscope calibration and external tracking; (2) By combining the observed and estimated values ​​of the endoscope camera pose, the optimal camera pose compensation matrix in the current operation space is calculated using the least squares method; In the step (1), define represents the transformation from coordinate system A to coordinate system B, and the related coordinate systems include optical tracking system {O}, real tracking marker {PM}, virtual object {V}, endoscope tracking marker {EM}, endoscope camera {EC} and endoscope image {EI}, where the virtual object is a 3D region of interest extracted from CT medical images; By using the image registration method based on the identification points, at least three non-collinear point pairs are found in the image virtual space and the patient physical space, respectively, denoted as P V and P PM , use SVD decomposition to minimize the following formula (1) and find the optimal transformation relationship between the two coordinate systems where N is the number of point pairs used for patient-image registration, ‖·‖ 2 represents the L2 norm; The corresponding camera calibration task includes the calibration of the camera's external parameters, intrinsic parameters, and distortion coefficients. The camera imaging model is simplified to a pinhole model, which is represented by the camera's intrinsic parameter matrix K: Among them, u0 and v0 are the coordinates of the main points, f x and f y are the focal lengths in the directions of the two coordinate axes respectively; the intrinsic parameters and distortion coefficients of the camera are calculated by taking at least three images containing a two-dimensional chessboard at different orientations; The marker {EM} is fixed on the endoscope tube, and a checkerboard {CB} with markers {CM} is designed. During the calibration process, the endoscope is used to capture data at N (N ≥ 2) positions, and the transformation from the endoscope camera to the endoscope marker is calculated by minimizing the following formula (3): in They are the matrices X and Y to be solved in the hand-eye calibration equation respectively; In step (2), assuming that {EC-V} and {EC-R} represent the observed value and the true value of the camera pose, respectively, the residual error of the spatial transformation in the system will cause a certain offset between {EC-V} and {EC-R}, which is expressed as T err ; The observation posture of the endoscope camera T ob It is expressed as: in Obtained through offline hand-eye calibration, Acquired in real time by an optical tracker; find the optimal T err So that it satisfies formula (5): min‖T ob T err -T est ‖ 2 (5); In the step (2), T est The solution problem is transformed into two homography transformation estimation problems, namely: the homography transformation from the chessboard to the camera coordinate system Homography transformation from chessboard to OTS coordinate system In the step (2), the problem of estimating the pose relationship between the camera and the 2D marker using the camera projection model is transformed into minimizing the following formula (6): Among them, P CB and P CAM denote the point coordinates of the feature point in the 2D marker coordinate system and the camera image coordinate system, respectively. The subscript i is the corner point index, N is the number of corner points, N ≥ 3. The operator Q(·) is used to project the points of the chessboard to the camera image plane. The camera's intrinsic parameters K and distortion coefficient D are calculated through an offline calibration process. The LM algorithm is used to iteratively obtain the optimal solution of formula (6): In the step (2), the homography transformation between the 2D marker and the OTS coordinate system is completed by a fast homography estimation method between noisy planes, including: first, using the tracked probe to select corner points in the 2D marker for homography matrix calculation, and select four corner points of the corners, denoted as P O , and draw on the 2D marker to collect the point set data of the plane; then, the RANSAC algorithm is used to quickly estimate the plane equation l of the obtained points P , and P O The projection point on the plane is taken as its approximate real coordinate; the homography estimation problem between the 2D marker and the OTS is transformed into minimizing the following formula (7): Among them, Q(·) is used to convert point P O Projection to l P plane, and use the LM algorithm to iteratively obtain the optimal solution of formula (7) The estimated value of the endoscope camera pose is: In step (2), let the part in brackets of formula (5) be 0, and solve the optimal error matrix T that makes formula (5) valid. err : T err =(T ob ) -1 T est (9) The compensated camera pose is It is expressed as: in is the fixed transformation from the endoscope camera {EC} to the endoscope landmark {EM} obtained through the hand-eye calibration process.

2. Endoscope augmented reality error compensation device, characterized in that: It is used to perform the method according to claim 1 and comprises: An observation value acquisition module configured to obtain observation values ​​of the endoscope camera posture obtained through endoscope calibration and external tracking; The camera pose compensation module is configured to calculate the optimal camera pose compensation matrix in the current operation space by combining the observed value and the estimated value of the endoscope camera pose using the least squares method.

Images