Systems and methods for describing an endoscope and an auto-calibrating endoscope camera system

Abstract

A system and method for determining the calibration of an endoscopic camera, the endoscopic camera comprising a camera equipped with replaceable, rotatable optics (rigid endoscope), wherein the calibration is performed in an operating room without user intervention. First, the rigid endoscope is described by a set of parameters. Then, this lens descriptor is used as input to real-time software that processes images acquired by any camera equipped with the rigid endoscope to provide calibration of the complete endoscopic camera arrangement at every frame, regardless of the relative rotation or zoom setting of the lens endoscope relative to the camera. An image software method is also disclosed that detects whether the lens descriptor is incompatible with the rigid endoscope used to prevent errors and warn the user of erroneous conditions.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 911,950 (“the ‘950 Application”), filed October 7, 2019, and U.S. Application No. 62 / 911,986 (“the ‘986 Application”), filed October 7, 2019. The entire contents of the ‘950 Application and the ‘986 Application are hereby incorporated by reference herein for all purposes. TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of computer vision and photogrammetry, and in particular, but not by way of limitation, the presently disclosed embodiments are used in the context of clinical handling of surgical and diagnostic procedures for calibrating endoscope camera systems with replaceable, rotatable optics (rigid endoscopes), identifying whether a specific endoscope is in use, or verifying whether the endoscope is properly assembled to the camera head. These endoscopy systems are used in various medical fields, such as orthopedics (arthroscopy) or abdominal surgery (laparoscopy), and camera calibration enables application to computer-assisted surgery (CAS) and enhances visualization. BACKGROUND

[0004] Video-guided procedures such as arthroscopy and laparoscopy utilize cameras with rigid endoscopes that enable the surgeon to visualize the interior of the anatomical cavity of interest. Depending on the medical specialty, the rigid endoscope can be an arthroscope, a laparoscope, a neuroscope, etc., combined with a camera comprising a camera head and a camera control unit (CCU) to form an endoscope camera. These cameras differ from conventional cameras mainly in two characteristics. The first characteristic is that, for ease of sterilization, the rigid endoscope (also referred to as a lens scope or optics) is usually replaceable, and the surgeon attaches the endoscope to the camera head in the operating room (OR) before starting the medical procedure. This attachment is achieved with a connector that allows the endoscope lens to rotate around its axis of symmetry (mechanical axis in ) relative to the camera head, thereby allowing the surgeon to change the direction of view without translating the endoscope camera. The second unique characteristic of the endoscope camera is that it usually contains a field stop mask (FSM) somewhere along the image relay system, which causes the acquired image to have meaningful content in a circular region surrounded by black frames. The FSM usually contains a marker on the circular border, the purpose of which is to allow the surgeon to infer the downward direction. This marker on the circular image periphery will be referred to as the notch ( ) hereafter. Figure 2 Figure 2

[0005] ​​An important enabling step for computer-assisted arthroscopy or laparoscopy is camera calibration, so that 2D image information can be related to the 3D scene for enhanced visualization, improved perception, measurement and / or navigation. Calibration in this context includes the camera system with a lensed scope comprising determining the parameters of a projection model that maps a projected ray in 3D to a point in pixel coordinates in the image, and vice versa Figure 1 ). In the context of medical endoscopy, the application of calibration is broad - from distortion correction and rendering of virtual views for enhanced visualization to surgical navigation, where the camera is used to measure 3D points and distances and the related information often encompasses the patient's body structure.

[0006] In endoscopic cameras with rotatable optics, the motion between the rigid endoscope and the camera sensor causes a change in the calibration parameters of the camera system, which means that the projection model does not remain constant over time as in regular cameras. Since performing an independent calibration for each possible position of the optics relative to the camera head is impractical, the calibration parameters must be updated according to a camera model that takes this relative motion into account. Literature has proposed solutions for determining this relative rotation and updating the calibration accordingly, examples include using a rotary encoder attached to the camera head [1], or employing an optical tracking system for determining the position of optical markers attached to the scope barrel [2]. These methods have a serious drawback in that they require additional equipment and instrumentation, which is costly, takes up space in the OR, and disrupts the established surgical procedure.

[0007] US Patent No. 9,438,897 discloses a method that solves some of the above problems for accomplishing endoscopic camera calibration without the need for any additional instrumentation. Image processing is used to estimate the rotation of the lensed scope at each frame instant, and the result is used as input to a model that outputs the calibration at the current angular position given the camera calibration at a specific angular position of the lensed scope relative to the camera head (reference position). However, this method has the following drawbacks: (i) the calibration at the reference position requires taking one or more frames of a known checkerboard pattern (calibration grid) in the OR, which is a time-consuming process that must be performed with a sterile grid in addition to requiring user intervention, thus being undesirable and should be avoided, (ii) the disclosed method does not allow changing the optical zoom during operation, since it is unable to update the calibration parameters to different zoom levels, and (iii) the method requires the lensed scope to rotate only relative to the camera head without translation, where the point where the mechanical axis intersects the image must be determined explicitly.

[0008] The presently disclosed embodiments relate to a method that avoids the need to calibrate the endoscope camera in the OR at a specific angular position after assembling the rigid scope into the camera. This patent discloses models, methods and devices for describing a rigid endoscope in such a way that it enables to determine the calibration of any endoscope camera system comprising an endoscope independent of the camera being used, the zoom amount introduced by the camera, and the relative rotation or translation between scope and camera at a specific frame instant. This allows the surgeon to change the endoscope and / or the camera during a surgical procedure and to adjust the zoom as needed in order to better display certain image content without causing any disturbance to the surgical flow.

[0009] The present disclosure shows how to calibrate a rigid endoscope individually to obtain a set of parameters that fully describe the optics - the lens descriptor. The lens calibration is performed upfront (e.g. at manufacturing time) and the descriptor is then loaded into the camera control unit (CCU) to be used as input in real-time software that automatically provides the calibration of the complete endoscope camera setup (including camera and lens) at every frame instant, regardless of the relative rotation between the two components. This is achieved in a seamless way for the user.

[0010] Since this descriptor describes one lens or a batch of lenses, it can also be used for identification and quality control. Therefore, on this functional basis, a method for detecting inconsistencies between the lens descriptor loaded into the CCU and the actual rigid endoscope assembled in the camera is also disclosed. This method helps to warn the user in case the lens used is incorrect and / or the lens is damaged or not correctly assembled in the camera.

[0011] The present disclosure can be used in particular, but not exclusively, in combination with the methods disclosed in US Patent No. 9,438,897 to correct image radial distortion and enhance visual perception, or in combination with the methods disclosed in US 20180071032 Al to provide guidance and navigation during arthroscopy in order to complete the camera calibration at every frame instant, which is a requirement for these methods to work. SUMMARY

[0012] System, method and device for determining the calibration of an endoscope camera consisting of a camera equipped with a replaceable, rotatable optic (rigid endoscope) so that 2D image points can be related to 3D projected rays in computer assisted surgery applications and wherein the rigid endoscope is pre-described (e.g. in the factory at manufacturing time) to complete the camera calibration without any user intervention in the operating room (OR).

[0013] A method for calibrating an endoscope camera by a set of parameters A method of describing a rigid endoscope, the parameters are then used as input for real-time software that processes images acquired by any camera equipped with a rigid endoscope to provide at each frame instant a calibration of the complete endoscope camera arrangement, regardless of the relative rotation or zoom setting between the lens scope and the camera.

[0014] Lens description sub-image software method whether or not compatible with the rigid endoscope used, which can be used to prevent errors and warn the user of error situations such as the use of the wrong lens, defects in the lens or camera or improper assembly of the lens in the camera. BRIEF DESCRIPTION OF DRAWINGS

[0015] For a more complete understanding of the present disclosure, reference is made to the following detailed description of exemplary embodiments considered in conjunction with the accompanying drawings.

[0016] Figure 1 is an embodiment of a camera projection model that depicts the mapping of a projection ray defined by a 3D point X in the scene and the projection center of the camera to a 2D point x in the pixel coordinates of the image plane. The model can be seen as a composition of the projection P according to the pinhole model that maps 3D points X to 2D, a function Γ that accounts for the non-linear effects of the radial distortion quantified by ξ introduced by the optics, and the intrinsic parameters K(f, O) of the camera that depend on the focal length f and the principal point O where the optical axis projects onto the image.

[0017] Figure 2 An exemplary embodiment of an image formed by an endoscope camera with interchangeable, rotatable optics (rigid endoscope) is shown. The lens is cylindrical and rotates in a rotation center Q around its longitudinal, symmetric axis of intersection with the image plane, which is hereafter referred to as the mechanical axis. The rigid endoscope has a field stop mask (FSM) at some position of the image relay system that projects as a black frame (circular border) around a circle Ω with center C onto the image plane, which contains meaningful visual content (see illustration of Fig. 3). The image border usually has a marker in the image point P, which is hereafter referred to as the notch, that allows the surgeon to infer the rotation between the scope and the camera. If the scope rotates around the mechanical axis by an angle δ with respect to the camera, then the circle center C, the notch P and the principal point O also rotate by the same amount δ around Q.

[0018] Figure 3AThe sequence of steps for detecting the circular boundary C and notch P at each frame time is illustrated. Starting with initial estimates of the boundary Ω and notch P, the algorithm includes: reconstructing a ring image centered on P (step 1); detecting edge points on the ring image (step 2); and iterating through the following steps until a stopping criterion is met: mapping the edge points back to image space for estimating boundary circle proposals (step 3); and reconstructing a new ring image for more accurate edge point detection (step 4). This loop stops when edge points in the ring image are collinear. The final step involves detecting the notch P in the ring image using a correlation-based strategy with a known notch template. The detected boundary Ω and notch P are used as initial settings in subsequent frame time steps.

[0019] Figure 3B The advantages of detecting multiple edge points in each column of the annular image are shown. In the case of strong light dispersion (dashed ellipse), the edge points (red dots) corresponding to the first local maximum along each column of the annular image do not always correspond to the correct image boundary, and the circle boundary estimation fails (yellow circle). This problem is solved by detecting multiple local maxima in each column of the annular image and using these edge points as input to a robust circle estimation scheme (green circle).

[0020] Figure 3C A strategy for resolving the problem of generating a ring-shaped image in which the lens notch region is discontinuous is shown. (Step 1) A fixed angular position (arrow) at the beginning of the ring-shaped image may cause the notch P to be partially cut off. (Step 2) The problem in Step 1 can be solved by determining the center of the ring-shaped image using an initial estimate of the lens notch P.

[0021] Figure 3D The strategy for obtaining a lens-specific notch template is illustrated. During calibration, and after generating the annular image (step 1), a specific region around the lens notch location P is extracted (the dashed rectangle in step 2). Then, an adaptive binarization strategy is employed to generate a notch template consisting of a bright triangular shape and a dark rectangular background (step 3).

[0022] Figure 4A The estimation of the rotation center Q is shown from two frames in which the notch P has been detected. i P j Q is obtained by simply connecting the line segment. and The value is estimated by the intersection of the angle bisectors, where C i C j It is the center of the boundary detected in two frames.

[0023] Figure 4B This demonstrates how the center C of the boundary detected in these frames is utilized. i Cj , C k to estimate the center of rotation Q from three frames. Q is the intersection of the bisector of the line segments and .

[0024] Figure 5 Embodiments of a field stop mask (FSM) containing a plurality of markers are shown, the markers having different shapes so as to be uniquely identifiable. The goal is to have redundancy to improve the accuracy of the rotation estimation and to be resilient to the situation where the circular boundary projects partially over the frame limit and a single notch can not be visible all the time, thus preventing or hindering the estimation.

[0025] Figure 6 Embodiments of a field stop mask (FSM) are shown, the FSM having an elliptical shape so that its lack of circular symmetry allows to always infer the position of the notch P.

[0026] Figure 7 A sequence of steps for obtaining camera calibration at a reference position i = 0 is shown. The user starts by acquiring K > 1 calibration images when the lens scope is at the reference angular position. Then an internal camera calibration is performed as well as the detection of the circular boundary and the notch. This data is stored in memory so that it is possible for the user to change the position of the lens scope and repeat the above steps N times. Then a final optimization scheme is performed that minimizes the re-projection error of all calibration images simultaneously.

[0027] Figure 8 An optimization scheme is shown that is employed when acquiring calibration images at N > 1 angular positions i of the scope, where i = 0 is the reference position and i = 1,..., N - 1 are additional positions. This optimization step enforces a rotation model of the scope that is a planar rotation around a center of rotation Q of an angle δ i transforming the principal point O0and the notch P0into points O i and P i respectively. The optimization scheme is used to estimate the calibration parameters at the reference position while minimizing the re-projection error in all acquired calibration images and enforcing this model of the scope rotation for all sampled angular positions. An initial setting of the center of rotation Q can be obtained from the method shown. Figure 7

[0028] Figure 9 A sequence of steps for the online update of the calibration parameters is shown. Every time a new frame j is acquired, the circular boundary and the notch are detected and the angular shift is computed using the estimate of the center of rotation Q obtained from the offline calibration results (mode 1) or by estimation through consecutive frames (mode 2). The last step, the calibration parameters are updated.

[0029] Figure 10 ​It is shown how to separate the parameters representing the lens comprised in a camera system describing a camera from the contribution of the camera, resulting in a lens descriptor and how to use it A procedure to estimate the calibration parameters of a new camera system (application camera) with the same lens as the camera from which the lens descriptor is derived. From the frames acquired with the camera, a helper reference frame attached to the border is considered, obtained by transforming the reference frame of the image by means of a similarity transformation A. The lens descriptor is obtained by representing the calibration parameters in border coordinates For the calibration of an application camera with the same endoscope lens, the entries of the lens descriptor represented in border coordinates are transformed into image coordinates using a similarity transformation B.

[0030] Figure 11 is an embodiment of the components of an endoscope showing the correlation between the center of rotation and the center of the FSM. This allows a better understanding of the physical meaning of the lens descriptor disclosed in this patent.

[0031] Figure 12 shows the sequence of steps of the anomaly detection procedure. For each acquired frame j, the detection of the border and the notch is performed for obtaining the updated calibration from the lens descriptor and for estimating the center of rotation using the previously detected border and notch. This results in two different center of rotation estimates (Q j and ) which are compared and allow the detection of anomalies.

[0032] Figure 13 is a diagrammatic view of an example computing system that includes a general purpose computing system environment. DETAILED DESCRIPTION

[0033] It should be understood that, even though a few embodiments of one or more implementations are described below, any number of other embodiments could be implemented in any number of ways. The disclosure should not be limited to the illustrative embodiments, drawings, and / or techniques shown below and described more fully in this specification, including the example designs and implementations illustrated and described herein. Additionally, the disclosure can be modified in any number of ways within the scope of the appended claims and their equivalents.

[0034] In this patent, 2D and 3D vectors are written in boldface lower case and upper case letters respectively. Functions are denoted in lower case italics, and angles in lower case Greek letters. Points and other geometric entities in a plane are represented in homogeneous coordinates, as is common practice in projective geometry, where 2D linear transformations in a plane are represented by 3x3 matrices, and equations are represented proportionally. Also, throughout the text, different parts are referenced by using the paragraph number of the symbol §.

[0035] 1. Camera model for endoscope systems with replaceable, rotatable optics (rigid endoscope)

[0036] Camera calibration is the process of determining the camera model that projects a 3D point X in the camera reference frame to a 2D image point x in pixel coordinates. Alternatively, the camera model can be interpreted as a function that back-projects an image point x to a ray that passes through a 3D point X in the scene. This process is illustrated in Figure 1 Camera calibration is a key component in many applications, including visual odometry and 3D reconstruction, and also including the removal of image artifacts in order to enhance the visual perception of e.g. radial distortion.

[0037] A regular, commonly used camera is described by the so-called pinhole model, which can be augmented with a radial distortion model that takes into account non-linear effects introduced by small optics and / or fish-eye lenses. In this case, a point X in the scene is projected onto a point x in the image according to the formula x = KΓ ξ (PX) onto the image, where x and X are expressed in homogeneous coordinates and the equation is expressed in proportion, P = [I 0 3x1 ] is a 3x4 projection matrix, where I denotes a 3x3 identity matrix, K is a so-called intrinsic parameter matrix with dimension 3x3, and Γ denotes a distortion function with parameters ξ. Hereafter, without loss of generality, it will be assumed that the camera is undistorted, with a uniform aspect ratio, resulting in a model that is very close to most modern cameras, where the deviation of the skew from zero and the deviation of the aspect ratio from one are negligible. Under this assumption, K depends only on the focal length f and the principal point O = [O x , O y , 1] T , such that

[0038]

[0039] The distortion function Γ represents a mapping in 2D, and can be any of the many distortion functions or models available in the literature, including but not limited to the polynomial model (also known as Brown's model), the split model, the rational model, the fish-eye lens model, etc., in its first or higher order (multi-parameter) version, with ξ being a scalar or a vector, respectively.

[0040] An endoscope camera resulting from the combination of a rigid endoscope and a camera has replaceable optics for easy disinfection, where the endoscope has an ocular (or eyepiece) at the proximal end that is assembled to the camera using a connector that typically allows the surgeon to rotate the scope relative to the camera. As Figure 2As shown, this rotation is performed around the longitudinal axis (mechanical axis) of the endoscope that intersects the image plane at point Q. The field stop mask (FSM) in the lens scope projects onto the image plane as a black frame around the region with visual content, which contains a circular boundary Ω with center C and notch P.

[0041] The mechanical axis is approximately coincident with the symmetry axis of the eyepiece, which is not necessarily aligned with the symmetry axis of the cylindrical scope and / or does not necessarily pass through the center of the circular region defined by the FSM. These alignments are purposeful but never perfectly achieved due to mechanical tolerances in the endoscope construction and manufacturing process. Thus, the center of rotation Q, the center C of the circular boundary, and the principal point O are generally different points in the image, which complicates the camera model but, as disclosed previously, can be used as a feature to identify a particular endoscope or batch of similar endoscopes.

[0042] Consider calibrating the endoscope camera for a particular position of the scope so that K(f, O) is the internal parameter matrix and ξ is a distortion parameter that quantifies the radial distortion according to a chosen model Γ Figure 1 ). If the scope is rotated by an angle δ with respect to the camera, then the distortion ξ and the focal length f remain unchanged, but the principal point O is rotated by the same amount δ around Q Figure 2 ). This makes the internal parameter matrix become K(f, R(δ, Q)O), where R(δ, Q) is a 3x3 matrix that represents a 2D rotation in the image around the point Q = [Q x , Q y , 1] T , by an angle δ.

[0043] Similarly to the rotation in the principal point O that makes it become O' = R(δ, Q)O, the rotation of the scope with respect to the camera makes the circle Ω with center C and notch P become Ω' with center C' = R(δ, Q)C and notch P' = R(δ, Q)P Figure 2 ).

[0044] 2. Calibrating an endoscope camera with interchangeable, rotatable optics

[0045] In summary, to always obtain the correct calibration parameters of an endoscope camera, one must know the focal length f, the distortion ξ, and the principal point O from a particular rotation angle between the camera and the lens scope (reference angular position), and one must update the position of the principal point during operation according to O' = R(δ, Q)O, which requires knowing the center of rotation Q and the angular shift δ between the current angular position and the reference angular position at each frame instant.

[0046] The calibration of an endoscope camera at a reference angular position that can be easily identified by the position of the notch P can be performed "offline", after which one follows Figure 7The steps to begin clinical treatment. The determination of f, ξ, and O requires the use of an internal camera calibration method ( Figure 7 In module A), the method receives one or more frames acquired at a reference position P (or P0) as input, as described in §§

[0045] -

[0049] . If the target is also to determine the rotation center Q, then additional angular positions P must be located at i = 1, ..., N. i The input frames are obtained in the middle, and these frames can be used to improve the accuracy of determining f, ξ and O at the reference angle position P0, as disclosed in §§

[0065] -

[0069] .

[0047] By following the further disclosures in §§

[0070] -

[0072] Figure 9 The steps involve updating the camera model "online" at each frame time during clinical treatment. The angular shift δ can be determined in several ways: using additional instruments such as an optical encoder [1] or optical tracking [2], or relying entirely on image processing. The disclosed embodiments will be considered in the use of the also disclosed image processing methods to determine the relative rotation between the lens endoscope and the camera without loss of generality. The method detects and estimates the rotation of the lens endoscope with center C in each frame i. i and notch P i Boundary contour Ω i The location (further disclosed in §§

[0050] -

[0057] ) Figure 9 Module B in the module then infers the corresponding angular shift δ relative to the reference position, given that the rotation center Q is previously known or unknown (further disclosed in §§

[0058] -

[0064] ). Figure 9 Module C in the middle.

[0048] 2.1 Camera calibration at a specific angular position, including intrinsic parameters K(f, O) and distortion ξ( Figure 7 Module A in

[0049] Numerous methods for calibrating pinhole cameras with radial distortion exist in the literature, and can be broadly categorized into two types: explicit methods and automatic calibration methods. Explicit methods use images of known calibration objects, such as general 3D objects, a set of spheres, or planar checkerboard patterns, while automatic calibration methods rely on the correspondence between consecutive frames of unknown, natural scenes. Both methods may require some degree of user supervision, ranging from manual to fully automatic, depending on the specific method and underlying algorithm.

[0050] The disclosed embodiments will be considered without loss of generality: camera calibration at specific angular positions of the endoscope relative to the camera will be performed using an explicit method that makes use of a known calibration object such as a planar checkerboard pattern and any other planar pattern that enables to establish point correspondences between the image and the calibration object. This method is advantageous over most competing methods due to good performance in terms of robustness and accuracy, ease of manufacturing the calibration object (planar grid) and the possibility to complete the whole calibration from a single image acquired from an arbitrary position of the rig. However, other explicit or automatic calibration methods can be employed to estimate the focal length f, the distortion ξ and the principal point O of the endoscope camera for specific relative rotations (reference angular positions) between the camera and the endoscope.

[0051] Explicit calibration using a planar checkerboard pattern typically comprises the following steps: acquisition of frames of the calibration object from arbitrary positions or 3D poses (rotation R and translation t of the object relative to the camera); establishment of point correspondences x, X between the image and the calibration object using image processing algorithms; execution of an appropriate calibration algorithm that uses the point correspondences to estimate the focal length f, the principal point O and the distortion parameters ξ, as well as the pose R, t of the object relative to the camera.

[0052] The method can be applied to a plurality of calibration frames I k , k = 0,..., K - 1, instead of a single calibration frame in order to improve robustness and accuracy. In this case, the calibration is performed independently for each frame and a final optimization step that minimizes the re-projection error is used to enforce the same intrinsic parameters K(f, O) and distortion ξ across the plurality of frames while taking into account the different poses R k , t k of each frame.

[0053] 2.2 Detection of the circular boundary and notches (module B in Figure 7 and Figure 9 )

[0054] As Figure 3AAs shown, the circular boundary and the notch of the FSM can be detected. The method first considers the initial setting of the boundary Ω and the notch P that are used as inputs to the warping function that reproduces the so-called annular image. The initial setting of the boundary Ω and the notch P can be obtained from a variety of methods including but not limited to deep / machine learning, image processing, statistics-based methods, and stochastic methods. For example, regarding the boundary Ω, it can be initialized by considering a circle centered at the center of the image with a radius equal to half of the minimum between the image width and height, by radially searching the transition between the black frame of the image and the region containing meaningful information, or by using a deep learning framework to detect a circle, a general conic, or any other desired shape. Regarding the notch P, it can be initialized in a random position on the boundary, or by using a learning scheme and / or image processing for detecting a known shape of the notch. Referring to Figure 3A steps 1 and 2 of the method, the annular image is obtained by considering an inner circle Ω i and an outer circle Ω o centered at C of Ω i with radii r i < r and r o > r, respectively, where r is the radius of Ω i . A uniform spacing between r o and r j defines a set of concentric circles Ω j . For each Ω i , the image signal is interpolated and concatenated. The assumed notch P is mapped to the center of the annular image.

[0055] Referring to step 2 of the method, edge points on the annular image are detected by searching for sharp changes in luminance along the direction from the periphery to the center of the boundary, which theoretically correspond to points belonging to the boundary. This is achieved by analyzing the magnitude of the 1-D spatial derivative response (gradient magnitude) along each column of the annular image. One possible solution to select these edge points is to pick the first local maximum of the gradient magnitude for each column. However, as shown in Figure 3B , there are cases where this approach fails (e.g., strong light dispersion near the boundary). To overcome this, for each column of the annular image, a set of M edge points corresponding to the local maxima of the gradient magnitude are selected.

[0056] The detected edge points are then mapped back to the Cartesian image space so that a circle boundary can be estimated. This is performed using a circle fitting method within a robust framework. Given a set of noisy data points that can be contaminated by outliers, the goal of circle fitting is to find a circle that minimizes or maximizes a certain error or cost function that quantifies the degree of fit of the given circle to the data points. The most widely used technique minimizes the geometric or algebraic (approximate) distance from the circle to the data points. To handle outlier data points, a robust framework such as RANSAC is typically employed. The annular image rendering step, the edge point detection step, and the robust circle estimation step are performed iteratively until the detected edge points are colinear in a robust manner. If this happens, the algorithm proceeds to the detection of the notch P by performing correlation with a known template of notches using the initial estimate of the notch position P. The output of this algorithm is the notch position P and the circle Ω with center C and radius r.

[0057] As shown in FIG. 3, by determining the center of the annular image using the initial estimate of the notch position P, it is guaranteed that the image portion corresponding to the notch is continuous so that detection can always be performed. Furthermore, the collinearity of the edge points is chosen as the stopping criterion because the edge points are collinear if and only if the estimated boundary is perfectly concentric with the true boundary. Figure 3C

[0058] As shown in FIG. 4, the notch template is extracted at calibration time and is specific to the lens. The template is typically composed of a bright triangle and a dark rectangular background. Figure 3D

[0059] The disclosed method for boundary and notch detection can have other applications, such as detecting the engraving in the FSM to read relevant information, including but not limited to specific characteristics of the lens.

[0060] In addition, although embodiments of this method assume that the boundary can be exactly represented by a circle, a general conic fitting can be used in the method without substantial modification.

[0061] 2.3 Image-based measurement of the relative rotation between the endoscope and the camera (Module C in FIG. 1) Figure 9

[0062] As mentioned previously, the calibration of the current frame i can be done by rotating the principal point O (or Oo) by an angle δ i around the center of rotation Q at the reference angular position. In this case, the center Q and the angular shift δ i between frame i and frame 0 corresponding to the reference position must be estimated.

[0063] Figure 4A ​​​The process of estimating Q from two frames i and j taken at two different angular positions is depicted. This is achieved by simply intersecting the bisector of the line segments whose endpoints are the centers C i , C j and the notch P i , P j . If the notch cannot be detected in the images due to occlusions, poor lighting, overexposure and light dispersion, etc., it is possible to estimate Q using only the centers of the circular boundaries detected in the three frames taken at different angular positions. The process is illustrated in Figure 4B , where it can be seen that Q is the intersection of the line segments obtained by connecting the centers of the boundaries detected in frames i, j and k.

[0064] If the center of rotation Q is known, and the center and notch at the reference angular position are C0and P0respectively, then the angular shift δ i can be inferred from the position of the notch P i , the center of the boundary C i or both . The position of the notch and the center of the boundary are determined by applying the steps of Figure 3 to the current frame i. If the notch P is not visible, then δ i can be determined from C0, C i .

[0065] Since the distance from the center of rotation Q to the notch P is significantly larger than the distance between Q and C, the estimation using the notch P is generally more robust and accurate, and it is therefore important that the notch can be detected in all frames. Since the detection of the notch is mainly affected by the occlusion situation, one solution is to consider multiple notches in the FSM to ensure that at least one notch is always visible in the frames. Figure 5 An exemplary FSM containing multiple markers with different shapes in order to be identified is shown, where one of these markers is used as a reference point or standard notch P. An alternative solution is to consider a FSM that projects the image boundaries that reproduce shapes that do not have circular symmetry onto black frames, in which case the lack of circular symmetry of the detected shape can be used to infer the position of the invisible notch or reference point P. Figure 6 An exemplary FSM that reproduces an elliptical boundary with the principal axis passing through the notch P is shown, which makes it possible to always infer the position of the notch.

[0066] The algorithm for detecting notches described in §§

[0050] -

[0057] can be extended to the case where the FSM contains multiple notches. To do this, the correlation signal of each notch is determined independently, all the signals are fused together using the known relative positions of the notches and the point with the highest correlation is found. Figure 3AThe last step in the process. This method ensures that at least one notch can be detected even if one or more notches are obscured.

[0067] Without loss of generality, in the remainder of this patent, it is assumed that the FSM has only one ever-visible notch P and that the rotation center Q is determined from two frames.

[0068] Whenever more than two frames are available at different angular positions, and in order to filter out the rotation center Q and relative rotation δ i The possible noise estimation can be achieved by applying filtering methods. The filter can take previous estimates of Q, the current boundary, and the notch as input and output the updated position and relative rotation δ of Q. i The estimation. This filtering technique can be implemented using any time filter, such as a Kalman filter or an extended Kalman filter.

[0069] 2.4 Offline calibration at the reference angle position

[0070] Figure 7 A schematic diagram is provided for the camera calibration process at a reference angle position i = 0, which can correspond to any relative angle between the rigid endoscope and the camera. With the lens endoscope at the reference angle position, the user begins by acquiring K ≥ 1 calibration images. Then, the 2D-3D correspondence of each image is extracted. And retrieve the orientation of the calibration object. and a set of internal parameters K(f) i O i ) and distortion ξ i To perform internal camera calibration as described in §§

[0045] -

[0049] (Module A). Also, to detect the center C by following the methods described in §§

[0050] -

[0057] (Module B). i radius r i and notch P i The circular boundary. This data is stored in memory, allowing the user to change the angular position by rotating the lens relative to the camera, and repeating the image acquisition, internal calibration, and boundary / notch detection process. This is performed for a total of N > 1 different angular positions i, where i = 0, 1, ... N-1 and i = 0 is the reference angular position, for which the final calibration can be obtained after a global optimization step, which fuses the estimates at each position i and simultaneously minimizes the reprojection error of all calibrated images ( Figure 8 ).

[0071] Figure 7The offline calibration method can be performed using frames acquired at a single angular position, in which case N = 1 and the position is the reference position i = 0, or at multiple angular positions, in which case N > 1. For each different position, the frames can be acquired as a single calibration frame (K = 1) or as multiple calibration frames (K > 1).

[0072] The case N = 1 and K = 1 is the case requiring the least user effort, and is particularly suitable for fast calibration in the OR, where the surgeon only has to acquire a single image of the checkerboard pattern after assembling the endoscope into the camera head. The estimation accuracy of the calibration parameters tends to improve with the number of frames K.

[0073] The use of information from two or more angular positions (N > 1) makes it possible to estimate the center of rotation Q in conjunction with f, ξ and O0, independently of the number of frames K acquired at each position. This can be achieved by following the method illustrated in Fig. 4 and disclosed in §§

[0058] -

[0064] . For N > 1, the calibrations obtained at different angular positions are fused in a large-scale optimization step that enforces the rotation model, as Figure 8 illustrated for N = 3 in Fig. 5. It can be observed that, for any two angular positions, the principal point O and the notch P rotate by the same amount around the center of rotation Q. The optimization scheme is used to estimate the distortion and the calibration parameters at the reference position, while minimizing the re-projection error in all the acquired calibration images and enforcing this model for all the sampled angular positions. Figure 8 The expression present in

[0063] provides the mathematical formulation of this optimization scheme for the case of N = 3 angular positions, which directly extends to the general value N. The function r is computed by projecting the points onto the image plane, obtaining and outputting the squared distance where d is the Euclidean distance between the points and K i is the number of calibration images acquired with the scope in the angular position i. As shown in the mathematical expression, the proposed optimization scheme finds the values of the distortion ξ, the center of rotation Q, the intrinsic parameters f and O0, and the calibration object pose that minimizes the sum of the re-projection errors computed for all frames k and angular positions i.

[0074] 2.5 Online update of the calibration parameters

[0075] During operation, each time a new frame j is acquired, the ongoing process must detect and measure the angular shift with respect to the reference position and update the calibration and camera model accordingly. Figure 9 A schematic of this process is provided. The frame j is processed for detecting the circle boundary center Cj and notch P j This can be achieved by following the steps disclosed in Figure 3 and §§

[0050] -

[0057] . Then, an angular shift δ is performed as disclosed in §§

[0058] -

[0064] . j For the estimation, the rotation center Q must be known.

[0076] There are two possible operating modes for retrieving Q: In mode 1, the rotation center is known in advance from an offline calibration step, which uses frames acquired at N > 1 angular positions as disclosed in §§

[0065] -

[0069] ; In mode 2, the rotation center is not known "deductively," but is estimated instantaneously from consecutive frames, for which the center C of the notch P and / or circular boundary is determined so that the methods disclosed in §§

[0058] -

[0064] and Figure 4 can be used. Figure 9 As shown, the current notch P j and center C j Combined with the notch and center detected based on the previous frame j-1 to estimate the rotation center Q, the previous frame being obtained through a delay operation, P j-1 and center C j-1 Accessed. As a final step, the calibration parameters are updated by applying a plane rotation to the principal point O0 corresponding to the reference position, i.e., by calculating the updated principal point as O. j =R(δ) j ,Q)O0.

[0077] 3. Off-site lens calibration to avoid explicit calibration steps in OR (Operational Orthogonal).

[0078] A method has been disclosed for consistently determining the calibration of an endoscopic camera, comprising two steps or stages: an offline step designed to estimate the focal length f, distortion ξ, and principal point O for an arbitrary reference angular position; and an online step that, at each frame time, determines the angular shift relative to the reference and updates the position of the principal point to provide calibration for the current frame. Because the lens of the endoscopic camera is replaceable, both the offline and online steps are performed on-site in the OR after the surgeon has assembled the endoscope into the camera. While the online step is designed to operate instantaneously and in parallel with image acquisition in a seamless manner for the user, the offline step requires explicit user intervention to acquire one or more calibration frames, which is undesirable.

[0079] To minimize disruption to existing surgical procedures, US 9438897 B2 describes a method, which is described in §§

[0065] -

[0069] and Figure 7The specific cases of N=1 and K=1 for the offline steps disclosed in the document. The workload of the surgeon is minimized by requiring a single frame to be acquired at the reference location, and in cases such as... Figure 9 The rotation center Q is determined in the online step of Mode 2. However, the method still requires intervention by a surgeon in the OR, which is still time-consuming and disruptive to the workflow, and requires the use of sterile calibration objects (in this case, checkerboard patterns), which are not always easy to produce and increase costs.

[0080] This patent overcomes these problems by disclosing a method for individually calibrating endoscope lenses, which can be performed off-site (e.g., during manufacturing) using a camera or other components, and provides a set of parameters that adequately describe a rigid endoscope, thereby generating lens descriptors that can be used for different purposes. One objective is to achieve calibration of any endoscopic camera system equipped with a lens, in which case the descriptor It is loaded into the camera control unit (CCU) to be used as input in an online method that runs in real time and outputs a complete calibration of the camera + lens arrangement at each frame.

[0081] Because lens calibration can be performed off-site, i.e., at the factory during manufacturing, and online calibration runs instantly in a seamless manner for the user, the surgeon does not perform any actions during the OR. This means that endoscopic camera calibration can be completed at any time without altering or interfering with established routines. Furthermore, unlike the method disclosed in US 9438897 B2, calibration can be performed even with variable zoom and / or lens translation relative to the camera.

[0082] The method for generating descriptors is disclosed in §§

[0079] -

[0083] . The method for off-site, offline calibration of rigid endoscopes is described in §§

[0084] -

[0085] , while an online method for calibrating an endoscope camera including a camera and optics is described in §§

[0084] -

[0085] .

[0083] 3.1 Offline lens calibration

[0084] Rigid endoscopes are assembled into arbitrary cameras (hereinafter referred to as description cameras), and employing... Figure 7 The offline calibration method described in §§

[0065] -

[0069] requires acquiring K calibration images at N different angular positions. This enables calibration to be performed at a reference angular position i = 0, which includes known internal parameters K(f, O), distortion ξ, notch P, circular boundary Ω with center C and radius r, and rotation center Q known in the case of N > 1.

[0085] The calibration result refers to the composite arrangement of camera and rigid endoscope, where the measurement depends on the specific camera used and the way the lens is mounted in the camera. Since the goal is to describe the endoscope alone, the influence of the camera must be eliminated so that the final descriptor depends only on the lens and remains invariant to the camera and / or device that generated it.

[0086] The method disclosed herein achieves this goal by establishing two key observations: (i) the camera generally follows an orthogonal (or close to orthogonal) projection model, which means that it only contributes to the imaging process and to the magnification and metric-to-pixel conversion; and (ii) the image of the Field Stop Mask (FSM) always has a similarity transformation relationship, which means that the FSM can be used as a reference to encode information about the lens that remains invariant to rigid motion and scaling.

[0087] Offline method Figure 7 The calibration result after the offline method includes the focal length f, the distortion ξ, the principal point O, the notch P, and the circular boundary Ω with center C and radius r. The lens descriptor is where dividing by r serves as a normalization to account for the magnification introduced by the camera, ξ and are as measured in the offline calibration step, since it is an intrinsic property of the optical device that is not affected by the camera, and O is the principal point referenced in the coordinate system attached to the circular boundary (lens coordinate system), with center C and x-axis aligned after scaling by r with the line segment connecting the center C and the notch P Figure 10 For this particular choice of lens reference frame, the coordinate change between image and lens is performed by a similarity transformation A, such that and and β is the angle between the x-axis of the image and the boundary reference frame. If the center of rotation Q is known, it can also be represented in the lens coordinates by stacking and the descriptor becomes These particular choices of image and lens reference frames are arbitrary, and other reference frames that have a rigid transformation relationship with the chosen reference frames can be considered without violating the disclosed method.

[0088] 3.2 Online camera calibration using the lens descriptor

[0089] When a lens with descriptor is mounted on an arbitrary camera (henceforth referred to as the application camera), the calibration of the complete arrangement camera + lens can be obtained automatically by operating as follows: for each frame j, apply the method of Figure 3 to detect the notch P j and the circular boundary Ω with center C j and radius r jthe position of the lens reference frame in the image, and determine a similarity transformation B that maps the lens coordinates to the current image coordinates, where a is the angle between the x-axes of the two reference frames; finally, the endoscope camera calibration for the current angular position can be determined by decoding the different descriptor entries, in which case the focal length becomes the principal point is now and the distortion ξ is constant as it is intrinsic to the optics. If the descriptor also includes the center of rotation, its position in frame j can be determined by making the focal length f

[0090] 3.3 Related Considerations

[0091] Off-line lens calibration using a single image: An important consideration is that the calibration method disclosed in this patent does not require the known center of rotation Q to determine the endoscope camera calibration at each frame instant. Thus, if the time and effort of the off-line calibration process is an issue, a lens descriptor can be generated by taking a single calibration image, in which case, Figure 7 The off-line method of §§

[0065] -

[0069] operates with K = 1 and N = 1. In this case, the descriptor will not contain the entries

[0092] Adjustment of the relative rotation (calibration by detection or by tracking): Since the endoscope rotates δ j with respect to the camera, a similar rotation of the lens reference frame in the image is induced, so the calibration update at each frame j can be performed implicitly without the need to compute the angular shift δ j and explicitly rotate the principal point around the center Q. In this case, the disclosed method based on the lens descriptor can be used separately, where Φ is decoded at each frame instant by the on-line method of §§

[0084] -

[0085] (calibration by detection). An alternative is to employ Figure 9 the method of §§

[0070] -

[0072] , in which case the lens descriptor is used to obtain the calibration at an arbitrary reference position, and this calibration is then updated by determining the angular shift δ j and the rotated principal point (calibration by tracking).

[0093] Adaptation to optical zoom and / or translation of the lens scope along the plane orthogonal to the mechanical axis: In this disclosure, the focal length f is determined at each frame instant by scaling the normalized focal length j by the magnification factor introduced by the camera, which is inferred from the radius r j of the circle boundary. If the magnification factor is constant, then fj It is also constant between consecutive frames j. However, if the optical zoom of the camera is applied, then f... j This will also change accordingly. Therefore, and unlike the method described in US 9438897 B2, the method disclosed herein can handle zoom changes and translations of the lens aperture along a plane orthogonal to the mechanical axis. The adaptation to zoom changes is because: zoom changes cause the radius r of the boundary to... j The change in radius, used for the relevant entry in the lens descriptor (i.e., f) j O i and Q j Decoding is performed to provide the necessary adjustments. The translation adjustment is because the circular boundary with the lens coordinate system translates along with the lens, and O has already been decoded. j and Q j The image coordinates are translated accordingly.

[0094] An alternative method for generating lens descriptors: descriptors Lenses are described by parameters or characteristics with clear physical meaning. For example, the mechanical axis of an endoscope, typically defined by the axis of symmetry of the eyepiece in the proximal end of the lens, should pass through the center of the circle defined by the FSM. If this condition holds, then the center C coincides with the rotation center Q and Generally speaking, such as Figure 11 As shown, due to mechanical tolerances in the manufacturing process, the conditions were not verified; in this case, non-zero... This indicates a misalignment between the eyepiece and the FSM. Since this is a mechanical misalignment, it is possible to generate [the image] using camera calibration and image processing, in addition to those disclosed in §§

[0079] -

[0083] . Other methods besides the standard measurement method. Such alternatives include using calipers, micrometers, protractors, gauges, robotic measuring equipment, or any combination thereof to physically measure the distance between the axis and center of the FSM.

[0095] Lens descriptor transfer to application camera: In the disclosed embodiments, the lens descriptor is generated off-site with the aid of the descriptor camera, and must then be transferred to the online method connected to the application camera to perform §§

[0084] -

[0085] ( Figure 10) or a computer platform. Such transmission or communication can be achieved by a variety of methods, including but not limited to: manual insertion of calibration parameters into the CCU by means of a keyboard or other input interface, download from a remote server, retrieval from a lens descriptor database, reading from a USB flash drive or any other storage medium, visual reading and decoding of a QR code, and visual reading and decoding of information carried in the FSM, such as the numerical or binary code disclosed in PCT / US2018 / 048322.

[0096] Descriptors for a batch of lenses: Descriptors A specific lens can be described or a batch of lenses with similar characteristics can be represented. In the case of a batch of lenses with similar characteristics, the descriptor can be generated by using calibration frames acquired with different lenses of the batch as input of the offline calibration method of Figure 7 , in which case the use of a single descriptor is forced in the final global optimization step; or alternatively, a descriptor is generated for each lens of the batch and the entries of the descriptors are averaged to obtain a single representation. Describing a batch of lenses by a single averaged descriptor can avoid the need to load a specific descriptor for each lens used in a specific application camera, or can be used for quality control at production time, in which case the variation of the parameters in the descriptor is a measure of the repeatability of the manufacturing process.

[0097] 4. Detection of endoscope camera anomalies

[0098] While the calibration method presented in §§

[0041] -

[0072] always provides a correct calibration of the endoscope camera, as it is assembled and explicitly calibrated in the OR, the calibration method disclosed in this patent (§§

[0073] -

[0092] ) relies on previous assumptions, such as the correct retrieval of the stored calibration information and the correct assembly of the lens in the camera head. If these assumptions are not met, the camera + lens arrangement will not be accurately calibrated and malfunctions can occur in systems that use the calibration information for performing distortion correction, virtual view rendering, visualization enhancements, surgical navigation, etc.

[0099] This patent discloses a method that exploits lens descriptors in to detect anomalies in the calibration of an endoscope camera, caused by a mismatch between the loaded calibration information and the lens used and / or by a wrong assembly of the lens in the camera head or by a defect of any of these components.

[0100] Figure 12An illustration of such an anomaly detection method is provided. For each acquired frame j, both the detection of the border and the notch are performed for obtaining an updated calibration from the loaded lens description sub, as described in §§

[0084] -

[0085] , and for estimating the center of rotation, as described in §§

[0058] -

[0064] . This results in two different center of rotation estimates (Q Figure 12 j and ), which can be compared to detect anomalies. The intuition behind this method is the following: due to mechanical tolerances in the construction of the optics, the two lenses will not rotate in exactly the same way. Therefore, the way each lens rotates with respect to any camera can be used as a signature to distinguish it from the other lens. In addition, if a lens is not assembled correctly or damaged due to a defect in the eyepiece connector, a defect in the lens itself or any other aspect leading to a defective fit between the camera and the lens, the way the lens will rotate will also be different from the way it would rotate if it was assembled correctly.

[0101] This variation in the lens motion model can be used to detect the presence of an anomaly, as well as to quantify the severity of the anomaly and to warn the user that the assembly needs to be verified and / or the lens replaced.

[0102] This method only provides information about the presence of an anomaly and does not specify the type of anomaly that is occurring, which would allow the system to provide user-specific instructions for fixing the anomaly. To achieve this, the anomaly detection method shown in Figure 12 can be complemented with another method for representing the cause of the anomaly. Since an incorrect assembly of the lens in the camera causes the projection of the FSM in the image plane to change, features quantifying the difference between the borders detected at calibration and operation time can be used to distinguish anomalies caused by a mismatch or defective assembly of the calibration optics.

[0103] In particular, if the FSM projects onto a circle in the image plane when the lens is properly assembled in the camera, this circle will tend to evolve into an ellipse when the optics are not assembled correctly. Therefore, in this case, the eccentricity of the border detected during operation can be measured to verify if the assembly is correct and it is not necessary to know the specific shape of the border detected during calibration of the lens.

[0104] This method is valid if the FSM has a shape that can be represented parametrically, such as an ellipse or any other geometric shape. In addition, template matching or machine learning techniques can be used to compare the border detected during operation with known shapes.

[0105] In summary, there are two important features that can be used to detect and identify anomalies. The first feature is Q j ​the difference between the center of rotation estimates obtained at calibration time and during operation. The second feature is the difference between the border profiles detected at calibration time and during operation. While the first feature allows to detect anomalies, whether a mismatch between the loaded calibration and the used camera+lens arrangement, a wrong assembly of the lens in the camera or a defect of any of these components, the second feature provides information about the type of anomaly as it occurs only when there is a defective assembly.

[0106] Therefore, the disclosed method for detecting and identifying anomalies exploiting these two different features can be implemented using a cascade classifier that first uses the first feature for an anomaly detection phase and then the second feature to discriminate between calibration mismatch and incorrect camera+ lens assembly. Besides cascade classifiers, other methods can be used, such as different types of classifiers, machine learning, statistical methods, data mining. In addition, depending on the required application, these features can be used separately, in which case the first feature will allow to detect anomalies without the need to identify the anomaly type and the second feature will be used only to detect incorrect assembly.

[0107] Figure 13 FIG. 1 is a diagrammatic representation of a general purpose computing system environment 1200 within which the illustrative embodiments can be implemented, according to an embodiment. While only one computing system 1200 is depicted in FIG. 1, those skilled in the art will recognize that an operating environment of various tasks described infra can be practiced in a distributed computing environment having multiple computing systems 1200 linked through a network, wherein the executable instructions can be associated with one or more of the multiple computing systems 1200 and / or executed by them. The computing system environment 1200 or portions thereof can be used for the processes, methods, and computing steps of the present disclosure.

[0108] In its most basic configuration, computing system environment 1200 typically includes at least one processing unit 1202 and at least one memory 1204 linked via a bus 1206. Depending on the exact configuration and type of computing system environment, the memory 1204 can be volatile (such as RAM 1210), non-volatile (such as ROM 1208, flash memory, etc.) or some combination of the two. The computing system environment 1200 can have additional features and / or functionality. For example, the computing system environment 1200 can also include additional storage such as removable and / or non-removable storage including, but not limited to, magnetic or optical disks, tape drives, and / or flash storage. The computing system environment 1200 can access such additional storage through, for example, a hard disk drive interface 1212, a disk drive interface 1214, and / or an optical disk drive interface 1216. As will be appreciated, these devices linked to the system bus 1206 respectively, allow for reading from and writing to a hard disk 1218, reading from or writing to a removable disk 1220, and / or reading from or writing to a removable optical disk 1222 such as a CD / DVD ROM or other optical media. The drive interfaces and their associated computer-readable media allow for the nonvolatile storage of computer-readable instructions, data structures, program modules, and other data for the computing system environment 1200. Those skilled in the art will further appreciate that other types of computer- readable media that can store data can be used for this same purpose. Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories, nanotechnological storage, memory sticks, other read / write and / or read-only memories, and / or any other physical medium or storage known in the art for storing information such as computer-readable instructions, data structures, program modules or other data. Any such computer storage media can be part of the computing system environment 1200.

[0109] A number of program modules can be stored in one or more of the memory / media devices. For example, a basic input / output system (BIOS) 1224 containing the basic routines that help to transfer information between elements within the computing system environment 1200, such as during startup, can be stored in the ROM 1208. Similarly, RAM 1210, the hard disk 1218, and / or the peripheral memory devices can be used to store computer-executable instructions including an operating system 1226, one or more application programs 1228 (such as an application that performs the methods and processes of the present disclosure), other program modules 1230, and / or program data 1232. Still further, computer-executable instructions can be downloaded to the computing environment 1200 from, for example, a network connection.

[0110] Final users such as clients, retail clerks, etc. can enter commands and information into the computing system environment 1200 through input devices such as a keyboard 1234 and / or pointing device 1236. Other input devices not shown, can include a microphone, joystick, game pad, scanner, or the like. These and other input devices are often connected to the processing unit 1202 through a peripheral device interface 1238 that is, in turn, coupled to bus 1206. The input devices can be connected by, for example, a parallel port, game port, a firewire, or a universal serial bus (USB) interface. To view information from the computing system environment 1200, a display 1240 or other type of display device can also be connected to bus 1206 via an interface, such as a video adapter 1242. In addition to the display 1240, the computing system environment 1200 can include other peripheral devices not shown, such as speakers and a printer.

[0111] The computing system environment 1200 can also utilize logical connections to one or more computing system environments. Communications between the computing system environment 1200 and a remote computing system environment can be exchanged through another processing device, such as a network router 1252 that is responsible for network routing. Communications with the network router 1252 can be performed via a network interface component 1254. Thus, in such networking environments, it should be appreciated that the program modules depicted relative to the computing system environment 1200 or portions thereof can be stored in the memory storage devices of the computing system environment 1200.

[0112] The computing system environment 1200 can also include positioning hardware 1256 for determining a location of the computing system environment 1200. In embodiments, the positioning hardware 1256 can include only a GPS antenna, an RFID chip or reader, a Wi-Fi antenna, or other computing hardware that can be used to capture or transmit signals that can be used to determine a location of the computing system environment 1200.

[0113] In a first aspect of the disclosure, a method for calibrating an endoscope camera is provided. The endoscope camera results from combining a rigid endoscope with a camera, where the rigid endoscope or lens scope has a field stop mask (FSM) reproducing an image boundary with a center C and a notch P, and can be rotated by an angle δ with respect to the camera around a mechanical axis intersecting the image plane at a point Q, in which case C, P and the principal point O undergo a 2D rotation by the same angle δ around Q, and where the calibration comprises determining the focal length f, the distortion ξ, the rotation center Q and the principal point O0 for a selected angular position of the lens scope with respect to the camera, hereafter referred to as the reference angular position i = 0. The method comprises: acquiring one or more calibration images of a calibration object at an angular position i with the endoscope camera without rotating the lens with respect to the camera; determining first estimates of the calibration parameters f, ξ and O i of the endoscope camera, and the 3D pose (rotation and translation) of the calibration object with respect to the camera for each calibration image; detecting the boundary with center C i and notch P i on the calibration images using an image processing method; rotating the lens scope with respect to the camera to a new angular position i and repeating the previous steps, where i is incremented, taking consecutive values i = 0, 1,..., N-1, where N > 1 is the number of different angular positions used for the calibration; determining first estimates of the rotation center Q and the angular shift δ i between the reference position i = 0 and the consecutive calibration positions i = 1,..., N-1; and refining the calibration parameters f, ξ, Q and O0 by a final optimization step that enforces the model of the principal point, the boundary center and the notch to rotate by the angle δ i around the center Q at the consecutive calibration positions i = 0,..., N-1.

[0114] In embodiments of the first aspect, the calibration object is a 2D plane with a checkerboard pattern or any other known pattern, a known 3D object, or is absent, and the calibration input is a set of point correspondences between images, in which case the first estimates of the calibration parameters are obtained by a camera calibration algorithm from a plane, a camera calibration algorithm from an object, or a suitable automatic calibration technique, respectively.

[0115] In embodiments of the first aspect, the final optimization step is performed using an iterative non-linear minimization of a re-projection error, a photo-geometry error or any other suitable optimization method.

[0116] In embodiments of the first aspect, the first estimates of the calibration parameters are determined according to any calibration method in the literature.

[0117] In an embodiment of the first aspect, the first estimate of the calibration parameters comprises any distortion model known in the literature, such as the Brown polynomial model, the rational model, the fisheye model or a split model with one or more parameters, in which case ξ is a scalar or a vector, respectively.

[0118] In an embodiment of the first aspect, the center of rotation Q is known a priori, in which case the calibration can be done from images acquired in one or more angular positions (N >= 1); said center of rotation being determined from the image positions of the border center C i and the notch P i , in which case the calibration is done from images acquired in two or more angular positions (N >= 2); or said center of rotation being determined from the image positions of the border center C i or the notch P i only, in which case the calibration is done from images acquired in three or more angular positions (N >= 3).

[0119] In a second aspect of the disclosure, a method is provided for updating the calibration parameters of an endoscopic camera at each frame time. The endoscopic camera results from the combination of a rigid endoscope and a camera, said camera comprising a camera head and a camera control unit (CCU), wherein the rigid endoscope or the lens scope has a field stop mask (FSM) reproducing an image border with a center C and a notch P, and can be rotated by an angle δ around a mechanical axis intersecting the image plane at a point Q, in which case C, P and the principal point O undergo the same 2D rotation by angle δ around Q, and wherein for a reference angular position i = 0 of the lens scope relative to the camera head, the calibration parameters focal length f, distortion ξ, center of rotation Q and principal point Oo and the border with center Co and notch Po are known. The method comprises: acquiring a new frame j with the endoscopic camera and detecting the border center C j and the notch P j ; estimating the angular shift δ of the endoscopic lens relative to the camera head from the notch Po, the notch P j and Q; and estimating the updated principal point O j of the endoscopic camera by performing a 2D rotation of the principal point Oo around Q by angle δ.

[0120] In an embodiment of the second aspect, the calibration parameters focal length f, distortion ξ, center of rotation Q and principal point Oo and the border with center Co and notch Po at the reference angular position i = 0 are obtained by calibrating the endoscopic camera at the reference position with the lens scope or by retrieving from the CCU.

[0121] In an embodiment of the second aspect, two or more border centers C j and / or notches P jTo determine the center of rotation Q.

[0122] In the second aspect of the embodiment, the angular displacement of the endoscope lens is estimated by mechanical components and / or by utilizing optical tracking, in which case it is not necessary to know the boundary centers C0 and C2. j and notches P0 and P j .

[0123] In an embodiment of the second aspect, the method further includes employing techniques for filtering the estimates of the rotation center Q and the angular shift δ, including, but not limited to, any recursive or time-based filters known in the literature, such as a Kalman filter or an extended Kalman filter.

[0124] In a third aspect of this disclosure, a method for obtaining a normalized focal length is provided. Distortion ξ, Normalized Principal Point and normalized rotation center descriptor A method for characterizing a rigid endoscope using a field-resistance mask (FSM) that induces an image boundary having a center C and a notch P, the method comprising: combining a rigid endoscope with a camera to obtain an endoscope camera (referred to as a characterizing camera); estimating calibration parameters (focal length f, distortion ξ, principal point O0, and rotation center Q) of the characterizing camera at a reference location; detecting a boundary having a center C0 and a notch P0 at the reference location; and determining a normalized focal length based on the center C0, the notch P0, the focal length f, the principal point O0, and the rotation center Q. Normalized principal point and normalized rotation center

[0125] In the third aspect of the embodiment, the normalized focal length From It is calculated, where r is the center C0 = [C x C y ,1] T The distance between the notch P0 and the normalized principal point. and the center of rotation It is calculated separately and And what was obtained, among which And β are line segments The angle between the downward direction and the downward direction.

[0126] In a fourth aspect of this disclosure, a method for calibrating an endoscope camera is provided. The endoscope camera is generated by combining a rigid endoscope with a camera including a camera and a camera control unit (CCU), wherein the rigid endoscope has a normalized focal length. Distortion ξ, Normalized Principal Point and normalized center of rotation descriptor and having a field stop mask (FSM) reproducing the image border with center C and notch P, and wherein the calibration comprises determining the focal length f, the distortion ξ, the center of rotation Q and the principal point O for a specific angular position of the lens scope relative to the camera. The method comprises: acquiring a frame i by the endoscopic camera, detecting the border center C i = [C x , C y , 1] T and the notch P i , and determining the radius and estimating the calibration parameters focal length f, center of rotation Q and principal point O of the endoscopic camera from the center C i , the notch P i , the normalized focal length the normalized principal point and the normalized center of rotation .

[0127] In embodiments rooted in the fourth aspect, the focal length f, the principal point O and the center of rotation Q are calculated by and respectively, wherein and a is the angle between the segment and the downward direction.

[0128] In embodiments of the fourth aspect, the endoscopic lens descriptor is obtained by measuring the endoscopic lens using the camera, using a caliper, a micrometer, a protractor, a gauge, a robotic measuring device or any combination thereof or by using a CAD model of the endoscopic lens.

[0129] In embodiments of the fourth aspect, the endoscopic lens descriptor is obtained by loading information from a database into the CCU or using a QR code, a USB flash drive, manual insertion, a pattern in the FSM, an RFID tag, an internet connection, etc.

[0130] In embodiments of the fourth aspect, the frame i comprises two or more frames, wherein two or more border centers C i and / or notches P i are used to determine the center of rotation Q.

[0131] In embodiments of the fourth aspect, the endoscopic camera can have an arbitrary angular position of the lens scope relative to the camera and an arbitrary zoom amount.

[0132] In a fifth aspect of the disclosure, there is provided a method for detecting anomalies caused by a defect or incorrect assembly of a rigid endoscope in a camera, or by a mismatch between a considered calibration and an endoscope lens used in an endoscope camera resulting from a combination of a rigid endoscope with an endoscope camera comprising a camera and a camera control unit (CCU), wherein the rigid endoscope has a descriptor of a normalized center of rotation and has a field stop mask (FSM) reproducing an image border with a center C and a notch P, the method comprising: acquiring at least two frames by the endoscope camera, having the rigid endoscope in different positions with respect to the camera, and detecting the border center C i and the notch P i of each frame; estimating the center of rotation i from the detected border center C i or notch P the border center C and the notch P i of the normalized center of rotation i ; and comparing the two centers of rotation and Q and determining the presence of an anomaly.

[0133] In embodiments of the fifth aspect, the endoscope lens descriptor is obtained by loading information from a database into the CCU or using QR codes, USB flash drives, manual insertion, patterns in the FSM, RFID tags, internet connection, etc.

[0134] In embodiments of the fifth aspect, the two centers of rotation and Q and the determination about the presence of an anomaly are performed by using one or more of an algebraic function, a classification scheme, a statistical model, a machine learning algorithm, a threshold or data mining.

[0135] In embodiments of the fifth aspect, the borders detected during calibration time and during operation are compared to identify the cause of the anomaly.

[0136] In embodiments of the fifth aspect, the method further comprises providing an alert message to a user, wherein the cause of the anomaly is identified, whether it is a mismatch between the lens used and the considered calibration, or a physical problem of the rigid endoscope and / or the camera.

[0137] ​In a sixth aspect of the disclosure, there is provided a method for detecting an image boundary having a center C and a notch P in a frame acquired by using a rigid endoscope having a field stop mask (FSM) that causes the image boundary having a center C and a notch P, the method comprising: using an initial estimate of the boundary having a center C and a notch P to render a ring image, the ring image being obtained by interpolating and connecting image signals extracted from the acquired frame at concentric circles centered at C with the notch P mapped to the center of the ring image; detecting salient points in the ring image; repeating the following steps until the detected salient points are colinear; mapping the salient points into the space of the acquired frame and fitting a circle having a center C to the mapped points; rendering a new ring image by using the fitted circle; detecting salient points in the new ring image; and detecting the notch P in the final ring image using correlation with a known template.

[0138] In embodiments of the sixth aspect, the FSM contains more than one notch, all notches having different shapes and / or sizes so as to be identifiable, in which case the template comprises a combination of notches whose relative positions are known.

[0139] In embodiments of the sixth aspect, the notches can have any desired shape.

[0140] In embodiments of the sixth aspect, the notch mapped in the center of the ring image is an arbitrary notch.

[0141] In embodiments of the sixth aspect, the initial estimate of the boundary having a center C and a notch P can be obtained from a variety of methods including but not limited to deep / machine learning, image processing, statistics-based methods, and stochastic methods.

[0142] In embodiments of the sixth aspect, a general conic is fitted to the mapped points.

[0143] In embodiments of the sixth aspect, the detected center C and / or notch P are used to estimate the angular shift of the rigid endoscope relative to the camera head on which it is mounted.

[0144] While various embodiments have been described for purposes of this disclosure, such embodiments should not be construed as limiting the teachings of this disclosure to those embodiments. Various changes and modifications can be made to the described embodiments with respect to the elements and operations described herein, without departing from the scope of the systems and processes described in this disclosure. All patents, patent applications, and published references cited herein are hereby incorporated by reference in their entirety. It is to be appreciated that certain features and subcombinations are of utility and can be employed in various

[0145] The described embodiments are to be considered in all respects only as illustrative and not restrictive, and the scope of embodiments of this disclosure is indicated by the appended claims rather than by the foregoing description. All changes and modifications that come within the meaning of the equivalent of the claims are to be embraced by the claims. Those skilled in the art can implement the described functionality in varying ways for each particular application, but such implementation decisions do not cause a departure from the scope of the disclosed systems and / or methods.

[0146] References

[0147] All cited references are expressly incorporated herein by reference. Discussion of any reference is intended merely to provide a general description of such a reference and does not constitute an admission that the reference is prior art. In particular, citation of any reference prior to its priority date does not constitute admission that the reference is prior art to this application.

[0148] [1] T. Yamaguchi, M. Nakamoto, Y. Sato, K. Konishi, M. Hashizume, N. Sugano, H. Yoshikawa, and S. Tamura, "Development of a camera model and calibration procedure for oblique-viewing endoscopes," Computer Assisted Surgery, vol. 9, no. 5, pp. 203-214, Feb. 2004.

[0149] [2] C. Wu, B. Jaramaz, and S. Narasimhan, "A full geometric and photometric calibration method for oblique-viewing endoscopes," Computer Aided Surgery, vol. 15, no. 1-3, pp. 19-31, Apr. 2010.

Claims

1. A method for calibrating an endoscope camera, the endoscope camera comprising a rigid endoscope and a camera, the camera comprising a camera head, the rigid endoscope comprising a lens scope, wherein the rigid endoscope or the lens scope has a field stop mask (FSM) that reproduces an image boundary having a center C and a notch P, and is rotatable about a mechanical axis that intersects the image plane at a point Q by an angle δ relative to the camera head, wherein C, P, and a principal point O undergo a 2D rotation by the same angle δ about Q, the method comprising: acquiring one or more first calibration images of a calibration object at a reference angular position with the endoscope camera; determining, for each first calibration image, first estimates of a focal length f, a distortion ξ, a center of rotation Q, and a principal point O0 of the endoscope camera relative to the camera; According to the image processing method, a border of the calibration image is detected i and the notch P i . performing an iterative process comprising iteratively performing one or more times: acquiring one or more further calibration images of the calibration object at a further angular position of the lens scope relative to the camera head with the endoscope camera; determining an estimate of an angular shift between the reference angular position and the further angular position; determining, for each further calibration image of the iteration, further estimates of the focal length f, the distortion ξ, the center of rotation Q, and the principal point O0 of the endoscope camera relative to the camera; and According to the image processing method, a center C i and a notch P i of the other calibration image of the iteration are detected; and improving f, ξ, and O0 from the further estimates of the focal length f, the distortion ξ, the center of rotation Q, and the principal point O0 and from the estimated angular shift.

2. The method of claim 1, further comprising: determining, for each first calibration image, a first estimate of a 3D pose of the calibration object, the 3D pose comprising a rotation and a translation; wherein the iterative process further comprises: determining, for each further calibration image of the iteration, a further estimate of a 3D pose of the calibration object, the 3D pose comprising a rotation and a translation; wherein f, ξ, and O0 are further improved from the further estimate of the 3D pose of the calibration object.

3. The method of claim 1, wherein the calibration object comprises one or more of: a 2D plane having a checkerboard pattern; a 2D plane having a known pattern; or a known 3D object.

4. The method of claim 1, wherein improving f, ξ, and O0 comprises one or more of: (a) an iterative nonlinear minimization of re-projection errors; (b) an iterative nonlinear minimization of photogrammetric geometric errors; or (c) a cost function minimization that is not (a) and not (b).

5. The method of claim 1, wherein the first estimates of a focal length f, a distortion ξ, a center of rotation Q, and a principal point O0 comprise a distortion model comprising one or more of: a Brown polynomial model; a rational model; a fisheye model; or a piecewise model having one or more parameters.

Images