SYSTEM AND METHOD FOR ENDOSCOPIC CHARACTERIZATION AND AUTOMATED CALIBRATION OF ENDOSCOPIC CAMERA SYSTEMS - Patent application

Abstract

A system and method for determining the calibration of an endoscopic camera comprising a camera head with interchangeable and rotatable optical components (rigid endoscope), which first comprises determining a set of parameters [Equation 1] Characterize the rigid endoscope through JPEG2023504344000113.jpg168170, then use this lens descriptor [Equation 2] A system and method is disclosed that is accomplished without user intervention in the operating room by using JPEG2023504344000114.jpg142170 as input for real-time software that processes images acquired by any camera head with a rigid endoscope, providing calibration of the entire endoscopic camera arrangement on a frame-by-frame basis, regardless of the relative rotation of the lens scope to the camera head or zoom setting. Also disclosed is an image software method that detects if a lens descriptor is incompatible with the rigid endoscope in use to prevent errors and alert the user to faulty conditions.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 911,950, filed October 7, 2019 (the "'950 Application"), and U.S. Provisional Patent Application No. 62 / 911,986, filed October 7, 2019 (the "'986 Application"), both of which are incorporated by reference in their entireties for all purposes.

[0002] The present disclosure relates generally to the field of computer vision and photogrammetry, and more particularly, but not exclusively, to embodiments of the present disclosure used in the context of surgical and diagnostic clinical procedures for the purpose of calibrating endoscopic camera systems with interchangeable and rotatable optical components (rigid endoscopes), identifying whether a particular endoscope is in use, or verifying that it is correctly assembled to the camera head. These endoscopic systems are used in several medical disciplines, such as orthopedics (arthroscopy) or abdominal surgery (lapascopy), where camera calibration allows for applications in computer-assisted surgery (CAS) and enhanced visualization. Summary of the Invention [Problem to be solved by the invention]

[0003] Video-guided procedures such as arthroscopy and laparoscopy offer surgeons the possibility to visualize the interior of anatomical cavities of interest utilizing a video camera with a rigid endoscope. Depending on the medical specialty, the rigid endoscope, which may be an arthroscope, laparoscope, neuroscope, etc., is combined with a camera, including a camera head and a camera control unit (CCU), to form an endoscopic camera. These cameras differ from traditional cameras primarily due to two features. The first feature is that the rigid endoscope, also called the lens scope or optics, is usually replaceable for easy sterilization and is attached to the camera head by the surgeon in the operating room (OR) before the endoscope begins the medical procedure. This attachment is accomplished by a connector that allows the endoscopic lens to rotate relative to the camera head about its axis of symmetry (the mechanical axis in Figure 2), allowing the surgeon to change the direction of the field of view without the need to translate the endoscopic camera. The second unique feature of endoscopic cameras is that they usually contain a field stop mask (FSM) somewhere along the image transfer system, which causes the acquired image to have meaningful content in the circular area surrounded by the black frame. The FSM usually contains a mark on the circular border to allow the surgeon to infer the downward direction. This mark on the periphery of the circular image will henceforth be referred to as the notch (FIG. 2).

[0004] A key enabling step for computer-aided arthroscopy or laparoscopy is camera calibration, whereby 2D image information can be correlated with a 3D scene for the purposes of enhanced visualization, improved perception, measurement and / or navigation. Calibration of a camera system, in this case with a camera head with a lens scope, consists of mapping 3D projection rays to points in pixel coordinates in the image and determining the parameters of the projection model inversely (Figure 1). In the context of medical endoscopy, the applications of calibration are vast and range from correction of distortions and rendering of virtual views for enhanced visualization to surgical navigation, where the camera is used to measure 3D points and distances and the related information is typically overlaid with the patient's anatomy.

[0005] In endoscopic cameras with rotatable optical components, the motion between the rigid endoscope and the camera sensor causes changes in the calibration parameters of the camera system, which means that the projection model is not constant along time as it happens in conventional cameras. Since it is not practical to perform an independent calibration for every possible position of the optical components with respect to the camera head, the calibration parameters must be updated according to a camera model that takes into account this relative motion. Solutions to determine this relative rotation and update the calibration accordingly have been proposed in the literature, with examples including the use of a rotary encoder attached to the camera head [1], or the employment of an optical tracking system to determine the position of an optical marker attached to the scope cylinder [2]. These approaches present serious drawbacks, which are the need for additional equipment and instruments, which are costly, occupy space in the OR, and disrupt established surgical workflows.

[0006] US Patent No. 9,438,897 discloses a method that solves some of the above-mentioned problems for achieving endoscopic camera calibration without requiring any additional equipment. The rotation of the lens scope is estimated at each frame time using image processing, and the results are used as input to a model that, given a camera calibration at a particular angular position of the lens scope relative to the camera head (reference position), outputs the calibration at the current angular position. However, this method presents the following drawbacks: (i) calibration at the reference position requires acquisition of one or more frames of a known grid pattern (calibration grid) in the OR, which is a time-consuming process that not only requires user intervention but also typically must be performed with a sterile grid, and is therefore undesirable and should be avoided, (ii) the disclosed method does not allow for changes in optical zoom during operation, since the calibration parameters cannot be updated to different zoom levels, and (iii) the lens scope only needs to rotate, never translate, relative to the camera head with the points of intersection of the image and mechanical axes needing to be explicitly determined.

[0007] The embodiments of the present disclosure refer to a method to avoid the need to calibrate an endoscopic camera at a specific angular position in the OR after assembling the rigid scope to the camera head. This patent discloses models, methods and apparatus for characterizing a rigid endoscope in a manner that allows the calibration of any endoscopic camera system with an endoscope to be determined independent of the camera head being used, the amount of zoom introduced by that camera head, and the relative rotation or translation between the scope and the camera head at a specific frame time. This allows the surgeon to change the endoscope and / or camera head during a surgical procedure and adjust the zoom as desired for better visualization of specific image content without causing any disruption to the workflow of the procedure.

[0008] This disclosure shows how to calibrate only the rigid endoscope to obtain a set of parameters, i.e., lens descriptors, that fully characterize the optics. The lens calibration is performed in advance (e.g., at the time of manufacture), and the descriptors are then read into a camera control unit (CCU) that is used as input for real-time software that automatically provides calibration of the entire endoscopic camera arrangement, including both the camera head and the lens, at every frame time, regardless of the relative rotation between the two components. This is accomplished in a seamless manner for the user.

[0009] This descriptor may also be used for identification and quality control purposes to characterize a lens or a batch of lenses. Thus, building on top of this functionality, a method is also disclosed for detecting discrepancies between the lens descriptor loaded into the CCU and the actual rigid endoscope assembled into the camera head. This method is useful for alerting the user if the lens being used is not the correct one and / or if it is damaged or not properly assembled into the camera head.

[0010] The present disclosure may be used in particular, but not by way of limitation, in conjunction with the methods disclosed in U.S. Pat. No. 9,438,897 to correct radial distortion of images and enhance visual perception, or in conjunction with the methods disclosed in U.S. Patent Application Publication No. 2018 / 0071032(A1) to provide guidance and navigation during arthroscopy, in order to achieve camera calibration at frame time, a requirement for these methods to function.

[0011] A system, method, and apparatus for determining the calibration of an endoscopic camera comprising a camera head with interchangeable and rotatable optical components (rigid endoscope) so that 2D image points can be related to 3D projection rays for computer-assisted surgical applications, wherein the rigid endoscope is characterized in advance (e.g., in the factory at the time of manufacture) to achieve camera calibration without requiring any user intervention in the operating room (OR).

[0012] A set of parameters

number

[0013] The image software method includes a lens descriptor

number

[0014] For a more complete understanding of the present disclosure, reference is made to the following detailed description of exemplary embodiments taken in conjunction with the accompanying drawings.

[0015] [Figure 1] 1 is an embodiment of a camera projection model illustrating the mapping of a projection ray, defined by a 3D point X in the scene and the projection center of the camera, onto a 2D point x in pixel coordinates of the image plane. The model can be seen as a composition of intrinsic parameters of the camera K(f,O) that depend on the focal length f and the principal point O at which the optical axis is projected onto the image, a projection P through a pinhole model that maps the 3D point X onto the 2D, a function Γ that accounts for the nonlinear effect of radial distortion introduced by the optics, quantified by ξ. [Diagram 2]FIG. 2 shows an exemplary embodiment of an image formed by an endoscopic camera with interchangeable and rotatable optical components (rigid endoscope). The lens is cylindrical and rotates about its longitudinal axis of symmetry, hereafter called the mechanical axis, which intersects the image plane at the center of rotation Q. The rigid endoscope has a field stop mask (FSM) somewhere along the image transfer system, which is projected onto the image plane as a black frame centered on a circle Ω (circular boundary) with center C, which contains significant visual content (see FIG. 3 for illustration). The image boundary typically has a marking of an image point P, hereafter called a notch, that allows the surgeon to infer the rotation between the scope and the camera head. If the scope rotates about the mechanical axis by an angle δ relative to the camera head, the circle center C, the notch P, and the principal point O also rotate by that same amount δ about Q. [Figure 3A] FIG. 3A shows the sequence of steps for detecting the circular boundary C and notch P at each frame time. Starting with an initial estimate for the boundary Ω and notch P, the algorithm involves rendering a ring image with P mapped to its center (step 1), detecting edge points on that ring image (step 2), mapping the edge points back to image space for estimation of the boundary circle proposal (step 3) and iterating to render a new ring image for detecting more accurate edge points (step 4) until a stopping criterion is met. The cycle stops when the edge points in the ring image become collinear. The final step consists of detecting the notch P in the ring image using a correlation-based strategy with a known notch template. The detected boundary Ω and notch P will be used as the initial values ​​in subsequent frame times. [Figure 3B]Figure 3B illustrates the benefit of detecting multiple edge points for each ring image column. In the case of strong light variance (dashed ovals), the edge point corresponding to the first local maximum along each column of the ring image (red points) does not necessarily correspond to the correct image boundary, causing the circular boundary estimation to fail (yellow circles). This is resolved by detecting multiple local maxima for each column of the ring image and using these edge points as input to a robust circle estimation scheme (green circles). [Figure 3C] FIG. 3C illustrates a strategy used to solve the problem of lens notch regions producing discontinuous ring images: (Step 1) A fixed angular position (arrow) of the ring image relative to the viewpoint may cause the notch P to be partially cut off. (Step 2) The problem in step 1 may be solved by using an initial estimate of the lens notch P to center the ring image. [Figure 3D] Figure 3D shows the strategy for obtaining the lens-specific notch template. During calibration and after generating the ring image (step 1), a specific region is extracted (dashed rectangle in step 2) centered on the lens notch location P. Then, an adaptive binarization strategy is employed to generate a notch template consisting of a light triangle with a dark rectangular background (step 3). [Figure 4A] FIG. 4A illustrates the estimation of the rotation center Q from two frames in which notches Pi and Pj are detected. Q is determined by the line segment

number

number

number

number

number

number

number

[0016] Although exemplary implementations of one or more embodiments are provided below, various specific embodiments may be implemented using any number of techniques known by those skilled in the art. The present disclosure should in no way be limited to the exemplary embodiments, drawings, and / or techniques illustrated below, including the exemplary designs and implementations illustrated and described herein. Furthermore, the present disclosure may be modified within the full scope of the appended claims and their equivalents.

[0017] In this patent, 2D and 3D vectors are written in bold lowercase and uppercase letters, respectively. Functions are represented by lowercase italics, and angles are represented by lowercase Greek letters. Points and other geometric entities in the plane are represented in homogeneous coordinates, as is commonly done in projective geometry, and 2D linear transformations of the plane are represented by 3x3 matrices, with equivalence up to scale. Additionally, throughout the text, different sections are referenced by their paragraph number using the symbol §.

[0018] 1. Camera model for endoscope systems with interchangeable and rotatable optical components (rigid endoscope) Camera calibration is the process of determining a camera model that projects a 3D point X in the camera reference frame to a 2D image point x in pixel coordinates. Alternatively, the camera model can be interpreted as a function that backprojects an image point x onto a ray that passes through a 3D point X in the scene. This process is illustrated in Figure 1. Camera calibration is a key component in many applications ranging from visual odometry to 3D reconstruction, but also the removal of image artifacts for enhanced visual perception, such as radial distortion.

[0019] Conventional, commonly used cameras are described by the so-called pinhole model, which can be extended with a radial distortion model that accounts for nonlinear effects introduced by small optics and / or fisheye lenses. In this case, a point X in the scene is represented by the equation x=KΓ ξ (PX) onto a point x in the image, where x and X are expressed in homogeneous coordinates whose equivalence depends on scale, and P = [I0 3x1 ] is a 3x4 projection matrix with I denoting the 3x3 identity matrix, K is a so-called matrix of intrinsic parameters with dimensions 3x3, and Γ denotes the distortion function with parameters ξ. Henceforth, and without loss of generality, it will be assumed that the camera is skewless with a single aspect ratio, resulting in a model that closely approximates the majority of modern cameras, where deviations of the skew from zero and the aspect ratio from one can be neglected. With this assumption, K is the 3x4 projection matrix with I denoting the 3x3 identity matrix, K is the so-called matrix of intrinsic parameters with dimensions 3x3, and Γ denotes the distortion function with parameters ξ. x , O y , 1] T depends only on the image coordinates of

number

[0020] The distortion function Γ represents the mapping in 2D and can be any of the many distortion functions or models available in the literature, including but not limited to the polynomial model (also known as Brown's model), the divide model, the rational model, the fisheye lens model, etc., either in their first order or higher order (multi-parameter) versions, and ξ is a scalar or a vector, respectively.

[0021] The endoscopic camera resulting from combining a rigid endoscope with a camera has replaceable optical components for easy sterilization purposes, and the endoscope has at its proximal end an ocular lens (or eyepiece) that is typically attached to the camera using a connector that allows the surgeon to rotate the scope relative to the camera head. As illustrated in FIG. 2, this rotation is performed about the longitudinal axis (mechanical axis) of the endoscope, which intersects the image plane at point Q. The field stop mask (FSM) of the lens scope is projected onto the image plane as a black frame around an area with visual content with a circular boundary Ω that includes a center C and a notch P.

[0022] The mechanical axis is approximately coincident with the axis of symmetry of the eyepiece, which is not necessarily aligned with the axis of symmetry of the cylindrical scope and / or does not necessarily pass through the center of the circular area defined by the FSM. These alignments are targeted but never perfectly achieved due to mechanical tolerances in the construction and manufacture of the endoscope. Thus, the center of rotation Q, center of the circular boundary C, and principal point O are generally distinct points in the image, which complicates camera modeling, but may be used as signatures to identify a particular endoscope or a batch of similar endoscopes, as disclosed above.

[0023] Note that an endoscopic camera is calibrated for a particular position of the scope such that K(f,O) is the matrix of intrinsic parameters and ξ are distortion parameters that quantify the radial distortion according to a selected model Γ (Figure 1). When the scope undergoes a rotation by an angle δ relative to the camera head, the distortion ξ and focal length f remain unchanged, but the principal point O rotates by the same amount around Q (Figure 2). This means that the matrix of intrinsic parameters is K(f,R(δ,Q)O), where R(δ,Q) is the degree of change in the orientation of the point Q=[Q x , Q y , 1] T is a 3x3 matrix representing a 2D rotation of the image around

[0024] Similar to causing a rotation about principal point O such that O'=R(δ,Q)O, rotation of the scope relative to the camera head causes a circle Ω with center C and notch P to become a circle Ω' with center C'=R(δ,Q)C and notch P'=R(δ,Q)P (Figure 2).

[0025] 2. Calibrating an Endoscopic Camera with Interchangeable and Rotatable Optical Components In summary, to always have the correct calibration parameters of the endoscopic camera, the focal length f, the distortion ξ, and the principal point O must be known for a specific rotation angle (reference angular position) between the camera head and the lens scope, and the location of that principal point must be updated during operation according to O' = R(δ, Q)O, which requires knowing at every frame time instant the center of rotation Q and the angular displacement between the current angular position and the reference angular position.

[0026] Calibration of the endoscopic camera at a reference angular position, which can be easily recognized by the position P of the notch, can be performed "off-line" before starting the clinical procedure by following the steps of Fig. 7. Determination of f, ξ, and O requires the use of an intrinsic camera calibration method (module A of Fig. 7), which receives as input one or more frames acquired at the reference position P (or P0), as described in §§

[0045] to

[0049] . If the goal is also to determine the center of rotation Q, the input frames are then calibrated at an additional angular position Pi , i = 1, ..., N, and these frames may be used to improve the accuracy of determining f, ξ, and O at the reference angular position P0, as disclosed in §§

[0065] to

[0069] .

[0027] The camera model update is performed "online" during the clinical procedure, every frame time, according to the steps of FIG. 9, which are further disclosed in §§

[0070] -

[0072] . The angular displacement δ can be determined from a number of methods, either using additional equipment, such as optical encoders [1] or optical tracking [2], or relying solely on image processing. The disclosed embodiments will consider, without loss of generality, that the relative rotation between the lens scope and the camera head is determined using the image processing method, which is also disclosed. This method involves the determination of the center C for every frame i. i and Notch P i A boundary contour Ω with i (Module B of FIG. 9, further disclosed in §§0050 to 0057) and then infers the corresponding angular displacement δ relative to a reference position, with or without prior knowledge of the center of rotation Q (Module C of FIG. 9, further disclosed in §§0058 to 0064).

[0028] 2.1 Camera calibration at a specific angular position including eigenfield K(f,O) and distortion ξ (module A in Fig. 7) The literature is extensive in methods for calibrating pinhole cameras with radial distortion, which can be divided into two large groups: explicit and automatic calibration methods. The former use images of a known calibration object, which can be a general 3D object, a collection of spheres, a planar grid pattern, etc., while the latter rely on correspondences across successive frames of an unknown natural scene. The two approaches may require more or less user supervision, ranging from manual to fully automatic, depending on the particular method and underlying algorithms.

[0029] The disclosed embodiments contemplate, without loss of generality, that the camera calibration at a particular angular position of the lens scope relative to the camera head will be performed using an explicit method that utilizes a known calibration object, such as a planar grid pattern or any other planar pattern that allows establishing point correspondence between the image and the calibration object. This approach is advantageous over most competing methods due to its good performance in terms of robustness and accuracy, ease of fabrication of the calibration object (planar grid), and the possibility of achieving complete calibration from a single image of the rig taken from any position. However, other explicit or automatic calibration methods may be used to estimate the focal length f, distortion ξ, and principal point O of the endoscopic camera for a particular relative rotation (reference angular position) between the camera head and the endoscope.

[0030] Explicit calibration using planar grid patterns typically involves the steps of acquiring frames of a calibration object from an arbitrary position or 3D pose (rotation R and translation t of the object relative to the camera), employing an image processing algorithm to establish point correspondences x, X between the images and the calibration object, and running a suitable calibration algorithm that uses estimates of the focal length f, principal point O, and distortion parameter ξ, as well as the point correspondences for the object pose R, t relative to the camera.

[0031] The approach uses multiple calibration frames I instead of a single frame, with the aim of improving robustness and accuracy. k , k=0,...,K-1. In this case, the calibration is performed independently for each frame, and the final optimization step, which minimizes the reprojection error, selects a different pose R for each frame. k , t k is used to enforce the same intrinsic parameters K(f,O) and distortion ξ across multiple frames while taking into account

[0032] 2.2 Circular Boundary and Notch Detection (Module B in Figures 7 and 9) The circular boundary and notches of the FSM can be detected as organized in FIG. 3A. The method starts by considering initial values ​​of boundary Ω and notch P, which are used as inputs to a warp function that renders a so-called ring image. The initial values ​​of boundary Ω and notch P can be obtained from multiple methods, including but not limited to deep / machine learning, image processing, statistically based and random approaches. As an example, boundary Ω can be initialized by considering a circle centered at the image center and with a radius equal to half the minimum between the image width and height, by radially searching for a transition between the black frame of the image and an area containing significant information, or by using a deep learning framework to detect a circle, a general cone, or any other desired shape. Notch P can be initialized at a random location on the boundary, or by using a learning scheme and / or image processing to detect the known shape of the notch. Referring to steps 1 and 2 of FIG. 3A, the ring image is initialized by considering a circle with radius r i <rおよびr o >r, the inner circle Ω centered on C in Ω i and outer circle Ω o where r is the radius of Ω. i andr o The uniform spacing between the concentric circles Ω j Define a set of Ω j For, the image signals are interpolated and concatenated. The hypothesized notch P is mapped to the center of the ring image.

[0033] Referring to step 2 of FIG. 3A, edge points on the ring image that theoretically correspond to points belonging to the boundary are detected by searching for abrupt intensity changes along a direction from the periphery towards the boundary center. This is achieved by analyzing the magnitude of the 1-D spatial differential response (gradient magnitude) along each column of the ring image. A possible solution for selecting these edge points is to select the first local maximum of the gradient magnitude for each column. However, as illustrated in FIG. 3B, there are situations where this approach fails (e.g., situations of strong light scattering near the boundary). To overcome this, for each column of the ring image, a set of M edge points corresponding to local maxima of the gradient magnitude are selected.

[0034] The detected edge points are then inversely mapped into Cartesian image space so that the circle boundary can be estimated. This is performed using a circle fitting approach within a robust framework. Given a noisy set of data points that may be contaminated by outliers, the goal of circle fitting is to find a circle that minimizes or maximizes a certain error, or a cost function that quantifies how well a given circle fits the data points. The most widely used techniques minimize either the geometric or algebraic (approximate) distance from the circle to the data points. To handle outlier data points, robust frameworks such as RANSAC are usually employed. The steps of ring image rendering, edge point detection, and robust circle estimation are performed iteratively until the detected edge points are collinear, in a robust manner. If this occurs, the algorithm proceeds to detect the notch P by performing a correlation with a known template of the notch. The output of the algorithm is the notch location P and a circle Ω with center C and radius r.

[0035] As illustrated in Figure 3C, centering the ring image using an initial estimate of the notch location P ensures that the image portion corresponding to the notch is continuous, always allowing its detection. Furthermore, collinearity of edge points is chosen as a stopping criterion since an edge point belongs to a straight line if and only if the estimated boundary is perfectly concentric with the real boundary.

[0036] As shown in FIG. 3D, a lens-specific notch template is extracted during calibration, which typically consists of a light triangle and a dark rectangular background.

[0037] The disclosed method of boundary and notch detection may have other applications, such as detecting inscriptions in the FSM to read relevant information, including, but not limited to, specific characteristics of the lens.

[0038] In addition, the implementation of this method assumes that the boundary can be accurately represented by a circle, although general cone fitting can be used in this method without significant modification.

[0039] 2.3 Image-based measurement of relative rotation between the endoscope and the camera head (module C in Figure 9) As mentioned above, finding the calibration of the current frame i is done by traversing the angle δ around the center of rotation Q. i This can be achieved by rotating the principal point O (or O0) about the reference angular position by δi. In this case, both the center Q and the angular displacement δi between frame i and frame 0 corresponding to the reference position must be estimated.

[0040] FIG. 4A illustrates the process of estimating Q from two frames i and j acquired at two different angular positions, each of whose endpoints is determined by applying the steps of FIG. 3 to the center C i , C j and Notch P i , P jThis is accomplished by simply intersecting the bisectors of the line segments where Q is the notch. If the notch cannot be detected in the image due to occlusion, poor illumination, overexposure due to light scattering, etc., then Q can be estimated using only the centers of the circular boundaries detected in three frames acquired at different angular positions. This process is illustrated in Figure 4B, where it can be seen that Q is the intersection point of the line segments obtained by connecting the centers of the boundaries detected in frames i, j, and k.

[0041] If the center of rotation Q is known and the center and notch at the reference angular position are C0 and P0, respectively, then the angular displacement δ i Notch P i , boundary center C i , or both at the same time,

number

[0042] Since the distance from the center of rotation Q to the notch P is significantly larger than the distance between Q and C, it is important that the estimation using the notch P is generally more robust and accurate and can therefore be detected in all frames. Since its detection is most affected by occlusion situations, one solution is to consider multiple notches in the FSM to ensure that at least one is always visible in the frame. FIG. 5 presents one exemplary FSM containing multiple marks with different shapes, allowing their identification, one of these marks is used as a point of reference or standard notch P. An alternative solution is to consider an FSM that projects onto a black frame that renders the image boundary of a shape that does not have circular symmetry, in which case the lack of circular symmetry of the detected shape can be used to estimate the location of the notch or point of reference P that is not visible. FIG. 6 presents one exemplary FSM that renders an elliptical shaped boundary with a major axis passing through the notch P, allowing its location to be inferred at all times.

[0043] The algorithm described in §§0050-0057 for detecting notches can be extended to the case when the FSM contains multiple notches. For this, the last step of FIG. 3A is modified by determining the correlation signal for each notch independently, fusing all the signals together using the known relative locations of the notches, and finding the highest correlation point. This approach ensures that at least one notch is detected even if one or more of the notches are occluded.

[0044] Without loss of generality, in the remainder of this patent it is assumed that the FSM has only one notch P that is always visible, and the center of rotation Q is determined from two frames.

[0045] Whenever more than two frames acquired at different angular positions are available, and the center of rotation Q and the relative rotation δ iA filtering approach may be applied to filter the potentially noisy estimate of Q. The filter accepts as input the previous estimate for Q, as well as the current bounds and notches, and calculates the updated location of Q and the relative rotation δ i This filtering technique may be implemented using any time filter, such as a Kalman filter or an extended Kalman filter.

[0046] 2.4 Offline calibration at the reference angle position FIG. 7 gives a schematic description of the procedure for obtaining a camera calibration at a reference angular position i=0, which may correspond to any relative angle between the rigid endoscope and the camera head. With the lens scope at the reference angular position, the user starts by acquiring K≧1 calibration images. Then, the intrinsic camera calibration (module A), described in §§0045-0049, obtains 2D-3D correspondences for each image.

number

number

[0050] to

[0057] (Module B). This data is stored in a memory, giving the user the option to change the angular position by rotating the lens scope relative to the camera head, and to repeat the process of image acquisition, intrinsic calibration, and boundary / notch detection. This is performed for a total number of distinct angular positions i of N>1, where i=0, 1, ..., N-1, and i=0 is the reference angular position where the final calibration is obtained after a global optimization step that fuses the estimates at angular position i and simultaneously minimizes the reprojection error for all calibration images (FIG. 8).

[0047] 7 may be performed using frames acquired at a single angular position, where N=1 and the position is the reference position i=0, or at multiple angular positions, where N>1. For each different position, either a single calibration frame (K=1) or multiple calibration frames (K>1) may be acquired.

[0048] The case N=1 and K=1 requires minimal user effort and is particularly suitable for rapid calibration in the OR, where the surgeon only needs to acquire a single image of the grid pattern after the endoscope is assembled into the camera head. The accuracy of the estimation of the calibration parameters tends to improve for increasing number of frames K.

[0049] The use of information from two or more angular positions (N>1) allows to estimate the center of rotation Q in conjunction with f, ξ, and O0, independent of the number K of frames acquired at each position. This can be achieved by following the approach illustrated in FIG. 4 and disclosed in §§

[0058] to

[0064] . For N>1, the calibrations obtained at different angular positions are fused in an extensive optimization step that enforces a rotation model, as illustrated in FIG. 8 for N=3. It can be observed that for any two angular positions, the principal point O and the notch P rotate by the same amount around the center of rotation Q. The optimization scheme serves to estimate the distortion and calibration parameters at the reference position while minimizing the reprojection error of all acquired calibration images and simultaneously enforcing this model for the scope rotation for all sampled angular positions. The formula presented in FIG. 8 provides the mathematical expression for this optimization scheme for the case of N=3 angular positions and is straightforward to extend to general values ​​of N. The function r is determined by the point

number

number

number

number

number

[0050] 2.5 Online update of calibration parameters In operation, each time a new frame j is acquired, an on-the-fly procedure must detect and measure the angular displacement relative to a reference position and update the calibration and camera model accordingly. Figure 9 gives a schematic description of this procedure. Frame j is captured at a circular boundary center C j and Notch P j 3 and the steps disclosed in §§0050-0057. Then, as disclosed in §§0058-0064, the center of rotation Q must be known, and the angular displacement δ j An estimate is made.

[0051] There are two possible operating modes for deriving Q: in mode 1, the center of rotation is known a priori from an offline calibration step using frames acquired at N>1 angular positions as disclosed in §§

[0065] -

[0069] ; in mode 2, the center of rotation is not known "a priori", but is estimated on the fly from successive frames in which the centers of the notch P and / or the circular boundary C are determined such that the methods disclosed in §§ and

[0058] -

[0064] and in FIG. 4 can be employed. As illustrated in FIG. 9, the current notch P j and center C j is found in the previous frame j-1, which is accessed through a delay operation, P j-1 and center C j-1 is used in conjunction with to estimate the center of rotation Q. As a final step, the calibration parameters are then calculated by applying a planar rotation to the principal point O0 corresponding to the reference position, i.e., by aligning the updated principal point O j =R(δ j , Q)O0.

[0052] 3. Off-site lens calibration to avoid an explicit calibration step in the OR A method is disclosed for continuously determining the calibration of an endoscopic camera, which includes two steps or phases: an offline step aimed at estimating the focal length f, the distortion ξ, and the principal point O for any reference angular position, and an online step that determines the angular displacement relative to the reference at each frame time instant and updates the position of the principal point to provide a calibration for the current frame. Because the lenses of the endoscopic camera are interchangeable, both the offline and online steps are performed on-site in the OR after the surgeon has assembled the endoscope into the camera head. While the online step is intended to be performed on-the-fly, in parallel with image acquisition in a manner seamless to the user, the offline step requires explicit user intervention to obtain one or more calibration frames, which is undesirable.

[0053] In order to minimize the disturbance to the existing surgical workflow, US9438897(B2) describes a method in the specific situation of N=1 and K=1 in the offline steps disclosed in §§

[0065] -

[0069] and FIG. 7. The surgeon's effort is minimized by requiring acquisition of a single frame at the reference position, and the center of rotation Q is determined in an online step as in mode 2 of FIG. 9. Nevertheless, the method still requires the intervention of the surgeon in the OR, which is still time-consuming and a disturbance to the workflow, and also requires the use of a sterile calibration object (in this case, a grid pattern) that is not necessarily easy to produce and adds costs.

[0054] This patent describes a lens descriptor that can be performed off-site (e.g., at the time of manufacture) with the aid of a camera or other means and that can be used for different purposes.

number

number

[0055] Because lens calibration can be performed off-site, i.e., in the factory at the time of manufacture, and online calibration is performed on the fly in a manner that is seamless to the user, there are no actions to be performed in the OR by the surgeon, which means that endoscopic camera calibration is always accomplished without altering or disrupting established routines. Furthermore, unlike what is possible with the method disclosed in US9438897(B2), calibration is also accomplished in the context of variable zoom and / or translation of the lens relative to the camera head.

[0056] descriptor

number

[0079] to

[0083] , while online methods for achieving calibration of an endoscopic camera, including a camera head and optical components, are described in §§

[0084] to

[0085] .

[0057] 3.1 Off-site and offline lens calibration The rigid endoscope is assembled on an arbitrary camera head, hereafter referred to as the characterization camera, and the offline calibration method of Fig. 7, §§

[0065] -

[0069] is adopted, which requires acquiring K calibration images at N distinct angular positions. This allows to calibrate at a reference angular position i=0, which includes knowing the intrinsic parameters K(f,O), the distortion ξ, the notch P, the circular boundary Ω with center C and radius r, and, for N>1, the center of rotation Q.

[0058] The calibration results refer to the composite arrangement of the camera head with the rigid endoscope, and the measurements depend on the particular camera head in use and the manner in which the lens is mounted within the camera head. Because the goal is to characterize only the endoscope, the effects of the camera head must be removed so that the final descriptor depends only on the lens and is invariant to the camera and / or equipment used to generate the descriptor.

[0059] The method disclosed herein achieves this objective by building on two key observations: (i) camera heads usually follow an orthogonal (or nearly orthogonal) model, which means that they only contribute to the imaging process with metric magnification and translation into pixels, and (ii) images of the field stop mask (FSM) are always related by a similarity transformation, which means that the FSM can be used as a reference to encode information about the lens that is invariant to rigid motion and scaling.

[0060] The calibration result after applying the offline method of Figure 7 shall include focal length f, distortion ξ, principal point O, notch P, and a circular boundary Ω with center C and radius r. The lens descriptor is

number

number

number

number

number

number

number

[0061] 3.2 Online Camera Calibration Using Lens Descriptors descriptor

number

number

number

number

number

[0062] 3.3 Related Considerations Off-site and offline lens calibration using a single image: One important consideration is that the calibration approach disclosed in this patent does not require knowledge of the center of rotation Q to determine the calibration of the endoscopic camera at every frame time. Therefore, if the time and effort of the off-site calibration procedure is a concern, the lens descriptor can be generated by acquiring a single calibration image, in which case the offline method of FIG. 7, §§

[0065] -

[0069] is performed with K=1 and N=1. In this case, the descriptor is generated by the entry

number

[0063] Adaptation of relative rotation (calibration by detection or by tracking): camera head δ j Since a rotation of the endoscope with respect to x causes a similar rotation in the lens reference frame in the image, the calibration update for every frame j is performed by adjusting the angular displacement δ j and can be implemented implicitly without the need to explicitly rotate the principal point around the center Q. In this case, the disclosed approach based on lens descriptors can be used alone,

number

[0064] Adaptation to optical zoom and / or translation of the lens scope along a plane perpendicular to the mechanical axis: in this disclosure, the focal length f j is the radius of the circular boundary, r j The normalized focal length, estimated from the magnification introduced by the applied camera.

number

[0065] Alternative to generating lens descriptors: Descriptors

number

number

number

[0079] to

[0083] to

number

[0066] Transmission of Lens Descriptor to Application Camera: In the disclosed embodiment, the lens descriptor must be generated off-site with the help of a characterization camera and then transmitted to a CCU or computer platform connected to the application camera that will execute the online method of §§

[0084] to

[0085] (FIG. 10). This transmission or transmission can be accomplished through multiple methods, including, but not limited to, manual insertion of calibration parameters into the CCU via a keyboard or other input interface, network connection and download from a remote server, retrieval from a database of lens descriptors, reading from a USB flash drive or any other storage medium, visual reading and decoding of a QR code, visual reading and decoding of information imprinted on the FSM, such as a number or binary code, as disclosed in PCT / US2018 / 048322.

[0067] Descriptor for a batch of lenses:descriptor

number

[0068] 4. Detection of Abnormalities in Endoscope Cameras While the calibration approach presented in §§

[0041] -

[0072] always provides accurate calibration of the endoscopic camera as it is assembled and explicitly configured in the OR, the calibration method disclosed in this patent (§§

[0073] -

[0092] ) relies on a priori assumptions, such as accurate retrieval of stored calibration information and proper assembly of the lens in the camera head. If these assumptions are not met, the camera+lens arrangement will not be accurately calibrated, and systems that use the calibration information to perform distortion correction, virtual view rendering, enhanced visualization, surgical navigation, etc. may malfunction.

[0069] This patent describes a lens descriptor for detecting anomalies in endoscopic camera calibration caused by a mismatch between loaded calibration information and the lens in use, and / or by improper assembly of the lens in the camera head, or by defects in any of these components.

number

[0070] FIG. 12 provides a schematic illustration of this method for anomaly detection. For each acquired frame j, boundary and notch detection is performed both to obtain an updated calibration from the loaded lens descriptor as described in §§0084-0085, and as described in §§0058-0064. This is done by comparing the center of rotation (Q in FIG. 12) with the center of rotation (Q in FIG. 12), which can be compared to detect anomalies. j and

number

[0071] This change in the lens motion model can be used to detect the presence of an anomaly, as well as to quantify how severe the anomaly is, and to alert the user to verify the installation and / or replace the lens.

[0072] This method only provides information about the presence of anomalies and does not specify what type of anomaly is occurring, which allows the system to provide user-specific instructions to correct the anomaly. To achieve this, the approach for anomaly detection codified in FIG. 12 can be complemented by another method for identifying the source of the anomaly. Since an incorrect assembly of the lenses in the camera head causes modifications in the projection of the FSM in the image plane, a feature that quantifies the difference between the boundaries detected at calibration and operation time can be used to distinguish anomalies caused by mismatches or defective assembly of the calibration optics.

[0073] In particular, if the FSM projects onto a circle in the image plane when the lens is properly assembled in the camera head, this circle will tend to evolve into an ellipse when the optics are not assembled correctly. Therefore, in this case, the eccentricity of the detected boundary during operation can be measured to verify whether the assembly is correct, without the need to know the specific shape of the detected boundary during the calibration of the lens.

[0074] This approach is effective when the FSM has a shape that can be represented parametrically, such as an ellipse or any other geometric shape. In addition, template matching or machine learning techniques can be used to compare boundaries detected during operation to known shapes.

[0075] In summary, there are two important features that can be used to detect and identify anomalies. The first feature is the difference between the center of rotation estimates obtained at calibration and during operation, Qj and Qj, respectively.

number

[0076] Thus, the disclosed method for anomaly detection and identification utilizing these two separate features may be implemented using a cascade classifier that starts by using the first feature for the anomaly detection stage and then utilizes the second feature to distinguish between calibration mismatch and incorrect camera + lens assembly. Instead of a cascade classifier, other methods such as different types of classifiers, machine learning, statistical approaches, data mining, etc. Additionally, depending on the desired application, these features may be used individually, in which case the first feature allows for the detection of anomalies without identifying the type of anomaly, and the second feature will only serve to detect incorrect assembly.

[0077] 13 is a diagram of an exemplary computing system including a general-purpose computing system environment 1200, such as a desktop computer, laptop, smartphone, tablet, or any other such device capable of executing instructions, such as those stored in a non-transitory computer-readable medium. Additionally, while described and illustrated in the context of a single computing system 1200, those skilled in the art will also appreciate that the various tasks described below may be performed in a distributed environment having multiple computing systems 1200 linked via a local or wide area network, where executable instructions may be associated with and / or executed by one or more of the multiple computing systems 1200. The computing system environment 1200, or portions thereof, may find use for the processing, methods, and computing steps of the present disclosure.

[0078] In its most basic configuration, the computing system environment 1200 typically includes at least one processing unit 1202 and at least one memory 1204, which may be linked via a bus 1206. Depending on the exact configuration and type of computing system environment, the memory 1204 may be volatile (such as RAM 1210), non-volatile (such as ROM 1208, flash memory, etc.), or some combination of the two. The computing system environment 1200 may have additional features and / or functionality. For example, the computing system environment 1200 may also include additional storage devices (removable and / or non-removable), including, but not limited to, magnetic or optical disks, tape drives, and / or flash drives. Such additional memory devices may be made accessible to the computing system environment 1200, for example, by a hard disk drive interface 1212, a magnetic disk drive interface 1214, and / or an optical disk drive interface 1216. As will be appreciated, these devices, each of which will be linked to the system bus 1206, enable reading from and writing to the hard disk 1218, reading from or writing to a removable magnetic disk 1220, and / or reading from or writing to a removable optical disk 1222, such as a CD / DVD ROM or other optical media. The drive interfaces and their associated computer-readable media enable non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing system environment 1200. Those skilled in the art will further appreciate that other types of computer-readable media capable of storing data may be used for this same purpose.Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memory, nano drives, memory sticks, other read / write and / or read-only memory, and / or any other method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Any such computer storage media may be part of the computing system environment 1200.

[0079] Some program modules may be stored in one or more of the memory / media devices. A basic input / output system (BIOS) 1224, containing the basic routines that help transfer information between elements within the computing system environment 1200, such as during start-up, may be stored in ROM 1208. Similarly, the RAM 1210, the hard drive 1218, and / or the peripheral memory devices may be used to store computer-executable instructions, including an operating system 1226, one or more application programs 1228 (such as applications that implement the methods and processes of the present disclosure), other program modules 1230, and / or program data 1232. Still further, computer-executable instructions may be downloaded to the computing environment 1200 as needed, for example, via a network connection.

[0080] An end user, e.g., a customer, retail associate, etc., may enter commands and information into the computing system environment 1200 through input devices such as a keyboard 1234 and / or a pointing device 1236. Although not illustrated, other input devices may include a microphone, joystick, game pad, scanner, etc. These and other input devices would typically be connected to the processing unit 1202 by a peripheral interface 1238, which in turn would be coupled to the bus 1206. The input devices may be connected directly or indirectly to the processor 1202 via an interface, such as, for example, a parallel port, game port, Firewire, or universal serial bus (USB). To view information from the computing system environment 1200, a monitor 1240 or other type of display device may also be connected to the bus 1206 through an interface, such as via a video adapter 1242. In addition to the monitor 1240, the computing system environment 1200 may also include other peripheral output devices, not shown, such as speakers and printers.

[0081] The computing system environment 1200 may also utilize logical connections to one or more computing system environments. Communications between the computing system environment 1200 and a remote computing system environment may be exchanged through an additional processing device, such as a network router 1252, responsible for network routing. Communications with the network router 1252 may be implemented through a network interface component 1254. Thus, it will be understood that within such a network environment, for example, the Internet, the World Wide Web, a LAN, or other similar types of wired or wireless networks, the program modules illustrated with respect to the computing system environment 1200, or portions thereof, may be stored in the memory storage devices of the computing system environment 1200.

[0082] The computing system environment 1200 may also include localization hardware 1256 for determining a location of the computing system environment 1200. In an embodiment, the localization hardware 1256 may include only a GPS antenna, an RFID chip or reader, a Wi-Fi antenna, or other computing hardware that may be used to capture or transmit signals that may be used to determine the location of the computing system environment 1200.

[0083] In a first aspect of the present disclosure, a method for calibrating an endoscopic camera is provided. The endoscopic camera results from combining a rigid endoscope with a camera, where the rigid endoscope or lens scope has a field stop mask (FSM) that renders an image boundary with a center C and a notch P, and can be rotated relative to the camera head by an angle δ around a mechanical axis that intersects the image plane at a point Q, and C, P, and a principal point O undergo a 2D rotation of the same angle δ around Q, and the calibration consists of determining the focal length f, distortion ξ, rotation center Q, and principal point O0 for a selected angular position of the lens scope relative to the camera head, hereafter referred to as the reference angular position i=0. The method includes acquiring one or more calibration images of a calibration object with the endoscopic camera at an angular position i without rotating the lens relative to the camera head, and determining the calibration parameters f, ξ, and principal point O of the endoscopic camera. i , as well as the 3D pose (rotation and translation) of the calibration object relative to the camera for each calibration image, and use image processing methods to estimate the center C i and Notch P i and rotating the lens scope relative to the camera head to a new angular position i, and repeating the above steps, where i is incremented to take on successive values ​​i=0, 1, ..., N-1, where N >= 1 is the number of different angular positions used for calibration; and detecting the angular displacement δ with respect to the center of rotation Q and between the reference angular position i=0 and the successive calibration positions i=1, ..., N-1. iand determining a first estimate for the angle δ about the center Q for the principal point, boundary center, and successive calibration positions i=0, . . . N-1. i and refining the calibration parameters f, ξ, Q, and O0 through a final optimization step that forces a model of the notch that rotates only by

[0084] In one embodiment of the first aspect, the calibration object is either a 2D plane with a grid pattern or any other known pattern, a known 3D object, or is absent, and the calibration input is a set of point correspondences across the image, where the first estimates of the calibration parameters are obtained by a camera calibration from a plane algorithm, a camera calibration from an object algorithm, or a suitable automatic calibration technique, respectively.

[0085] In one embodiment of the first aspect, the final optimization step is performed using an iterative non-linear minimization of the reprojection error, the photogeometry error, or any other suitable optimization approach.

[0086] In one embodiment of the first aspect, the first estimate of the calibration parameter is determined from any calibration method in the literature.

[0087] In one embodiment of the first aspect, the first estimate of the calibration parameters comprises any distortion model known in the literature, such as a Brownian polynomial model, a rational model, a fisheye model, or a division model having one or more parameters, where ξ is a scalar or a vector, respectively.

[0088] In one embodiment of the first aspect, the center of rotation Q is known a priori, in which case calibration can be achieved from images acquired at one or more angular positions (N>=1), and the center of rotation Q is determined by the boundary center C i and Notch P i in which case calibration is accomplished from images acquired at two or more angular positions (N>=2), or from the boundary center C i or Notch P iis determined only from image positions N, in which case calibration is achieved from images acquired at three or more angular positions (N>=3).

[0089] In a second aspect of the present disclosure, a method is provided for updating the calibration parameters of an endoscopic camera every frame time. The endoscopic camera results from combining a rigid endoscope with a camera, the camera comprising a camera head and a camera control unit (CCU), the rigid endoscope or lens scope has a field stop mask (FSM) that renders an image boundary with a center C and a notch P, and can rotate with respect to the camera head by an angle δ around a mechanical axis that intersects the image plane at a point Q, C, P, and principal point O perform a 2D rotation of the same angle δ around Q, and the calibration parameters focal length f, distortion ξ, rotation center Q, and principal point O0, as well as the boundary with center C0 and notch P0 are known for a reference angular position i=0 of the lens scope relative to the camera head. The method includes acquiring a new frame j by the endoscopic camera, and acquiring the boundary center C j and Notch P j and detecting notch P0, notch P j , and Q, estimating the angular displacement δ of the endoscope lens relative to the camera head, and performing a 2D rotation of the principal point O0 around Q by an angle δ, thereby obtaining an updated principal point O j and estimating

[0090] In one embodiment of the second aspect, the calibration parameters, focal length f, distortion ξ, center of rotation Q, and cardinal point O0, as well as a boundary having center C0 and notch P0 at a reference angular position i=0, are obtained by calibrating the endoscopic camera with a lens scope at the reference position or removed from the CCU.

[0091] In one embodiment of the second aspect, the center of rotation Q is located between two or more boundary centers C j and / or Notch P j is determined using

[0092] In one embodiment of the second aspect, the angular displacement of the endoscope lens is estimated by mechanical means and / or by utilizing optical tracking, in which case the boundary centers C0 and C j , and notches P0 and P j does not need to be known.

[0093] In one embodiment of the second aspect, the method further includes using techniques for filtering the estimates of the center of rotation Q and the angular displacement δ, including any recursive or temporal filter known in the literature, such as, but not limited to, a Kalman filter or an extended Kalman filter.

[0094] In a third aspect of the present disclosure, a normalized focal length

number

number

number

number

number

number

number

[0095] In an embodiment of the third aspect, the normalized focal length

number

number

number

number

number

number

number

[0096] In a fourth aspect of the present disclosure, a method for calibrating an endoscopic camera is provided, the endoscopic camera resulting from combining a rigid endoscope with a camera, the camera comprising a camera head and a camera control unit (CCU), the rigid endoscope calibrating a normalized focal length

number

number

number

number

number

number

number

number

[0097] In one embodiment of the fourth aspect, the focal length f, the principal point O, and the center of rotation Q are respectively expressed as:

number

number

number

[0098] In one embodiment of the fourth aspect, an endoscope lens descriptor

number

[0099] In one embodiment of the fourth aspect, an endoscope lens descriptor

number

[0100] In one embodiment of the fourth aspect, the frame i includes two or more frames, and the rotation center Q is determined by two or more boundary centers C i and / or Notch P i is determined using

[0101] In one embodiment of the fourth aspect, the endoscopic camera can have any angular position of the lens scope relative to the camera head and any amount of zoom.

[0102] In a fifth aspect of the present disclosure, there is provided a method for detecting anomalies caused by defects or inaccurate assembly of a rigid endoscope in a camera head or by a mismatch between a calibration taken into account and an endoscope lens during use in an endoscopic camera resulting from combining a camera with a camera head and a camera control unit (CCU) and a rigid endoscope, the method comprising the steps of: (a) detecting anomalies caused by defects or inaccurate assembly of a rigid endoscope in a camera head and a camera control unit (CCU) and combining a camera with a rigid endoscope;

number

number

number

number

number

[0103] In an embodiment of the fifth aspect, an endoscope lens descriptor

number

[0104] In one embodiment of the fifth aspect, two centers of rotation

number

[0105] In one embodiment of the fifth aspect, comparing the boundaries detected during calibration and during operation to identify the cause of the anomaly.

[0106] In one embodiment of the fifth aspect, the method further includes providing a warning message to the user, and the cause of the anomaly is identified, whether it is a mismatch between the lens in use and the considered calibration, or a physical problem with the rigid endoscope and / or camera head.

[0107] In a sixth aspect of the present disclosure, a method for detecting an image boundary having a center C and a notch P in a frame acquired by using a rigid endoscope is provided, the rigid endoscope having a field stop mask (FSM) that induces the image boundary having a center C and a notch P, the method includes: rendering a ring image by interpolating and concatenating image signals extracted from the acquired frames in concentric circles centered on C, where the notch P is mapped to the center of the ring image, using an initial estimate of the boundary having a center C and a notch P; detecting corner points in the ring image; and, until the detected corner points are collinear, performing the following steps: mapping the corner points into the space of the acquired frame and fitting a fitted circle having a center C to the mapped points; rendering a new ring image by utilizing the fitted circle; detecting the corner points in the new ring image; and detecting the notch P in the final ring image using correlation with a known template.

[0108] In one embodiment of the sixth aspect, the FSM contains two or more notches, all having different shapes and / or sizes such that they can be identified, in which case the template includes a combination of notches whose relative locations are known.

[0109] In an embodiment of the sixth aspect, the notch may have any desired shape.

[0110] In an embodiment of the sixth aspect, the notch that is mapped to the center of the ring image is an arbitrary notch.

[0111] In one embodiment of the sixth aspect, an initial estimate of the boundary having a center C and a notch P may be obtained from a number of methods, including but not limited to deep / machine learning, image processing, statistically based and random approaches.

[0112] In an embodiment of the sixth aspect, a general cone is fitted to the mapped points.

[0113] In one embodiment of the sixth aspect, the detected center C and / or notch P are used to estimate the angular displacement of the rigid endoscope relative to the camera head to which it is attached.

[0114] Although various embodiments have been described for the purposes of this disclosure, such embodiments should not be considered as limiting the teachings of this disclosure to those embodiments. Various changes and modifications can be made to the above elements and operations to obtain results that remain within the scope of the systems and processes described in this disclosure. All patents, patent applications, and published references cited herein are incorporated herein by reference in their entirety. It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations, and are set forth merely for a clear understanding of the principles of the present disclosure. Many variations and modifications may be made to the above-described embodiments without substantially departing from the concepts and principles of the present disclosure. It may be desirable to combine some of the above-disclosed and other features and functions, or alternatives thereof, into many other different systems or applications. All such modifications and variations are intended to be included herein within the scope of the present disclosure, so long as they fall within the scope of the appended claims.

[0115] The described embodiments are considered in all respects only as illustrative and not restrictive, the scope of the disclosed embodiments being thus indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are intended to be embraced within their scope. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as a departure from the scope of the disclosed systems and / or methods.

[0116] References All cited references are expressly incorporated herein by reference. The discussion of any reference is not an admission that it is prior art to the embodiments of the present disclosure, especially any reference that may have a publication date after the priority date of this application. [1] T. Yamaguchi, M. Nakamoto, Y. Sato, K. Konishi, M. Hashizume, N. Sugano, H. Yoshikawa, and S. Tamura, “Development of a camera model and calibration procedures for oblive-viewing endoss,” Computer Aided Surgery, vol.9, no5, pp.203-214, February 2004. [2] C. Wu, B. Jaramaz, and S. Narasimhan, “A full geometry and photometric calibration method for oblive-viewing endoscopes,” Computer Aided Surgery, vol.15, no.1-3, pp.19-31, April 2010. [Additional note 1] 1. A method for calibrating an endoscopic camera including a rigid endoscope and a camera, the camera comprising a camera head, the rigid endoscope comprising a lens scope, the rigid endoscope or the lens scope having a field stop mask (FSM) that renders an image boundary having a center C and a notch P, and can be rotated relative to the camera head by an angle δ about a mechanical axis that intersects an image plane at a point Q, such that C, P, and a principal point O undergo a 2D rotation about Q by the same angle δ, the method comprising: acquiring one or more first calibration images of a calibration object with the endoscopic camera at the reference angular position; For each first calibration image, the focal length f of the endoscopic camera relative to the camera, the distortion ξ, the center of rotation Q, and the principal point O 0 determining a first estimate of According to the image processing method, the center C on the calibration image i and Notch P i and detecting a boundary having performing an iterative process, said iterative process comprising: acquiring one or more additional calibration images of a calibration object with the endoscopic camera at additional angular positions of the lens scope relative to the camera head; determining an estimate of an angular displacement between the reference angular position and the further angular position; For each further calibration image of the iteration, the focal length f of the endoscopic camera relative to the camera, the distortion ξ, the center of rotation Q, and the principal point O 0 determining a further estimate of According to the image processing method, the center C on the further calibration image of the iteration i and Notch P i and performing one or more iterations of detecting a boundary having The focal length f, the distortion ξ, the rotation center Q, and the principal point O 0 and according to the estimated angular displacement, f, ξ, and O 0 and refining [Additional note 2] determining a first estimate of a 3D pose of the calibration object, the 3D pose including a rotation and a translation, for each first calibration image; The iterative process comprises: determining a further estimate of a 3D pose of the calibration object, the 3D pose including a rotation and a translation, for each further calibration image of the iteration; f, ξ, and O 0 The method of claim 1 , wherein refining {tilde over (x)} further follows the further estimate of the 3D pose of the calibration object. [Additional note 3] The calibration object is A 2D plane with a grid pattern, A 2D plane with a known pattern, or The method of claim 1 , further comprising one or more of: [Additional note 4] Refining the calibration parameters (a) Iterative nonlinear minimization of the reprojection error, (b) iterative nonlinear minimization of the photogeometry error, or 2. The method of claim 1 , further comprising one or more of: (c) cost function minimization that is not (a) and that is not (b). [Additional note 5] The first estimate of the calibration parameter is Brown's polynomial model, Rational model, Fisheye model, or A distortion model comprising one or more of: a segmentation model having one or more parameters; [Additional note 6] 1. A method for updating calibration parameters of an endoscopic camera, the endoscopic camera comprising a rigid endoscope and a camera, the camera comprising a camera head and a camera control unit (CCU), the rigid endoscope or lens scope having a field stop mask (FSM) that renders an image boundary having a center C and a notch P, and capable of rotating relative to the camera head by an angle δ about a mechanical axis that intersects an image plane at a point Q, the C, P, and principal point O undergo a 2D rotation about Q by the same angle δ, and the calibration parameters focal length f, distortion ξ, center of rotation Q, and principal point O are updated. 0 , and center C 0 and Notch P 0 is known for a reference angular position i=0 of the lens scope relative to the camera head, and the method comprises: acquiring a new frame j using the endoscopic camera; The boundary center C in the new frame j j and Notch P j Detecting Notch P 0 , Notch P j estimating an angular displacement δ of the endoscope lens relative to the camera head according to With Q as the center, the principal point O 0 The updated principal point O of the endoscopic camera is obtained by performing a 2D rotation of j and estimating [Additional note 7] The focal length f, the distortion ξ, the rotation center Q, and the principal point O at the reference angular position i=0 0 , and center C 0 and Notch P 0 is obtained by calibrating the endoscopic camera with the lens scope at the reference position or by removing it from the CCU. [Additional note 8] The rotation center Q is 3 or more boundary centers C j 、 3 or more notches P j ,or At least one boundary center C j and at least one notch P j The method of claim 6, wherein the signal is determined using one or more of: [Additional note 9] The method of claim 6 , further comprising filtering the estimates of center of rotation Q and angular displacement δ according to one or more of a recursive filter or a temporal filter. [Additional Note 10] Normalized Focal Length

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

Claims

1. 1. A method for calibrating an endoscopic camera including a rigid endoscope and a camera, the camera comprising a camera head, the rigid endoscope comprising a lens scope, the rigid endoscope or the lens scope having a field stop mask (FSM) that renders an image boundary having a center C and a notch P, and can be rotated relative to the camera head by an angle δ about a mechanical axis that intersects an image plane at a point Q, such that C, P, and a principal point O undergo a 2D rotation about Q by the same angle δ, the method comprising: acquiring one or more first calibration images of a calibration object with the endoscopic camera at a reference angular position; For each first calibration image, the focal length f of the endoscopic camera relative to the camera, the distortion ξ, the center of rotation Q, and the principal point O ０ determining a first estimate of According to the image processing method, the center C on the calibration image ｉ and Notch P ｉ and detecting a boundary having performing an iterative process, said iterative process comprising: acquiring one or more additional calibration images of a calibration object with the endoscopic camera at additional angular positions of the lens scope relative to the camera head; determining an estimate of an angular displacement between the reference angular position and the further angular position; For each further calibration image of the iteration, the focal length f of the endoscopic camera relative to the camera, the distortion ξ, the center of rotation Q, and the principal point O ０ determining a further estimate of According to the image processing method, the center C on the further calibration image of the iteration ｉ and Notch P ｉ and performing one or more iterations of detecting a boundary having The focal length f, the distortion ξ, the rotation center Q, and the principal point O ０ and according to the estimated angular displacement, ０ and refining

2. determining a first estimate of a 3D pose of the calibration object, the 3D pose including a rotation and a translation for each first calibration image; The iterative process comprises: determining a further estimate of a 3D pose of the calibration object, the 3D pose including a rotation and a translation, for each further calibration image of the iteration; The f, the ξ, and the O ０ The method of claim 1 , wherein refining {tilde over (x)} further follows the further estimate of the 3D pose of the calibration object.

3. The calibration object is A 2D plane with a grid pattern, A 2D plane with a known pattern, or The method of claim 1 , further comprising one or more of: a known 3D object;

4. Refining the f, the ξ, and the O 0 comprises: (a) Iterative nonlinear minimization of the reprojection error; (b) Iterative nonlinear minimization of the photogeometry error, or The method of claim 1 , further comprising one or more of: (c) a cost function minimization that is not (a) and that is not (b).

5. A first estimate of the focal length f, the distortion ξ, the center of rotation Q, and the principal point O of the endoscopic camera relative to the camera is Brown's polynomial model, Rational model, Fisheye model, or A distortion model comprising one or more of: a segmentation model having one or more parameters.

Images