Systems and methods for changing the direction of view during video-guided clinical procedures using real-time image processing

Abstract

Arthroscopes and laparoscopes are available with several lens cutouts to cover different clinical situations of interest, requiring surgeons to exchange optical elements to change the camera's direction of view. Embodiments of the present disclosure disclose real-time image processing systems and methods that allow users to arbitrarily change the surgical camera's direction of view, with the advantage of avoiding workflow interruptions caused by physically exchanging optical elements. Furthermore, embodiments of the present disclosure disclose implementing zooming along any direction of view (directional zoom), allowing for increasing the scale of a region of interest without reducing the overall field of view or losing image content.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 911,986, filed October 7, 2019 (the "'986 Application"), and U.S. Patent Application No. 62 / 911,950, filed October 7, 2019 (the "'950 Application"), both of which are incorporated by reference in their entireties for all purposes.

[0002] The present disclosure relates generally to the field of computer vision and image processing, and in particular, but not by way of limitation, the presented disclosed embodiments may be used for enhanced visualization in surgical and diagnostic video-guided minimally invasive clinical procedures, such as arthroscopy, laparoscopy, or endoscopy, for the purpose of changing the direction of view of a surgical camera with an endoscopic lens, where the system can render a field of view obtained by a physical scope with a different lens cut than the lens effectively used, or perform zooming along any direction of view without reducing the image field of view or losing image content. Summary of the Invention [Problem to be solved by the invention]

[0003] In video-guided procedures, such as arthroscopy or laparoscopy, the anatomical cavity of interest is accessed through small incisions designated as surgical ports. One of these ports provides access to a video camera equipped with a rigid endoscope, allowing the surgeon to visualize the interior of the cavity for guidance during the surgical or diagnostic procedure. A rigid endoscope is a long, tubular structure inserted into a body cavity. An endoscope typically includes an objective lens at the distal end, an image transmission system such as a series of spaced lenses, and an ocular lens at the proximal end, usually fitted with a camera means such as a charge-coupled device (CCD) chip. The image transmission system is responsible for transmitting the image from the distal end to the proximal end. A camera receives this image and generates a signal for a video display viewed by the surgeon.

[0004] Rigid endoscopes, also referred to herein as lensscopes or endoscopic lenses, are used in different specialties and, depending on their characteristics and field of application, may alternatively be named arthroscopes (orthopedics), laparoscopes (abdominal surgery), cystoscopes (bladder), ENT (enterological surgery), neuroscopes, etc. Endoscopic cameras, resulting from the combination of a rigid endoscope with a camera, have certain features that are rare in conventional cameras. The optical elements are usually replaceable for easy sterilization, and the endoscope is assembled into the camera head by the surgeon before starting the procedure. The ocular lens (or eyepiece) at the proximal end of the endoscope is typically integrated into the camera using a connector that allows the user to rotate the scope at an angle a relative to the camera head. This rotation is in azimuthal angles around the longitudinal axis of the endoscope, hence referred to as the mechanical axis, which is close to, but does not necessarily coincide with, the axis of symmetry of the tubular structure.

[0005] Rigid endoscopes typically have a field stop mask (FSM) somewhere in the image transmission system that causes the acquired image to have meaningful content in a circular area surrounded by a black frame (this black frame is shown with diagonal lines in the figures of this application). The design of the FSM is usually such that there is a mark at the circular boundary defined by the image frame transitions that allows the surgeon to estimate the angle of the azimuth angle α. This mark at the periphery of the circular image will hereafter be referred to as a notch.

[0006] Additionally, the lenses at the distal end of an endoscope are often mounted so that the optical axis of the entire lens system has an angular offset from the longitudinal or mechanical axis of the scope. This angle β between the optical and mechanical axes is commonly referred to as the lens cut of the endoscope. If the lens cut is different from zero, the optical and mechanical axes will not align, and the surgeon can change the direction of the field of view in azimuth by rotating the scope about the mechanical axis without moving the camera head.

[0007] Variable direction of view (DoV) allows surgeons to change the endoscopic viewing direction without changing the position of the endoscope itself. This is useful for viewing structures to the side or behind the endoscope tip when the endoscope shaft cannot be easily moved due to anatomical constraints or constraints imposed by other surgical instruments in the surgical field. Therefore, a variable DoV is clearly a desirable feature that provides surgeons with greater flexibility in their procedural approach. As discussed, rigid endoscopes currently used in clinics offer surgeons the ability to change the DoV by rotating the lens scope in azimuth, which allows the optical axis to describe a cone in space with its apex close to the center of projection O (the DoV cone). The half angle of this cone is defined by the lens cut β, which is fixed and known a priori by the surgeon. Surgeons rely heavily on prior knowledge of the lens cut of a particular endoscope to ensure they know exactly what the anatomical structures should be.

[0008] Endoscopes with different angular offsets from the longitudinal or mechanical axis are available, with the most commonly used lens cuts being 0°, 30°, 45°, and 70°. Surgical procedures typically require endoscopes with most of these cut angles, with particular emphasis on one. For example, in knee arthroscopy, a 30° lens cut is preferred, providing both a good forward view and a certain amount of lateral visibility, while in hip arthroscopy, the lens cut of choice is typically 70°, due to the much narrower anatomical cavity. Similarly, the preferred lens cut in laparoscopy can vary between 0° (forward-looking scope) and 30°. However, despite the emphasis on specific lens cut angles, most procedures can benefit from different lateral and partial rearward viewing angles at different moments or steps of surgery. Unfortunately, because a lateral change in DoV requires a physical exchange of the endoscope with one having a different lens cut, surgeons rarely actually make the change, even if they realize that such a change could improve visualization for accomplishing the task at hand. Changing endoscopes mid-procedure is tedious (both the optical and camera cables must be disconnected and reconnected), time-consuming, and sometimes dangerous. For example, inserting an off-angle endoscope can be dangerous because it is not pointed in the direction it is being inserted, and neurosurgeons often refrain from using endoscopes with 45° or 70° lens cuts because they fear blindly forcing the endoscope into delicate tissue.

[0009] In summary, rigid endoscopes commonly used in medicine allow surgeons to vary the DoV in azimuth (angle α) but not in tilt or elevation (angle β), where the optical axis is constrained to move within a cone with an axis of symmetry aligned with the mechanical axis and an angle defined by the lens cut β (the cone of DoV). The discussion that follows demonstrates that unconstrained variation in DoV is highly beneficial by allowing surgeons to change the lateral viewing angle (lens cut β) mid-procedure without physically changing the endoscope. Several specially designed endoscopes have been disclosed that allow angular shifts in the direction of view in both azimuth and elevation (US 2016 / 02.09 A1, US 20050177026 A1, US 2012 / 0078049, US 9,307,892). Unfortunately, these endoscopes use movable image sensors or optical elements at the distal end to change the lens cut β. Because of these moving parts, these scopes are complex and expensive to manufacture, are less robust than traditional fixed-lens cut scopes, and often have inferior lighting and image quality.

[0010] An alternative approach to achieving unconstrained variation in DoV is to employ computational means to process endoscopic images or videos. It is well known in the field of computer vision that a change in DoV corresponds to a rotation of the camera around an axis passing through its projection center, and that a virtual image acquired by such a rotated camera can be rendered by warping the real source image with an appropriate homographic map, often referred to as a collinear transformation of the plane at infinity. This method is used in patent applications US005313306A, US20154 / 0065799A1, and EP3130276A1, which disclose a rigid endoscope with a wide-angle lens for a hemispherical field of view (FoV), which, in combination with computational means, renders images with a variable DoV covering selected portions of the FoV. The result is a camera system with controllable pan and tilt orientation, where the user uses electronic means to command the viewing direction in both X (azimuth) and Y (elevation).

[0011] The problem with these systems is that in the context of endoscopic and video-guided clinical procedures, surgeons are used to change the azimuthal viewing direction by simply rotating the lens scope relative to the camera head. Therefore, moving from mechanical to electronic control is difficult and requires surgical retraining, presenting a major obstacle to widespread adoption.

[0012] What is desired, therefore, is an endoscopic system having a DoV that can be varied in both azimuth angle α and tilt β, where the azimuth angle change is achieved by standard mechanical means, i.e., by rotating the rigid endoscope relative to the camera head, and the tilt change is achieved by electronic or computer means, i.e., by using real-time processing to render an image at a target viewing direction.

[0013] It would still further be desirable to provide an endoscopy system that operates in a standard manner but allows the user to electronically switch between rigid endoscopes having different lens cuts for optimal visualization conditions at any given time.

[0014] It is further desirable to provide an endoscopic system in which the user can control the amount of radial distortion in the image to either manipulate an undistorted image for optimal perception of scene depth, or to vary the DoV to select a region of interest and enlarge the region of interest while preserving the FoV and all visualized content in the image.

[0015] A system and method that uses real-time image processing to change the direction of view during surgical and diagnostic video-guided clinical procedures while allowing the surgeon to physically rotate the lens scope relative to the camera head, as is currently common practice in different surgical procedures such as arthroscopy, laparoscopy, and endoscopy.

[0016] Embodiments of the present disclosure disclose a software-based system that allows the viewing direction of a particular scope to be changed by any amount γ, such that the cut angle of the lens (or lens cut) is virtually switched from β to β + γ. This is accomplished in a seamless, fully automated manner that is realistic for a user who may physically rotate the endoscope in azimuth about its longitudinal axis during operation.

[0017] The software-based system processes images and videos acquired by an endoscopic camera with a lens having a wide field of view (FoV) to give the surgeon the ability to electronically switch between two or more virtual endoscopes with different lens cuts.

[0018] Additionally, embodiments of the present disclosure may be implemented to target two virtual endoscopes with desired lens cuts.

number

number

[0019] Embodiments of the present disclosure may be implemented in terms of the desired shift in lens cut γ as well as the image resolution m×n, field of view

number

number

number

number

[0020] Embodiments of the present disclosure provide for controlling the amount of image radial distortion to manipulate the undistorted image for optimal perception of scene depth, or for reducing radial distortion.

number

number

[0021] For a more complete understanding of the present disclosure, reference is made to the following detailed description of exemplary embodiments taken in conjunction with the accompanying drawings.

[0022] [Figure 1] 1 is an embodiment of an endoscopic camera comprising an endoscope and a camera head, showing all of its components. A normal image formed by the endoscope including a field stop mask (FSM) in its image transmission system is shown, as well as an image in pixels acquired after application of intrinsic parameters by a sensor in the camera head.

[0023] [Figure 2]1 is an embodiment of an endoscope showing two different directions of view (DoV) that result in two different cones of DoV, each corresponding to an orbit described by the optical axis while the endoscope rotates about a mechanical axis.

[0024] [Figure 3] 1 illustrates the image warping process between a source image and a target image, where the color of a pixel in the target image is obtained using an interpolation scheme. The warping function is a combination of the source and target camera models cs and ct and the motion model m.

[0025] [Figure 4] The process of obtaining a camera model c is outlined. A 3D point X is projected onto the image plane according to a pinhole model, resulting in a 2D point x in the normal image. Then, distortion d is an operation performed in the normal image that applies a distortion to point x, resulting in point x'. As a final step, application of the intrinsic parameters by function k transforms the point in millimeters x' to a point u in the image in pixels. The camera model c is a function applied in 2D space that transforms point x in the normal image to a point u in the image in pixels,

number

number

[0026] [Figure 5] shows the effect of scope rotation on the boundary location and principal point O. The scope rotates about a mechanical axis that intersects the image plane at the center of rotation Q. The FSM is projected onto the image plane as a circle with center C, and a mark in the FSM is projected onto a point P called the notch. When the scope rotates through an angle δi, the circle center C, notch P, and principal point O also rotate by the same amount δi around Q.

[0027] [Figure 6] shaft

number

number

number

number

number

[0028] [Figure 7] It illustrates the concept of diameter d, which is the length of the segment whose endpoint is the intersection of the line containing the principal point O with the boundary. The diameter d is used to determine the field of view (FoV) Θ of the camera. The figure shows the specific case of a line passing through a notch P.

[0029] [Figure 8] 10 outlines the disclosed method for changing the direction of view by illustrating the steps that allow for rendering of a video that would be acquired by a virtual camera with predetermined characteristics located in the same 3D position as a real endoscopic camera, the output of which is a pixel shift in a target image acquired by the virtual camera.

number

[0030] [Figure 9] The warping function is the target image

number

number

number

number

number

[0031] [Figure 10] We show a strategy to prevent the image rendered by the target camera from having regions with no visual content (empty regions). Figure 10(a) shows a portion of the interior of the target boundary being mapped beyond the source image boundary, resulting in a blank region at the bottom with no image content. Figure 10(b) shows a method to change the focal length so that the point furthest from the source boundary moves to coincide with it.

number

number

number

[0032] [Figure 11] The different steps of the disclosed method are described for estimating the focal length and principal point of a target image that aligns changes in DoV with rendering an image with a desired FoV and without sky regions.

[0033] [Figure 12] We show how to dimension the source camera to properly accommodate the lens cut and FoV of the target camera by rendering a realistic target image without sky regions and having the principal point close to the image center.

[0034] [Figure 13] The image results are shown by rendering two images acquired with two endoscopes with different lens cuts and FoVs. The top image shows the image acquired with the real camera, which is a laparoscope with a lens cut β=45° and a diameter of 10 mm. The middle image shows the virtual view acquired with γ1=-15°, in this case with a lens cut β=45°.

number

number

[0035] [Figure 14] The image results show varying the DoV by angle γ while predefining the target camera parameters to be the same as the source camera calibration estimate. The top image is the original real image acquired by the arthroscope with lens cut β = 30°. The middle image shows a virtual view acquired with γ = -30°, where the arthroscope looks forward along its axis of symmetry (β + γ = 0°). The bottom image shows a virtual view acquired with γ = 35°, which means the camera is looking downward at β + γ = 65° relative to the scope's axis of symmetry. The filled circles are the points to which the principal points in the source image are mapped.

number

[0036] [Figure 15] The image results of directional zoom, which allows scaling up an image around any viewing direction while preserving the entire FoV and image content, are shown. The top image shows the original field of view. The middle image shows the rendering result for β=65° when the target camera preset is the source camera calibration. The bottom image shows the same lens cut as the middle image, but with radial distortion.

number

number

[0037] [Figure 16] FIG. 1 is a schematic diagram of an exemplary computing system, including a general-purpose computing system environment. DETAILED DESCRIPTION OF THE INVENTION

[0038] It should be understood that, while example implementations of one or more embodiments are provided below, various specific embodiments may be implemented using any number of techniques known by those skilled in the art. The present disclosure should in no way be limited to the example embodiments, drawings, and / or techniques illustrated below, including the example designs and implementations shown and described herein. Furthermore, the present disclosure may be modified within the scope of the appended claims, and the full range of equivalents thereof.

[0039] Disclosed herein are systems and methods for changing the direction of view during video-guided clinical procedures. The systems and methods may be used for clinical procedures including, but not limited to, arthroscopic, laparoscopic, endoscopic, or other surgical procedures, including minimally invasive orthopedic surgical procedures. The systems and methods may be used with real-time or delayed image processing.

[0040] FIG. 1 schematically illustrates an endoscopic camera 34 used in the aforementioned video-guided clinical procedure, consisting of an endoscope 10 whose proximal end 16 is coupled to a camera head 28 by a connector that allows the endoscope 10 to rotate in azimuth about a mechanical axis 12 by an angle α 18. The endoscope 10 typically includes a field stop mask (FSM) along the image transmission system that causes light transmitted from the distal end 14 to the proximal end 16 to form a regular image 22, including a circular boundary 24 and a notch 26. The camera head 28 converts the regular image 22 into an image in pixels 30, further exhibiting the circular boundary 24 and the notch 26. Furthermore, as discussed in the background section, the direction of view (DoV) 36 changes while rotating the endoscope 10 relative to the camera head 28 due to the lens at the distal end 14 causing the optical axis 36 and the mechanical axis 12 to misalign by an angle β, shown as lens cut 20.

[0041] Rotation of the endoscope 10 about the mechanical axis 12 in azimuthal angle 18 causes the optical axis 36 to describe a cone of space 32 (the cone of DoV) whose half angle is the lens cut 20, as shown in FIG.

[0042] In this disclosure, 2D and 3D vectors are depicted with bold lowercase and uppercase letters, respectively. Functions are represented with lowercase italicized letters, and angles are represented with lowercase Greek letters. Points and other geometric entities in the plane are represented with homogeneous coordinates, as is commonly done in projected geometries, 2D linear transformations in the plane are represented with 3x3 matrices, and equations are scaled up. Furthermore, when representing functions, the symbol ; is used to distinguish between the function's variables (appearing to the left of ;) and parameters (appearing to the right of ;). Finally, different sections of the text are referenced by paragraph number using the symbol §.

[0043] Image Warping

[0044] The disclosed method and system for rendering virtual views with arbitrary shifts in viewing direction trends relates to image warping techniques, particularly a software-based method for creating a virtual Pan-Tilt-Zoom (PTZ) camera from a wide field of view (FoV), panoramic camera, where an image acquired by the PTZ camera (target image) is rendered from an image acquired by the panoramic camera (source image) by a function that maps pixels of one image to pixels of the other image.

[0045] Without loss of generality, as shown in Figure 3, w is the pixel coordinate u of the target image. t the pixel coordinates u of the source image s is a function that transforms a pixel u in the target image t The color values ​​of u may be determined using any type of interpolation approach in the spatial or frequency domain, including, but not limited to, nearest neighbor, next neighbor, previous neighbor, bilinear, bicubic, Lanczos, bicubic b-spline, Mitchell-Nettravalli, Catt-Malrom, kriging-based, wavelet-based, or edge-directed interpolation. Data-driven interpolation filters trained using machine learning or deep learning can also be used to interpolate u. t can be used to obtain the color value of

[0046] Existing warping techniques include, but are not limited to, direct mapping, inverse mapping, warping by resampling in the continuous or discrete image domain, warping by resampling and filtering, warping using lookup tables, warping using decomposable transforms, and learned warping transforms.

[0047] The warping function w is a function c s and c t and the camera motion, function m. s and c tdescribes the mapping between the normal image 22 in millimeters and the image 30 in pixels. Since the source camera is a real camera, c s can be determined using an appropriate calibration method. Meanwhile, the target camera c t is chosen such that the desired imaging characteristics (resolution, zoom, FoV, etc.) are predefined. In terms of the function m, the relative motion between the virtual (target) camera and the real (source) camera is described. More specifically, it represents the rotation performed by the virtual camera in 3D space that causes a homography mapping in projective coordinates between the normal image of the source camera and the target camera.

[0048] Warping images acquired with an endoscopic camera is significantly more difficult than doing so with images acquired with a conventional camera for two main reasons. First, the camera model c s changes from frame to frame due to the relative rotation of the endoscope lens with respect to the camera head, which must be taken into account when constructing the warping function w. Second, the motion model m depends not only on the rate of change of the desired elevation γ, but also on the mechanical change in the azimuth angle δ, which must be measured at every frame. These challenges do not exist in conventional cameras because there are no moving parts interfering with the camera model.

[0049] Endoscope Camera Model

[0050] In this section, we introduce the endoscopic camera model c by describing a general camera model that exhibits radial distortion introduced by optical elements, providing an overview of related concepts, and explaining how an endoscopic camera can be described by an adaptive model that is updated instantaneously for each frame.

[0051] Common camera models

[0052] Figure 4 lists the different steps involved in the camera model function c. 3D point X = [XYZ] Tis projected onto the image plane according to the pinhole model, resulting in a 2D point x in the normal image 22. This projection is denoted as p and computed

number

number

number

number

number

[0053] In this model, the distortion function d is general, meaning that it can be any distortion model in the literature, such as Brown's polynomial model, rational model, fisheye model, or division model with one or more parameters, where ξ is a scalar or vector, respectively. The distortion function d may also take into account other types of distortion, such as, but not limited to, tangential distortion, prismatic distortion, or perspective distortion caused by tilting the sensor. Without loss of generality, in the rest of the description, d applies the first-order division model, and ξ is a scalar:

number

[0054] Furthermore, point u and the principal point

number

number

[0055] Handling lens rotation

[0056] Rotation of the endoscope 10 relative to the camera head 28 causes the principal point O, the center C of the circular boundary 24, and the notch 26, shown as P, to rotate the image at pixel 30 around point Q, as shown in FIG. 5. In fact, O and C rotate around Q because they typically do not coincide due to mechanical tolerances during lens manufacturing. Knowing that O is the image of the optical axis, C is the center of the circular boundary, which depends on how the FSM is positioned relative to the lens rod, and Q is the intersection of the mechanical axis 12 with the image plane, which depends on the eyepiece, coincidence of all three points occurs when the eyepiece and FSM are perfectly aligned with the lens rod. Misalignment between Q and C causes the boundary to change position in the image, and C moves in a circle around Q as the lens rotates. If Q and O are misaligned, O also describes a circular orbit around Q.

[0057] Therefore, if the calibration parameter,f,O,ξ,refers to a particular azimuthal position,α,0,, the angle,δ, i =α i A rotation in azimuth by -α0 changes the position of the principal point

number

number

[0058] In other words, let f, O, and ξ be the calibration parameters of a generalized endoscopic camera at a particular azimuthal position α. ​​The camera model that maps a point x in the normal image to a point u in pixel coordinates may change for each frame instant i, which is

number

[0059] In the remainder of this document, without loss of generality, it is assumed that both the calibration parameters f,O,ξ of the reference azimuthal position α and the center of rotation Q are known a priori, and consequently the rotation δ with respect to α i is estimated for each frame instant i, the calibration parameters f,O i , ξ can be determined as described above. If O matches Q, then O for each frame i i It is pertinent to note that Q also coincides with Q, and therefore, no adaptation of this camera model c to the rotation of the scope is necessary. However, misalignment between these entities frequently occurs due to the mechanical tolerances in the construction of optical elements, as mentioned above. The effect of this misalignment on the calibration parameters is not negligible and must be taken into account in practice.

[0060] Camera rotation in 3D

[0061] We now describe how to determine the motion model m introduced in §§

[0043] to

[0048] , and the accompanying Figure 6 provides an exemplary scheme to better explain the most relevant concepts. As mentioned before, changing the DoV of a generic camera can be achieved by rotating the camera around an axis passing through its projection center. In the specific case of an endoscopic camera, the lens cut β from a real camera with

number

number

number

number

number

number

[0062] As shown in Figure 6, the reference plane

number

number

[0063] In an ideal situation, a flat surface

number

number

number

number

number

[0064] Alternatively, in ideal conditions, points Q, O in the image at pixel 30 i , and P i is the boundary center C i Since it is collinear with line n i 38 is the point Q and O i or C i and O ican be obtained by determining the line passing through points Q and C in the previous equation, respectively. i But, P i Replace.

[0065] Further considerations

[0066] Without loss of generality, in this disclosure, it is assumed that all measurements and calculations are performed in a dry environment. Adaptation to a wet environment can be performed straightforwardly by multiplying the focal length by the ratio of the refractive indices of two or more media through which the light travels before reaching the imaging sensor.

[0067] Another important consideration is the update of the camera model c and the image resolution of line n at pixel 30 in every frame i. i 38 detections are performed at an azimuth angle δ relative to the reference angular position α0. i This can be done by determining the angular displacement of the center C of every frame i, which can be achieved using exclusive image processing techniques. In this disclosure, without loss of generality, for this task, i and Notch P i The method disclosed in U.S. Patent Application No. 62 / 911,950 for detecting the boundary between the FSM and the notch is considered to be employed. This method contemplates the possibility that the FSM may contain multiple notches, which is useful for ensuring that at least one notch is always visible in the image and for being able to detect multiple notches whose relative locations are known and which are distinguished by their different shapes and sizes. Therefore, the angular displacement δ i is the point P i , Q, and

number

number

[0068] As discussed in §

[0054] , the distance r measured in the image at pixel 30 between any point and principal point O corresponds to a 3D angle θ, which depends on both f and ξ and can be calculated using Equation 1. Let d be the length of the line segment whose endpoints are the two intersections of the boundary with line n38 in Figure 7 that passes through principal point O. This length d is called the diameter, even though it does not correspond to the diameter of the circular boundary, because it occurs only when principal point O coincides with boundary center C. As shown in Figure 7, d is determined by the lower and upper radii: d = r l +r u It is the addition of r l and r u Using both, the angle θ l and θ u can be found from Equation 1. The field of view (FoV) of the camera is defined as the addition of the following two angles: Θ=θ l +θ u Any line n passing through the principal point O defines a different diameter d, and Figure 7 shows the particular case of n passing through the notch P.

[0069] Methods for changing the direction of view (DoV)

[0070] This section presents a method disclosed in this disclosure for modifying the DoV by rendering a video that would be acquired by a virtual camera with predetermined characteristics located in the same 3D position as a real endoscopic camera. In particular, i is a frame i acquired by a real endoscopic camera, hence called the source camera, whose endoscope has a lens cut β and rotates in azimuth around a mechanical axis that intersects the image plane at a center of rotation Q. The calibration of the endoscopic camera with respect to a reference angular position α0, corresponding to a particular notch position P, is known, which means that its focal length f, radial distortion ξ, and principal point O have been determined. The goal of this method is to determine the lens cut

number

number

number

number

number

number

[0071] As shown in Figure 8, the method i Center C i and Notch P i We start by finding the boundary between point P and i , Q, and P, as the angle subtended by i By determining the rotation at i By rotating O by the principal point O i This allows us to estimate the location of point x in the normal image, which is the pinhole projection of point X in the 3D scene, by u=c s (x;f,O i ,ξ) to a point in the image u at a pixel of the source camera c s An updated model for is provided.

[0072] Line n iis the image plane in §§

[0060] ~

[0064]

number

number

number

number

number

number

number

[0073] The point in the canonical image

number

number

number

number

number

[0079] to

[0091] .

[0074] As a final step, the algorithm uses image warping techniques to warp the target image, as described in §§

[0043] to

[0048] .

number

number

number

number

number

[0075] The disclosed method for modifying the DoV assumes that the calibration parameters of the source camera relative to a reference position at azimuth angle α are known in advance. These can be determined in several different ways, including, but not limited to, using a set of calibration parameters that are predefined at the factory or that represent a set of similar endoscopic cameras, or using a suitable calibration method for performing calibration in the operating room prior to a medical procedure, such as that disclosed in U.S. Patent No. 9,438,897 (Serial No. 14 / 234,907), entitled "Method and Apparatus for Automatic Camera Calibration Using One or More Images of a Checkerboard Pattern."

[0076] Furthermore, the method of the present disclosure also assumes that the center of rotation Q is known a priori. However, this is not possible using the method disclosed in U.S. Application No. 62 / 911,950 to determine the center of rotation P detected in successive frames. i and / or C i is not a strict requirement, since Q can be determined on the fly from . Another alternative is to use an estimate for the principal point O at each frame instant i by using i In this case, the position of O i is obtained directly from a normalized estimate of the principal point, as disclosed in U.S. application Ser. No. 62 / 911,950, and therefore the center of rotation Q, or a priori azimuth angle δ i It is not necessary to know the angular displacement at

[0077] The azimuth angle δ shown in the second block of the diagram in FIG. iEstimation of the angular displacement of the scope is performed using exclusive image processing methods. However, there are alternative estimation methods that can be employed to measure the rotational change in azimuth angle of the scope relative to the camera head at every frame instant. These include the use of sensing devices such as rotary encoders or optical tracking systems attached to the camera head to determine the position of optical markers attached to the scope cylinder.

[0078] The methods for changing the DoV disclosed herein can be used for different purposes by setting the parameters of the target camera to desired values.

number

number

number

number

number

number

number

[0079] Adjusting the target camera model

[0080] The disclosed warping function w is a function of three functions: s and the tilt γ and orientation of the reference plane relative to the camera head.

number

number

number

number

[0081] The first two functions are fully determined while the focal length of the target camera is

number

number

[0082] Function c sSince both θ and m are known, where the principal points of the target image are mapped by the warp function w in the source image

number

number

number

number

[0083] Target Camera C t Parameters for

number

[0084] Configuration A: One possibility is the center of the boundary.

number

number

number

number

number

number

number

[0085]

number

number

number

number

number

[0086] Empty regions occur when the chosen warping function w maps the target image beyond the boundaries of the source image where there is no visual content (Fig. 10(a) left). This is clearly an undesirable artifact that detracts from the user experience of varying DoV via software.

[0087] Configuration B: A possible solution to avoid artifacts in the sky region is to relax the FoV requirement and limit it to a limited viewing angle.

number

number

number

number

number

[0088] Setting C: The present disclosure is desired

number

[0089] For illustrative purposes, limited viewing angles

number

number

number

number

number

number

[0090] Note that if γ<0, the shift of the FoV is from upward to downward and the principal point translates upward, i.e.,

number

[0091]

number

number

[0092] Uses and Functions

[0093] The disclosed image processing method for changing the DoV of an endoscopic system using interchangeable rotatable optical elements lends itself to multiple applications and can be employed in several different systems, and this disclosure describes some embodiments of these systems and applications without prejudice to other possible applications.

[0094] Electronic switch between rigid endoscopes with different lens cuts β

[0095] Manufacturers of rigid endoscopes offer lenses with different angular offsets from the mechanical axis; the most common lens cuts are β = 0°, 30°, 45°, and 70°. While surgical procedures typically favor endoscopes with specific cut angles, there are certain surgical moments or steps where a different lens cut would be more convenient. For example, in knee arthroscopy, an arthroscope with a 30° lens cut is desirable, while meniscus examinations of the anterior or intercondylar region benefit from a 0° or 70° cut angle.

[0096] The problem is that because a lateral change of DoV requires a physical switch of endoscopes, which can cause breakage and pose risks to the patient, surgeons rarely actually do it and perform the procedure with the same endoscope, even when visualization is suboptimal.

[0097] The disclosed method can be used in a system to process images and videos acquired by an endoscopic camera with a lens having a wide FoV to empower a surgeon with an electronic switch between two or more virtual endoscopes with different lens cuts β. Such a system overcomes the aforementioned difficulties that prevent a surgeon from obtaining the best visualization at every surgical moment or step.

[0098] Two desired virtual endoscopes, each with a lens cut

number

number

number

[0099] An embodiment is to use the disclosed method in a system to electronically switch between the two most common cut angles used in arthroscopy. A standard 30° arthroscope is:

number

number

number

[0100] Electronic change of DoV, distortion correction and directional zoom

[0101] So far, we have used the method in Figure 8 to select a portion of the source camera's FoV to mimic the video output that would be acquired by a camera with a smaller FoV and a different lens cut placed in the same 3D location.

[0102] Another possible embodiment or application is to use the disclosed method to shift the DoV of a particular endoscopic camera while maintaining the overall FoV, where the principal point in the rendered image moves toward the periphery as the DoV shift increases (FIG. 14). In this case, the disclosed method of FIGS. 8 and 11 is implemented in a system connected to a source camera, where the distortion of the target camera is adjusted.

number

number

number

[0103] The described system can be further enhanced by the possibility for the user to control the distortion, in this case:

number

[0104] The disclosed system can also be used to implement what is called directional zoom, where the angular shift γ is adjusted so that the principal point in the rendered image is superimposed with the region of interest (ROI) and distorted.

number

number

[0105] Other uses

[0106] The disclosed methods of the schemes of Figures 8 and 11 may be used to train and / or test machine learning or deep learning based algorithms for automated learning to process individual sets of images for the purpose of rendering images of virtual target cameras with different known calibrations and / or lens cut angles to be used as inputs for identifying and estimating different camera and / or lens characteristics, including, but not limited to, learning to identify lens cuts, estimate calibration parameters, and generate virtual images using different lens cuts.

[0107] The disclosed method can also be applied to other types of images, such as fundus images in ophthalmology.

[0108] The disclosed method for selecting the focal length and location of the principal point in a target image (FIG. 11) is not limited to endoscopic applications, but can be employed for general image and video warping. One possible embodiment is the use of the method in a visual surveillance system to implement electronic pan-and-tilt on images and videos acquired by a fisheye camera.

[0109] 16 is a schematic diagram of an exemplary computing system, including a general-purpose computing system environment 1200, such as a desktop computer, laptop, smartphone, tablet, or any other such device capable of executing instructions, such as those stored in a non-transitory computer-readable medium. Additionally, while described and illustrated in the context of a single computing system 1200, those skilled in the art will also understand that various tasks described below may be performed in a distributed environment having multiple computing systems 1200 linked via a local or wide area network, where executable instructions may be associated with and / or executed by one or more of the multiple computing systems 1200. The computing system environment 1200, or portions thereof, may find use with the processes, methods, and computational steps of the present disclosure.

[0110] In its most basic configuration, computing system environment 1200 typically includes at least one processing unit 1202 and at least one memory 1204, which may be coupled via a bus 1206. Depending on the exact configuration and type of computing system environment, memory 1204 may be volatile (e.g., RAM 1210, etc.), non-volatile (e.g., ROM 1208, flash memory, etc.), or some combination of the two. Computing system environment 1200 may have additional features and / or functionality. For example, computing system environment 1200 may also include additional storage (removable and / or non-removable), including, but not limited to, magnetic or optical disks, tape drives, and / or flash drives. Such additional memory devices may be accessible to computing system environment 1200, for example, by a hard disk drive interface 1212, a magnetic disk drive interface 1214, and / or an optical disk drive interface 1216. As will be appreciated, each of these devices is coupled to the system bus 1206 and enables reading from and writing to the hard disk drive 1218, the removable magnetic disk drive 1220, and / or the removable optical disk drive 1222, such as a CD / DVD ROM or other optical medium. The drive interfaces and their associated computer-readable media enable non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing system environment 1200. Those skilled in the art will further appreciate that other types of computer-readable media capable of storing data may be used for this same purpose.Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memory, nano drives, memory sticks, other read / write memory and / or read-only memory, and / or any other method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Any such computer storage media may be part of computing system environment 1200.

[0111] A number of program modules may be stored in one or more of the memory / media devices. For example, a basic input / output system (BIOS) 1224, containing the basic routines that help transfer information between elements within the computing system environment 1200, such as during start-up, may be stored in ROM 1208. Similarly, RAM 1210, hard drive 1218, and / or peripheral memory devices may be used to store computer-executable instructions, including an operating system 1226, one or more application programs 1228 (such as applications that implement the methods and processes of the present disclosure), other program modules 1230, and / or program data 1232. Further, computer-executable instructions may be downloaded to the computing environment 1200 as needed, for example, via a network connection.

[0112] End users, e.g., customers, retail clerks, and the like, may enter commands and information into computing system environment 1200 through input devices such as a keyboard 1234 and / or a pointing device 1236. Although not shown, other input devices may include a microphone, joystick, game pad, scanner, etc. These and other input devices would typically be connected to processing unit 1202 by a peripherals interface 1238, which would then be coupled to bus 1206. Input devices may be connected directly or indirectly to processor 1202 via an interface such as, for example, a parallel port, game port, Firewire, or universal serial bus (USB). For displaying information from computing system environment 1200, a monitor 1240 or other type of display device may also be connected to bus 1206 via an interface, such as via a video adapter 1242. In addition to the monitor 1240, computing system environment 1200 may also include other peripheral output devices, not shown, such as speakers and printers.

[0113] The computing system environment 1200 may also utilize logical connections to one or more computing system environments. Communications between the computing system environment 1200 and a remote computing system environment may be exchanged through an additional processing device, such as a network router 1252, responsible for network routing. Communications with the network router 1252 may be implemented through a network interface component 1254. Thus, it will be understood that within such a network environment, e.g., the Internet, the World Wide Web, a LAN, or other similar types of wired or wireless networks, program modules depicted with respect to the computing system environment 1200, or portions thereof, may be stored in memory storage devices of the computing system environment 1200.

[0114] Computing system environment 1200 may also include localization hardware 1256 for determining the location of computing system environment 1200. In an embodiment, localization hardware 1256 may include, for example, only a GPS antenna, an RFID chip or reader, a Wi-Fi antenna, or other computing hardware that may be used to acquire or transmit signals that may be used to determine the location of computing system environment 1200.

[0115] In a first aspect of the present disclosure, a camera head and a point sensor are used to image and point images at a specific azimuth angle α0, with known focal length f, radial distortion ξ, and principal point O. Q and a rigid endoscope having a lens cut β that rotates in azimuth about mechanical axes that intersect at .

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0116] In one embodiment of the first aspect, the azimuth angle α corresponds to a particular notch position P and the source image I i is processed, and the center C i and notch position P i The boundary between the two is detected, and the angular displacement in the azimuth angle is

number

[0117] In one embodiment of the first aspect, the method comprises:

number

number

number

number

[0118] In one embodiment of the first aspect, the region containing meaningful image content can take any desired geometric shape, for example, but not limited to, a cone shape, a rectangle shape, a hexagon shape, or any other polygonal shape.

[0119] In one embodiment of the first aspect, the point Q is located between the points Pi and / or point C i is determined on the fly using detection of , in which case it does not need to be known or determined in advance.

[0120] In one embodiment of the first aspect, the line n i is point O i and Q, or O i and C i is alternatively defined by

[0121] In one embodiment of the first aspect, the position of the principal point is given in a normalized reference frame attached to a circular boundary, where its pixel location O for each frame instant i is i The calculation is based on the center of rotation Q and the azimuth angle δ i This can be achieved without explicitly knowing the angular displacement in

[0122] In one embodiment of the first aspect, the source camera measures the rotation of the scope relative to the camera head and determines the azimuth angle δ in step (i). i The sensor may be equipped with an optical encoder or any other sensing device that estimates the angular displacement at the sensor.

[0123] In one embodiment of the first aspect, the principal point O coincides with the center of rotation Q, in which case the camera model of the source camera does not need to be updated at every frame instant, as in step (ii).

[0124] In one embodiment of the first aspect, the distortion of the target camera

number

number

[0125] In one embodiment of the first aspect, the target image in step (vi)

number

[0126] In one embodiment of the first aspect, in step (v), the principal point is

number

number

number

number

[0127] In one embodiment of the first aspect, in step (v), the principal point is

number

number

number

number

number

number

number

number

number

number

number

number

number

[0128] In one embodiment of the first aspect, in step (vi), the focal length

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0129] In one embodiment of the first aspect, the parameters of the target camera may be arbitrary or may be predefined to achieve a specific purpose, and the cut angle

number

[0130] In one embodiment of the first aspect, the method includes: cutting a lens; and

number

[0131] In one embodiment of the first aspect, the source camera is configured to

number

number

[0132] In one embodiment of the first aspect,

number

number

number

number

number

[0133] In an embodiment of the first aspect, the strain

number

[0134] In one embodiment of the first aspect, regardless of the selected angular shift γ, the distortion is

number

[0135] In one embodiment of the first aspect, the principal point is located within the region of interest of the target image, and the radial distortion parameter

number

[0136] In one embodiment of the first aspect, the source and / or target cameras have radial distortions that are described by any of the models known in the literature, including, but not limited to, a learning-based model, a Brownian polynomial model, a rational model, a fisheye model, or a division model.

[0137] In one embodiment of the first aspect, a central C i and Notch P i The boundary between is detected using generalized cone detection methods, using machine learning or deep learning techniques, or using any other image processing technique.

[0138] In one embodiment of the first aspect, a plurality of notch positions are detected, and a particular notch among the plurality of notches is located at point P i , Q, P, such as the angle defined by the azimuth angle δ i is used to determine the angular displacement at

[0139] In one embodiment of the first aspect, the field of view of the virtual target camera

number

number

number

[0140] In one embodiment of the first aspect, the field of view of the virtual target camera

number

number

number

[0141] In one embodiment of the first aspect, the field of view of the virtual target camera

number

number

number

[0142] In one embodiment of the first aspect, the field of view of the virtual target camera

number

number

number

[0143] In one embodiment of the first aspect, the field of view of the virtual target camera

number

number

number

number

number

[0144] While various embodiments have been described for purposes of this disclosure, such embodiments should not be construed as limiting the teachings of the present disclosure to those embodiments. Various changes and modifications can be made to the elements and operations described above to obtain results that remain within the scope of the systems and processes described in this disclosure. All patents, patent applications, and published references cited herein are incorporated herein by reference in their entirety. It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations, set forth merely for a clear understanding of the principles of the present disclosure. Many variations and modifications can be made to the above-described embodiments without substantially departing from the spirit and principles of the present disclosure. It will be understood that some of the other features and functions disclosed above, or alternatives thereof, can be desirably combined into many other different systems or applications. All such modifications and variations are intended to be included herein within the scope of the present disclosure, as defined by the appended claims.

[0145] The described embodiments are to be considered in all respects only as illustrative and not restrictive, the scope of the disclosed embodiments being accordingly indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are intended to be embraced within their scope. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as a departure from the scope of the disclosed system and / or method. [Additional note 1] Source image I captured by the real source camera i Based on the virtual target camera image

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

Claims

1. The source image I captured by the real source camera ｉ The target image of the virtual target camera based on [Equation 1] 1. A method of operating a system for rendering an image, comprising: a source camera having a camera head and a point image, at a particular azimuth angle α, with known focal length f, radial distortion ξ, and principal point O; Ｑ and a rigid endoscope having a lens cut β that rotates in azimuth about mechanical axes that intersect at Azimuth angle δ ｉ = α ｉ -α ０ By rotating O around Q by an angular displacement in ｉ the source image I having ｉ Principal point O at ｉ Finding a location for The focal length f, the radial distortion ξ, and the principal point O ｉ a camera model c of the source camera that maps points x in the normal image to points u in the pixel image according to the location of ｓ and updating Line n ｉ and backprojecting to find a vertical plane that contains or closely passes through the mechanical and optical axes of the source camera. [Equation 2] determining a 3D location of [Equation 3] is the source image I ｉ and determining whether or not [Equation 4] The 3D motion m between the target camera and the source camera is calculated based on the vertical plane [Equation 5] The direction perpendicular to [Equation 6] Angle around [Equation 7] and defining the rotation as [Equation 8] is a point in the normal image of the target camera; Focal length for the target camera [Equation 9] and the principal point [Equation 10] Calculate the location of the focal length [0011] and the principal point [0012] According to the location of the points in the normal image of the target image, [0013] the pixel image of the target image [0014] Camera model c ｔ and [Equation 15] Multiple pixels within [0016] the camera model c of the source camera ｓ the 3D motion m and the camera model c of the target camera ｔ and the source image I ｉ to a point u in the target image [Equation 17] and generating a

2. The azimuth angle α ０ corresponds to the first notch location P, and the method comprises: The source image I ｉ By processing the center C ｉ and the second notch position P ｉ and detecting a boundary between the Rotating O around Q by angular displacement in azimuth angle causes the first notch position P, the second notch position P ｉ estimating the angular displacement in azimuth according to the point Q; The line n ｉ But point O ｉ and P ｉ The method of claim 1 , wherein the saturation is defined by:

3. the rendered target image [Equation 18] Process and center [Equation 19] and diameter [Equation 20] and creating a black frame that defines an image area having a point within said circular boundary. [Equation 21] creating a notch by placing a visual mark on the ｉ is image line n ｉ The method of claim 2 , wherein the 2D unit direction is

4. The method of claim 3 , wherein the image region comprises one or more of a circular shape, a conical shape, a rectangular shape, a hexagonal shape, or another polygonal shape.

5. The source image I ｉ comprises two or more source image frames, and point Q is located at point P in successive ones of said two or more frames. ｉ or point C ｉ The method of claim 2 , wherein the determination is made by detecting one or more of:

6. The source image I ｉ By processing the center C ｉ and the second notch position P ｉ and detecting the boundary between the plurality of second notch positions P ｉ and detecting one of the plurality of second notch positions at an azimuth angle δ ｉ The method of claim 2 , wherein the method is used to determine the angular displacement at

7. The azimuth angle α ０ corresponds to the first notch location P, and the method comprises: The source image I ｉ By processing the center C ｉ and the second notch position P ｉ and detecting a boundary between the Rotating O around Q by angular displacement in azimuth angle causes the first notch position P, the second notch position P ｉ estimating the angular displacement in azimuth according to the point Q; Line n ｉ but, Point O ｉ and Q, or Point O ｉ and C ｉ The method of claim 1 , wherein the saturation is defined by:

8. The method of claim 1 , wherein the location of the principal point is given in a normalized reference frame located within or attached to the endoscope.

9. The source camera measures the rotation of the endoscope relative to the camera head and determines an azimuth angle δ ｉ The method of claim 1 , further comprising a sensor for estimating the angular displacement at

10. The method of claim 1 , wherein the principal point O coincides with the center of rotation Q.

11. 5. The target camera [Equation 22] The method of claim 1 , wherein the distortion of is set to zero.

12. the target image [Equation 23] 2. The method of claim 1 , wherein generating {overscore (x)} is performed using image warping or pixel value interpolation, including one or more of nearest neighbor interpolation, bilinear interpolation, or bispherical interpolation.

13. The principal point [0000] is the center of the boundary of the target image [Equation 25] and the focal length [Equation 26] To calculate [0000] 2. The method of claim 1, comprising solving: where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the backprojection rays.

14. The principal point [0000] is the center of the boundary of the target image [0000] and wherein the method comprises: The camera model c ｓ and the motion m, the origin [0,0] of the source image is calculated by a function g. Ｔ Transform the principal point [Equation 30] a location in the source image that maps to [Equation 31] and finding [Equation 32] and the back-projected light beam of [Equation 33] the limiting viewing angle between the point within the circular boundary closest to [Equation 34] and determining [Equation 35] If the camera field of view is [Equation 36] and reducing the formula [Equation 37] By solving [Equation 38] where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the backprojection rays; [Number 39] When [Equation 40] By solving [Equation 41] 10. The method of claim 1, comprising: obtaining a value of Φ = Φ ⁢ ...

15. The camera model c ｓ and the motion m, the origin [0,0] of the source image is calculated by a function g. Ｔ Transform the principal point [Equation 42] a location in the source image that maps to [Equation 43] and finding [Equation 44] and the back-projected light beam of [Equation 45] the limiting viewing angle between the point within the circular boundary closest to [Equation 46] and determining [Equation 47] If the focal length [Number 48] To solve the following simultaneous equations, [Number 49] ； and [Number 50] If [Equation 51] Set [Number 52] By solving [Number 53] 10. The method of claim 1, further comprising: determining Φ = Φ ( Φ ) , where Φ is a mathematical expression related to focal length, radial distortion, image distance, and the angle between backprojection rays, all of which are interdependent parameters.

Images