Systems and methods for changing a view direction during a video-guided clinical procedure using real-time image processing

Abstract

Arthroscopy and laparoscopy have several lens incisions to cover different clinical situations, in order to change the camera's view direction, the surgeon has to change the optics. The presently disclosed embodiments disclose real-time image processing systems and methods that enable the user to arbitrarily change the view direction of a surgical camera, with the advantage of avoiding workflow interruptions caused by the physical change of said optics. In addition, the presently disclosed embodiments disclose zooming along an arbitrary viewing direction (directional zooming), enabling the increase of the scale of the area of interest without reducing the total field of view or losing image content.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 62 / 911,986 (“'986 Application”), filed October 7, 2019, and U.S. Application No. 62 / 911,950 (“'950 Application”), filed October 7, 2019. Both '986 Application and '950 Application are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] This disclosure generally relates to the fields of computer vision and image processing, and specifically, but not limitingly, the disclosed embodiments presented can be used to enhance visualization in video-guided minimally invasive clinical surgeries for surgical and diagnostic purposes, such as arthroscopy, laparoscopy, or endoscopy, for the purpose of changing the viewing direction of a surgical camera with an endoscopic lens, in which case the system reproduces a view acquired by a physical observation instrument having a lens incision different from the lens incision actually used, or performs zoom along any viewing direction without reducing the image field of view or losing image content. Background Technology

[0004] In video-guided surgery, such as arthroscopy or laparoscopy, access to the anatomical cavity of interest is made through a small incision designated as the surgical opening. One of these openings allows access to a camera equipped with a rigid endoscope, which enables the surgeon to visualize the interior of the cavity to provide guidance during surgical or diagnostic procedures. A rigid endoscope is a long, tubular structure inserted into a body cavity. An endoscope typically includes a distal objective lens, an image relay system (e.g., a series of spaced-apart lenses), and a proximal eyepiece, which typically houses a camera component, such as a charge-coupled device (CCD) chip. The image relay system transmits images from the distal end to the proximal end. The camera receives this image and generates a signal for a video display for the surgeon to observe.

[0005] Rigid endoscopes, also referred to herein as lens observation instruments or endoscopic lenses, are used in various specialties and, depending on their characteristics and application areas, can be alternatively named arthroscopy (orthopedic surgery), laparoscopy (abdominal surgery), cystoscopy (bladder), sinusoscopy (otolaryngology), neuroscopy, etc. Endoscopic cameras, resulting from the combination of rigid endoscopes and camera components, possess specific features not commonly found in conventional cameras. For ease of sterilization, the optics are typically replaceable, and the surgeon assembles the endoscope into the camera before initiating surgery. The eyepiece (or eyepiece adapter) at the proximal endoscope is assembled to the camera using a connector that typically allows the user to rotate the observation instrument relative to the camera by an angle α. This is an azimuth rotation about the longitudinal axis of the endoscope, which is therefore referred to as the mechanical axis, and which approximates but does not necessarily coincide with the axis of symmetry of the tubular structure.

[0006] Rigid endoscopes typically have a field-stop mask (FSM) somewhere along the image forwarding system. This FSM causes the acquired image to contain meaningful content within a circular region surrounded by a black frame (shown diagonally in the accompanying drawings). The FSM is designed such that a mark is typically present within the circular boundary defined by the image-frame conversion, allowing the surgeon to infer the azimuth angle α. This mark around the circular image will thereafter be referred to as a notch.

[0007] Furthermore, the mounting method of the distal endoscope lens typically results in an angular offset between the optical axis of the entire lens system and the longitudinal or mechanical axis of the observation instrument. This angle β between the optical and mechanical axes is commonly referred to as the endoscope's lens slit. If the lens slit is not zero, the optical and mechanical axes are misaligned, and the surgeon can change the azimuth angle of view by rotating the observation instrument around the mechanical axis without having to move the camera.

[0008] Variable direction of view (DoV) allows surgeons to change the direction of endoscopic observation without having to change the position of the endoscope itself. This is helpful when the endoscope axis cannot be easily moved due to anatomical constraints or constraints imposed by other surgical instruments in the surgical area, facilitating the visualization of structures adjacent to or behind the endoscope tip. Therefore, variable DoV is undoubtedly a desirable feature that provides greater flexibility for the surgeon's surgical approach. As discussed, rigid endoscopes currently used clinically allow surgeons to change the DoV by rotating the azimuth of the lens observation instrument, thereby allowing the optical axis to describe in space a cone (the cone of DoV) with its apex near the projection center O (the cone of DoV). The half-angle of this cone is defined by the lens incision β, which is predetermined and known to the surgeon. Surgeons rely heavily on prior knowledge of the lens incision of a particular endoscope to reliably understand what the anatomical structures should look like.

[0009] Some endoscopes have different angular offsets from the longitudinal or mechanical axis, with the most commonly used lens incisions being 0°, 30°, 45°, and 70°. Surgical procedures typically require endoscopes with most of these incision angles, with particular emphasis on one. For example, in knee arthroscopy, a 30° lens incision is preferred, providing good anterior and some lateral visibility, while in hip arthroscopy, due to the much narrower anatomical cavity, a 70° lens incision is usually chosen. Similarly, the preferred lens incision in laparoscopy can vary between 0° (anterior viewing instrument) and 30°. However, despite the emphasis on specific lens incision angles, most procedures benefit from different lateral and partially posterior viewing angles at different times or steps of the surgical procedure. Unfortunately, because lateral changes in DoV require physically replacing the endoscope with one of different lens incisions, surgeons rarely do this in practice, even though they are aware that such a change would improve visualization for the task at hand. Changing endoscopes mid-surgery is cumbersome (light and camera cables must be disconnected and reconnected), time-consuming, and sometimes dangerous. For example, inserting angled endoscopes can be dangerous because they are not facing the direction in which they are inserted, and neurosurgeons often avoid using endoscopes with 45° or 70° lens incisions because they are afraid of blindly pushing the endoscope into delicate tissues.

[0010] In summary, rigid endoscopes commonly used in medicine allow surgeons to change the DoV (angle α) rather than the tilt or elevation angles (angle β). In this case, the optical axis is constrained to move within a cone (DoV cone) whose axis of symmetry is aligned with the mechanical axis and whose angle is defined by the lens incision β. As can be seen from the discussion, unconstrained variation of the DoV is highly advantageous, allowing surgeons to change the lateral viewing angle (lens incision β) without physically changing the endoscope mid-operation. Several specially designed endoscopes have been disclosed that allow for angular shifts in both azimuth and tilt to change the viewing direction (US2016 / 02.09A1, US20050177026A1, US2012 / 0078049, US9,307,892). Unfortunately, these endoscopes use movable image sensors or optics at the distal end to change the lens incision β. Due to these moving parts, manufacturing these instruments is complex and expensive, and such instruments are not as robust as traditional fixed-lens-incision instruments. Moreover, they typically offer lower lighting and image quality.

[0011] An alternative approach to achieving unconstrained variations in DoV is to employ computational methods to process endoscopic images or videos. In the field of computer vision, it is well known that a change in DoV corresponds to a rotation of the camera about an axis passing through its projection center, and a virtual image acquired by such a rotated camera can be reproduced by distorting the real source image through a suitable homography mapping, often referred to as an infinite plane. This method is used in patent applications US005313306A, US20154 / 0065799A1, and EP3130276A1, which disclose rigid endoscopes with wide-angle lenses for a hemispherical field of view (FoV), combined with computer components to reproduce images using a variable DoV covering selected portions of the FoV. As a result, the camera system has controllable pan and tilt directions, and the user can electronically control the direction of observation in the X (azimuth) and Y (elevation) angles.

[0012] One problem with these systems is that, in the context of endoscopic and video-guided clinical surgery, surgeons are accustomed to changing the azimuth of the observation direction by simply rotating the lens relative to the camera. Therefore, moving control from mechanical to electronic control can be challenging, requires surgical retraining, and has become a major obstacle to widespread adoption.

[0013] Therefore, what is needed is an endoscope system with a DoV that can be changed in azimuth angle α and tilt angle β, wherein the change in azimuth angle is achieved by standard mechanical means, namely by rotating a rigid endoscope relative to a camera, while the change in tilt angle is achieved by electronic or computer means, namely by reproducing an image with the target viewing direction through real-time processing.

[0014] There is also a need for an endoscope system that operates in a standard manner, but allows the user to electronically switch between rigid endoscopes with different lens cutouts to always obtain optimal visualization conditions.

[0015] Additionally, there is a need for an endoscope system in which the user can control the amount of radial distortion of the image to obtain optimal scene depth perception using a distortion-free image, or to select the region of interest by changing the DoV and magnifying it, while preserving the FoV and maintaining all visible content in the image. Summary of the Invention

[0016] Systems and methods that use real-time image processing to change the direction of view during video-guided clinical surgery and diagnosis, while allowing surgeons to physically rotate the lens relative to the camera to observe the instrument, are currently common practices in various surgical procedures such as arthroscopy, laparoscopy, and endoscopy.

[0017] The currently disclosed embodiments disclose a software-based system capable of changing the viewing direction of a specific observation instrument by an arbitrary amount γ, such that the notch angle of the lens (or lens notch) actually switches from β to β+γ. This is achieved in a seamless, fully automated manner, allowing the user to physically rotate the endoscope in azimuth angles around its longitudinal axis during operation.

[0018] The software-based system processes images and videos acquired by an endoscopic camera equipped with a lens with a wide field of view (FoV), enabling surgeons to electronically switch between two or more virtual endoscopes with different lens incisions.

[0019] Additionally, the currently disclosed embodiments disclose a method for, given the desired lens cutouts of two target virtual endoscopes. and as well as and The lens cutout β and FoVΘ of the source camera are determined such that the acquired images and videos are well-suited for use as input in the disclosed image processing method for reproducing real virtual images and videos.

[0020] The currently disclosed embodiments disclose a method that addresses not only the desired offset of the lens notch γ, but also the image resolution m x n and field of view. and image distortion In terms of quantity, given any preset for the virtual camera, the method automatically adjusts the principal point at each frame time. focal length The position ensures that the corresponding virtual view is reproduced correctly, without any empty top or bottom areas (empty areas) without content.

[0021] The currently disclosed embodiments disclose a method capable of controlling the amount of radial distortion in an image to achieve optimal scene depth perception using a distortion-free image, or by changing the radial distortion. The amount of image zoom is performed along any viewing direction to select the area of ​​interest. In this case, zooming can be done without loss of field of view (FoV), or by changing the focal length. To complete. Attached Figure Description

[0022] To gain a more complete understanding of this disclosure, reference is made to the following detailed description of exemplary embodiments considered in conjunction with the accompanying drawings.

[0023] Figure 1This is an embodiment of an endoscopic camera that includes an endoscope and a camera, and all its components are shown. A canonical image formed by an endoscope that includes a field-stop mask (FSM) in its image forwarding system is depicted, as well as an image in pixels obtained after applying inline function parameters to a sensor in the camera.

[0024] Figure 2 This is an embodiment of an endoscope, illustrating two different DoVs produced by two different cones of sight (DoV). As the endoscope rotates about its mechanical axis, each cone corresponds to a trajectory described by the optical axis.

[0025] Figure 3 The image warping process between the source and target images is illustrated, where the colors of pixels in the target image are obtained using an interpolation scheme. The warping function is the source camera model c with motion model m. s and target camera model c t The combination of .

[0026] Figure 4 The process of obtaining the camera model c is illustrated graphically. A 3D point X is projected onto the image plane based on the pinhole model, thus generating a 2D point x in the canonical image. Next, distortion d is an operation performed in the canonical image that applies distortion to point x, generating point x'. Finally, an inline function k is applied to convert the point x' (in millimeters) into a point u in the image (in pixels). The camera model c is a function applied in 2D space that converts the point x in the canonical image into a point u in the image (in pixels), u = c(x; f, 0, ξ), and the camera model is obtained by combining distortion d and the inline function k.

[0027] Figure 5 The effect of rotating the observation instrument at the boundary position and principal point O is shown. The observation instrument rotates about a mechanical axis that intersects the image plane at the rotation center Q. The FSM is projected onto the image plane as a circle with center C, and the mark in the FSM is projected onto point P, called the notch. When the observation instrument rotates at an angle δ... i During rotation, the center C of the circle, the notch P, and the principal point O also rotate around Q by the same amount δi.

[0028] Figure 6 The relevant entities are shown when estimating the motion model m, which is about an axis. Angle 3D rotation. This rotation is something a real camera with a lens cutout β must perform to achieve the lens cutout. The camera virtualizes virtual motion. This is achieved by detecting the line n in each frame i. iAnd project it into 3D space, estimating its perpendicularity to the reference plane Π. i direction

[0029] Figure 7 The concept of diameter d is illustrated, which is the length of a line segment whose endpoint is the intersection of a straight line containing the principal point O and the boundary. Diameter d is used to determine the camera's field of view (FoV) Θ. The figure shows a specific case P where a line passes through a notch.

[0030] Figure 8 The method for changing the viewing direction disclosed in this disclosure is illustrated graphically by showing the steps that allow for video reproduction, wherein the video is acquired by a virtual camera with predefined features located at the same 3D position as a real endoscopic camera. The output of this method is a mapping function w, which maps pixels in the target image acquired by the virtual camera. Converted to pixels u in the source image captured by a real camera.

[0031] Figure 9 Explain how to calculate points in the source image. The point mentioned is the principal point on which the distortion function will map the target image. The starting point [0,0] of the reference frame of the target camera's canonical image can be transformed using the function g. T Determine The function is the camera model c. s and movement m The combination of . Known points Allow estimation for Figure 10 This method aims to prevent the presence of empty areas in the target image that limit the viewing angle.

[0032] Figure 10 A strategy is shown to prevent images reproduced by the target camera from having areas (empty areas) with no visual content. Figure 10 (a) shows that a portion of the interior of the target boundary is mapped beyond the boundary of the source image, resulting in an empty area at the bottom with no image content. Figure 10 (b) shows the method for estimating focal length. This approach aims to move the point furthest from the source boundary to coincide with it, resulting in an easily solvable solution. Unfortunately, this reduces the FoV, which is undesirable. Figure 10 (c) illustrates the proposed solution, in which the focal length is adjusted simultaneously. and the main point Both aim to remove empty regions while avoiding a desired decrease in FoV.

[0033] Figure 11The present disclosure describes different steps of a method for estimating the focal length and principal point of a target image, which coordinates variations in DoV with the reproduction of an image having the desired FoV and no empty regions.

[0034] Figure 12 This demonstrates how to set the size of the source camera so that it appropriately fits the lens cutout and FoV of the target camera by reproducing a realistic target image with no empty areas and a principal point close to the center of the image.

[0035] Figure 13 The image results reproduce two images obtained by two endoscopes with different lens incisions and FoVs. The upper image shows the image obtained by a real camera, which is a laparoscope with a lens incision β = 45° and a diameter of 10 mm. The middle image shows a virtual view obtained for γ1 = -15°, in this case with a lens incision. and The endoscope is virtualized. The lower figure depicts the virtual view obtained at γ2 = 25°, which means the endoscope camera is relative to a view with... and The observation instrument's axis of symmetry is pointing downwards.

[0036] Figure 14 The image shows the results of changing the DoV by angle γ when the parameters of the target camera are predefined to be the same as the calibration estimates of the source camera. The upper image is the original true image obtained by an arthroscope with a lens notch β = 30°. The middle image shows a virtual view obtained for γ = -30°, in which case the arthroscope is looking forward along its axis of symmetry (β + γ = 0°). The lower image depicts a virtual view obtained for γ = 35°, which means the camera is looking downwards with β + γ = 65° relative to the axis of symmetry of the observation instrument. Solid circles represent... The position of the point is actually the point in the source image that maps to the principal point of the target image, indicating that the change in DoV causes the principal point to shift.

[0037] Figure 15 This diagram illustrates the image results of directional zoom, which allows for magnification of the image around any viewing direction while preserving the overall FoV and image content. The top image shows the original view. The middle image shows the reproduction result for β = 65°, where the target camera's preset is the source camera's calibration. The bottom image shows the radial distortion when the lens cutout is the same as in the middle image. A virtual view of the situation. It can be observed that by increasing radial distortion, it is possible to magnify the region of interest while preserving all image content appearing in the upper image. Solid circles represent... The position of the point is actually the point in the source image that maps to the principal point of the target image, indicating that the change in DoV causes the principal point to shift.

[0038] Figure 16 It is a diagrammatic view of an example computing system that includes a general computing system environment. Detailed Implementation

[0039] It should be understood that although illustrative embodiments of one or more examples are provided below, various specific embodiments may be implemented using any number of techniques known to those skilled in the art. This disclosure should in no way be limited to the illustrative embodiments, drawings, and / or techniques shown below, but includes exemplary designs and embodiments illustrated and described herein. Furthermore, this disclosure may be modified within the scope of the appended claims and their full equivalents.

[0040] This document discloses systems and methods for altering the line of sight during video-guided clinical surgery. These systems and methods can be used in clinical surgeries, including but not limited to arthroscopy, laparoscopy, endoscopy, or other surgical procedures involving minimally invasive orthopedic surgery. The systems and methods can be used in conjunction with real-time image processing or delayed image processing.

[0041] Figure 1 The endoscopic camera 34 used in the aforementioned video-guided clinical surgery is illustrated in diagram form. The endoscopic camera comprises an endoscope 10, the proximal end 16 of which is assembled to a camera 28 via a connector that allows the endoscope 10 to rotate about a mechanical axis 12 in an azimuth angle of α18. The endoscope 10 typically includes a field-stop mask (FSM) along an image forwarding system, which causes light forwarded from the distal end 14 to the proximal end 16 to form a canonical image 22 including a circular boundary 24 and a notch 26. The camera 28 converts the canonical image 22 into an image in pixels 30, which also includes the circular boundary 24 and the notch 26. Furthermore, and as discussed in the background section, because the lens at the distal end 14 causes the optical axis 36 and the mechanical axis 12 to be misaligned at an angle β, represented as a lens notch 20, the field of view (DoV) 36 changes when the endoscope 10 is rotated relative to the camera 28.

[0042] Rotation of the mechanical axis 12 of the endoscope 10 in the azimuth angle 18 causes the optical axis 36 to describe a cone (DoV cone) in space 32, half of which is the lens notch 20, as... Figure 2 As shown.

[0043] In this disclosure, 2D and 3D vectors are written in bold lowercase and uppercase letters, respectively. Functions are represented in lowercase italic letters, and angles are represented in lowercase Greek letters. Points and other geometric entities in the plane are represented in uniform coordinates, as is commonly done with projected geometry, where 2D linear transformations in the plane are represented by 3x3 matrices, and equations are expressed proportionally. Furthermore, when representing functions, the symbol ; is used to distinguish the function's variables (appearing to the left of the ;) and parameters (appearing to the right of the ;). Finally, the symbol § is used to refer to different parts of the text by paragraph numbering.

[0044] Image distortion

[0045] The disclosed method and system for reproducing virtual views with arbitrary tilt offsets relate to image distortion techniques, particularly to software-based methods for creating virtual PTZ cameras from wide field-of-view (FoV) panoramic cameras. In this case, an image (target image) acquired by a PTZ camera is reproduced from an image acquired by a panoramic camera (source image) by a function that maps pixels in one image to pixels in another image.

[0046] Without loss of generality, let w be the pixel coordinates u in the target image. t Convert to pixel coordinates u in the source image s functions, such as Figure 3 As shown, the pixel u in the target image can be determined by using any type of interpolation method in the spatial or frequency domain. t The color value, wherein the interpolation method includes, but is not limited to, nearest neighbor, next neighbor, previous neighbor, bilinear, bicubic, Lanczos, bicubic B-spline, Mitchell-Netravali, Catmull-Rom, Kriging-based, wavelet-based, or edge-guided interpolation. Data-driven interpolation filters learned using machine learning or deep learning can also be used to obtain u. t The color value.

[0047] Existing warping techniques include, but are not limited to, direct mapping, inverse mapping, warping by resampling in continuous or discrete image domains, warping by resampling and filtering, warping using lookup tables, warping using decomposable transformations, and learned warping transformations.

[0048] The distortion function w is a function c corresponding to the camera models of the source camera and the target camera, respectively. s and c t The combination with m, which is a function of camera motion. Camera model c s and c tThe mapping between the canonical image 22, represented in millimeters, and the image represented in pixels 30 is described. Since the source camera is a real camera, c can be determined using an appropriate calibration method. s On the other hand, select target camera c t This allows for the predefined desired imaging characteristics (resolution, zoom, FoV, etc.). The function m describes the relative motion between the virtual (target) and real (source) cameras. More specifically, it represents the rotation experienced by the virtual camera in 3D space, which produces an in-phase mapping in projected coordinates between the canonical images of the source and target cameras.

[0049] Distorting images acquired with an endoscopic camera is more challenging than distorting images acquired with a conventional camera for two main reasons. First, due to the relative rotation of the endoscope lens relative to the camera, the camera model c... s The distortion changes at every frame, and this must be taken into account when constructing the distortion function w. Secondly, the motion model m depends not only on the change in the desired elevation angle γ, but also on the mechanical change in the azimuth angle δ that must be measured at every frame. Conventional cameras do not present these challenges because they do not have moving parts that interfere with the camera model.

[0050] Endoscopic camera model

[0051] This section provides an overview of the relevant concepts by describing a model of a general-purpose camera that presents radial distortion introduced by optical devices, and explains how an endoscope camera can be described using an adaptive model that is updated at every frame time, introducing the endoscope camera model c.

[0052] General camera model

[0053] Figure 4 Use a diagram to represent the different steps involving the camera model function c. Based on the pinhole model, define the 3D point X = [X YZ]. T Projecting this onto the image plane produces a 2D point x in the canonical image 22. This projection is denoted as p and corresponds to the calculation x = p(X) = [X / Z, Y / Z]. TNext, distortion d is the operation performed on the canonical image 22, which applies distortion to point x to produce x': x' = d(x; ξ), where ξ is the distortion parameter. Finally, an inline function parameter is applied to convert point x' (in millimeters) to point u in the image represented by pixels 30. This is achieved by the function k, which calculates u = k(x'; f, O) = fx' + O, where f is the focal length, O is the principal point, and it is assumed, without loss of generality, that pixels are squared, i.e., the aspect ratio is global, and the skew is zero. If this assumption is not made, function k is simply updated to include two additional parameters that take into account aspect ratio and skew. The camera model c is a function applied in 2D space that converts point x in the canonical image 22 to point u in the image represented by pixels 30, u = c(x; f, O, ξ), and the camera model is obtained by a combination of distortion d and the inline function k:

[0054] In this model, the distortion function d is general, meaning it can be any distortion model in the literature, such as the Brownian polynomial model, the rational number model, the fisheye model, or a segmentation model with one or more parameters, in which case ξ is a scalar or a vector, respectively. The distortion function d can also consider other types of distortion, such as, but not limited to, tangential distortion, prism distortion, or perspective distortion caused by sensor tilt. Without loss of generality, the remainder of the description assumes that d applies a first-order segmentation model, where ξ is a scalar:

[0055] Furthermore, the distance r measured in pixels in the image represented by pixel 30 between known point u and principal point O (r = ||u-0||) corresponds to angle θ in 3D, which depends on the joint effects of f and ξ, such that...

[0056] Φ(f,ξ,θ,r)=0. (1)

[0057] Handling lens rotation

[0058] The rotation of the endoscope 10 relative to the camera 28 causes the principal point O, the center C of the circular boundary 24, and the notch 26, denoted as P, to rotate around point Q in the image represented by pixels 30, as shown. Figure 5As shown. In fact, O and C rotate around Q because they typically do not coincide due to mechanical tolerances during lens manufacturing. O is known to be the image of the optical axis, C is the center of the circular boundary, which depends on the placement of the FSM relative to the lens rod, and Q is the intersection of the mechanical axis 12 with the image plane, which depends on the eyepiece. When the eyepiece and FSM are perfectly aligned with the lens rod, all three points coincide. The misalignment between Q and C causes the boundary to change its position in the image, and C moves around Q in a circle as the lens rotates. When Q and O are misaligned, O also describes a circular trajectory around Q.

[0059] Therefore, if the calibration parameters f, O, and ξ refer to a specific azimuth position α0, then the azimuth rotation angle δ i =α i -α0 changes the position of the principal point to O. i =r(O;δ i , Q), where r represents the angle δ around the axis passing through Q. i The 2D rotation. Similarly, the center of the circular boundary becomes C. i =r(C;δ i Q).

[0060] In other words, let f, O, and ξ be the calibration parameters of a general-purpose endoscope camera at a specific azimuth angle α0. Therefore, the camera model that maps a point x in the standard image to a point u in pixel coordinates can change at each frame time i, as shown by u = c(x; f, O). i Given ξ, where O i =r(O;δ i ,Q)andδ i =α i -α0, where α i It is the azimuth angle in frame i.

[0061] In the remainder of this specification, it is assumed, without loss of generality, that the calibration parameters f, O, and ξ of the reference azimuth position α0 and the rotation center Q are known a priori, such that if the rotation δ relative to α0 is estimated at each frame time i... i Then, the calibration parameters f and O can be determined as described above. i ξ. It is worth noting that if O coincides with Q, then O in each frame i... i This will also coincide with Q, so such adaptation of the camera model c to the rotation of the observation instrument is unnecessary. However, misalignment often occurs between these entities due to mechanical tolerances in the construction of the optics, as discussed previously. Since this misalignment effect in the calibration parameters is not negligible, it must be taken into account in practice.

[0062] Camera rotation in 3D

[0063] The method for determining the motion model m, as described in §§

[0043] -

[0048] , is now described, wherein the accompanying Figure 6 Illustrative schemes are provided to better explain the most relevant concepts. As previously mentioned, the DoV of a general-purpose camera can be changed by rotating the camera around an axis passing through its projection center. In the specific case of our endoscopic camera, a virtual camera with a lens cutout β is created from a real camera with a lens cutout β. The camera corresponds to a change in DoV, which can be achieved by changing the orientation. 42. Virtually rotate the real camera around its projection center in 3D. To achieve this. This direction. 42 perpendicular to plane Π i 40, therefore called the reference plane, is defined by the mechanical axis and the optical axis, which are ideally coplanar but do not coincide due to the non-zero lens notch. Since the optical axis 36 defines the DoV that changes with rotation in the azimuth angle 18, therefore around the axis... The motion model m of the 3D rotation of γ of 42 can change at each frame time i and must be estimated accordingly.

[0064] like Figure 6 As shown, the reference plane Π i 40 lines projected onto an image represented in pixels 30 i 38. Therefore, it is possible to detect the line n in each frame i. i 38 and then back-project it into 3D space to dynamically determine

[0065] Under ideal conditions, plane Π i 40 contains both the optical and mechanical axes and intersects the FSM in the notch, its purpose being to inform the surgeon of the direction of the lens incision. This is true if and only if points Q and O in the image represented by pixel 30... i and P i Collinear, and in this case, line n i 38 is a number that includes all three points Q and O. i and P i These conditions only hold true when the line is straight. In reality, this usually doesn't occur due to mechanical tolerances during lens manufacturing, thus affecting the optical and mechanical axes and / or plane Π. i 40 does not pass exactly through the notch of the FSM due to its non-planarity. Therefore, in the remainder of this specification, without loss of generality, it is assumed that n... i 38 is formed by optical axis O i and notch P iThe defined line. Therefore, in the projected coordinates, through n i =P i ×O i Calculate line n i 38, then you can use a simple and direct method... Determine the plane Π i The normal to 40, where K i It is the matrix of inline function parameters in frame i. And O i =[O ui O vi ] T .

[0066] Alternatively, under ideal conditions, points Q and O in an image represented by pixels 30 i and P i Collinear, boundary center C i This is also true, therefore it can be determined by passing through points Q and O. i Or C i and O i To obtain line n i 38. In this case, points Q and C i Replace P in the previous equation respectively i .

[0067] Further consideration

[0068] In this disclosure, it is assumed that all measurements and calculations are performed in a dry environment without loss of generality. Adaptation to humid environments can be performed in a simple and straightforward manner by multiplying the focal length by the ratio of the refractive indices of the two or more media that travel before the light reaches the imaging sensor.

[0069] Another important consideration is that the angular displacement δ of the azimuth relative to the reference angular position α0 can be determined. i To perform the update of camera model c and detect lines n in the image represented by pixels 30 in each frame i. i 38. This can be achieved using only image processing techniques. In this disclosure, without loss of generality, the method disclosed in U.S. Application No. 62 / 911,950 for detecting a central C in each frame i is considered. i and notch P i This task is performed using a boundary-based approach. This method assumes the possibility that the FSM contains more than one notch, which can be used to ensure that at least one notch is always visible in the image, enabling the detection of multiple notches whose relative positions are known and who are identified by their different shapes and sizes. Therefore, the angular displacement δ i It is from point P i Q and P Define an angle, where P is the position of one of the notches at the reference angle position α0. Alternatively, it can be determined from the boundary center C. i And C estimate δ i Where C is the center of the boundary at the reference position:

[0070] As described in §

[0054] , the distance r between any point and the principal point O measured in the image represented by pixels 30 corresponds to the angle θ in 3D, which depends on f and ξ, and can be calculated using Equation 1. Let d be the endpoints of the boundary and Figure 7 The length of the line segment d that passes through the principal point O and intersects the two points of the circle is called the diameter. This length d is referred to as the diameter even though it does not correspond to the diameter of the circle's boundary, because this only occurs when the principal point O coincides with the boundary center C. For example... Figure 7 As shown, d is the sum of the upper and lower radii: d = r l +r u Using r l and r u Both angle θ can be determined from Equation 1. l and θ u The camera's field of view (FoV) Θ is defined as the sum of these two angles: Θ = θ l +θ u Any line n passing through the principal point O defines a different diameter d, and Figure 7 This illustrates a special case of n passing through notch P.

[0071] Methods to change the view direction (DoV)

[0072] This section describes the method disclosed in this disclosure for altering DoV by reproducing video, which is acquired by a virtual camera with predefined characteristics located at the same 3D position as a real endoscope camera. Specifically, let I i Frame i is acquired by a real endoscopic camera (hereinafter referred to as the source camera), whose endoscope has a lens notch β and is rotated by an azimuth angle about a mechanical axis that intersects the image plane at the rotation center Q. The endoscope camera is known to be calibrated for a reference angular position α0 (corresponding to a specific notch position P), meaning its focal length f, radial distortion ξ, and principal point O have been determined. The aim of this method is to reproduce a resolution of m x n with a diameter of... And by point Image of the central circular boundary The image will be acquired by a virtual camera (hereinafter referred to as the target camera), which has a lens cutout. Field of view and distortion The source camera is positioned in the same 3D location.

[0073] like Figure 8 As shown, the method detects image I i The center C i and notch P i The boundary begins. This is achieved by rotating the azimuth angle δ... i It is determined to be from point P i The angle defined by Q and P can be reached by rotating O around Q to the angle δ. i To estimate the principal point O i The location is as described in §§

[0055] -

[0059] . This provides an updated model c for the source camera. s The model maps a point x in the canonical image (a pinhole projection of point X in the 3D scene) to a point u in the pixel-represented image, such that u = c s (x;f,O i ,ξ).

[0074] The reference plane Π as described in §§

[0060] -

[0064] i The line n projected onto the image plane i It is determined to pass through point O. i and P i The line. As previously described, by n i By projecting backwards into 3D space, a normal vector can be obtained. plane Π i This generates a motion model m, which revolves around an axis. In 3D space, the rotation reaches an angle γ, which corresponds to the difference between the lens cutouts of the target camera and the source camera. The motion model m places points in the canonical images of the target camera and the source camera, respectively. Transform into point x such that...

[0075] To extract the points from its normalized image Points mapped to an image represented in pixels Camera model c of the target camera t The principal point in each frame i must be calculated. focal length And location. For details on the estimation of these parameters, see §§

[0079] -

[0091] .

[0076] As a final step, the algorithm uses image warping techniques to generate the target image. For example, §§

[0043] -

[0048] As stated in the text, among which Each pixel in Mapped to the source image I via mapping function w i Point u in the middle, such that we can... The color values ​​are interpolated using the function c. s m and The combination of these functions. As previously described, this mapping function w can implement any method used for image distortion or pixel value interpolation. The disclosed methods for changing DoV assume that the calibration parameters of the source camera at the reference position in the azimuth angle α0 are known in advance. These can be determined in several different ways, including, but not limited to, using a set of calibration parameters predetermined at the factory or representing a set of similar endoscopic cameras and performing calibration in the operating room before the start of the medical procedure using appropriate calibration methods, such as the method disclosed in U.S. Patent No. 9,438,897 (Application No. 14 / 234,907) entitled "Method and apparatus for automatic camera calibration using one or more images of a checkerboard pattern". Furthermore, the currently disclosed method also assumes that the rotation center Q is known a priori. However, this is not a stringent requirement, as the method disclosed in U.S. Application No. 62 / 911,950 can be used to detect point P from consecutive frames. i and / or C i Q is determined dynamically. Another alternative is to determine the principal point O at time i in each frame by utilizing an estimate of the principal point given in a normalized reference frame attached to the circular boundary. i The location. In this case, due to O i It is obtained directly from the normalized estimate of the principal point, as disclosed in U.S. Application No. 62 / 911,950, therefore it does not require prior knowledge of the rotation center Q or the angular displacement δ of the azimuth. i . Perform using only image processing methods Figure 8 The second box in the figure depicts the angular displacement δ of the azimuth angle. i The estimation is not possible. However, alternative estimation methods exist that can be used to measure the change in the azimuth rotation of the observation instrument relative to the camera at each frame time. This involves using sensing devices, such as a rotary encoder or optical tracking system attached to the camera, to determine the position of an optical marker attached to the observation instrument tube. The method for changing DoV disclosed in this paper can be used for different purposes by setting the parameters of the target camera to the desired values. Specifically, if the distortion of the target camera... When set to zero, the image reproduced by the target camera This will result in distortion-free images. Furthermore, to perfectly mimic real endoscopic cameras, including FSMs, the reproduced target images can be processed. To generate a black frame, so that meaningful image content is centered. And the diameter is Within the circular area, and by placing visual markers at points on the circle's boundary. A notch is created in the middle, where v i yes Figure 8 The line n calculated in box 4 of the graph. i unit direction (v) i =P i -O i / ||P i -O i ||). The image boundary can also take any other desired geometric shape, such as, but not limited to, a conical shape, a rectangular shape, a hexagonal shape, or any other polygonal shape. Adjust the target camera model The disclosed distortion function w is a combination of three functions: the camera model c, whose parameters are the calibration parameters of the real source camera at the current frame i. s It depends on the tilt angle γ of the reference plane relative to the camera and the orientation. The expected change in motion model m, and the camera model c of the virtual target camera. t It must ensure that image distortion is addressed. and image diameter FoV is Although the first two functions are fully determined, the focal length of the target camera still needs to be defined. and the main point To fully specify c t . Since the function c is known s Both m and tw can be used to predict the location of the principal point of the target image that will be mapped to the source image by the warp function w. ( Figure 9 Consider g as c. s The combination with m makes The position can be seen as Where 0 is the starting point [0,0] T This location is not yet selected. and The value is irrelevant. The target camera c can be executed according to three different settings. t parameters and The choice Figure 10 The diagrams for these three settings are shown below. Setting A: One possibility is to choose... With the center of the boundary Coincidence. Due to the virtual center of rotation. It is also assumed to be with The principal point coincides with the focal point, therefore it is fixed on frame i, independent of the lens rotation relative to the camera. In this case, and taking into account the dependency between focal length, distortion, image diameter, and FoV denoted by Φ, the specified FoV is... focal length yes The solution. choose and The problem is that the reconstructed target image is usually in n i Near the intersection with the boundary, there exists an area (empty area) without visual content. Figure 10 (a) Right side). Whenever the angle That is, by and in the source image The restricted viewing angle defined by the back-projection ray at the nearest point on the boundary is smaller than expected. This happens when it's halfway through. Empty regions appear whenever the chosen warp function w maps the target image to the boundary of the source image, which contains no visual content. Figure 10 (a) Left side). This is clearly an unwanted artifact that ruins the user's experience when changing DoV through software. Option B: A possible solution to avoid empty area artifacts is to relax the FoV requirement, making it a limiting factor for the field of view. twice ( Figure 10 (b) Right side). In this case, the principal point remains at the center of the boundary. And through regarding Solve Determine The disadvantage of setting B is that the FoV of the target image may be much smaller than specified, which often causes problems in applications. Figure 10 (b) Left side). Setting C: This disclosure discloses an alternative method for selecting the focal length and principal point of a target image, said method coordinating the change in DoV tilt angle γ with the desired reconstructed image. And images without empty areas ( Figure 10 (c) Right side). In Figure 11The scheme provides different steps including this novel method. The strategy is used to compensate for the lack of visual content at the top or bottom of the source image by using visual content available below or above (in the case of γ < 0 or γ > 0). For illustrative purposes, assume the offset in DoV is a positive angle offset γ>0 ( Figure 10 (c) Left side), which restricts the viewpoint Measure towards the notch (downward direction). The FoV of the target image must be reduced by a certain amount in the downward direction. To avoid creating empty zones, the idea is to increase the FoV by the same amount in the upward direction to achieve the same overall value as specified for the target camera. The upward offset of FoV causes the principal point to shift downward by an amount λ, making... In this case, the focal length can be determined by solving the following system of equations. and λ It should be noted that if γ < 0, the offset in FoV is from top to bottom, and the principal point is converted upwards, i.e. if Where Θ is the FoV of the source camera, then the disclosed value used for selection and The method consistently and successfully reproduces target images with the desired FoV and no empty zones. Harmonizing these two characteristics is important for providing a good user experience, but in some cases, this is achieved by significantly offsetting the principal point from the edge of the target image. This may be undesirable for some applications, particularly those where the target camera is designed to simulate a specific real-world endoscope camera, in which case the principal point is typically close to the image center. Applications and features The disclosed image processing method for modifying the DoV of an endoscope system with replaceable rotatable optics can generate multiple applications and be used in several different systems. This disclosure describes some embodiments of these systems and applications without affecting other possible applications. Electronic switching between rigid endoscopes with different lens aperture β Manufacturers of rigid endoscopes offer lenses with different angular offsets from the mechanical axis, with the most common lens incisions being β = 0°, 30°, 45°, and 70°. While surgical procedures generally favor endoscopes with specific incision angles, there are times or steps in surgery where using different lens incisions is more convenient. For example, in knee arthroscopy, an arthroscope with a 30° lens incision is preferred, but anterior or intercondylar meniscus examinations benefit from 0° or 70° incision angles. The problem is that because lateral changes in DoV require physically switching endoscopes, which can lead to interruptions and potentially pose risks to patients, surgeons rarely use the same endoscope for surgery in practice, even when visualization is not ideal. The disclosed method can be used in systems to process images and videos acquired by endoscopic cameras equipped with lenses having a wide FoV, enabling surgeons to electronically switch between two or more virtual endoscopes with different lens incision β. Such systems overcome the aforementioned difficulties that prevent surgeons from obtaining the best possible visual experience at every surgical moment or step. Suppose that the two desired virtual endoscopes each have a lens cutout. and as well as and Although the disclosed methods for selecting focal length and principal point ( Figure 10 (c) and Figure 11 While most source camera settings provide the required FoV, it's possible for the principal point to deviate too much from the center of the target image, which detracts from the user experience in the realistic virtualization of endoscopes with different lens apertures. A strategy to prevent this is to give the source camera a lens aperture β and FoV Θ that appropriately adapt to the lens aperture and FoV of the target camera. Figure 12 As shown, and taking into account the specifications of the two target cameras. and This is approximately in β and Θ. and This can be achieved under certain circumstances. One embodiment involves electronic switching between the two most common incision angles used in arthroscopy using the disclosed method within the system. This is because a standard 30° arthroscopy has... And 70° arthroscopy has Therefore, based on the above calculations, an ideal source camera should have a cut-out angle β of approximately 45° and a FoVΘ of approximately 140°. These values ​​are merely indicative and are used to ensure the reproduction of a realistic target image using the principal point in the central region. Let f, ξ, and O... i These are the precise calibration of the source camera, and the settings... and These are the image diameter and distortion of the two target cameras, respectively, which can be arbitrary or obtained from the calibration of the actual lens. Figure 8 The method is used to process images acquired by a real source camera, where the user commands to switch the target camera settings to alternate between a 30° lens (target 1) and a 70° lens (target 2). Figure 13 ). DoV's electronic alterations, distortion correction, and directional zoom To date, Figure 8 The method has been used to select a portion of the FoV of the source camera and to mimic the video output that will be acquired by cameras with smaller FoV and different lens cutouts placed in the same 3D position. Another possible embodiment or application is to use the disclosed method to offset the DoV of a particular endoscopic camera while maintaining the overall FoV, in which case the principal point in the reconstructed image moves toward the periphery as the offset in the DoV increases. Figure 14 ).in this case, Figure 8 and Figure 11 The disclosed method is executed in a system connected to a source camera, wherein the distortion of the target camera... Image diameter and It is set to the same value as the source camera, and the angular offset γ in the DoV is commanded by the user, who can freely move it up or down. Figure 14 ). The described system can be further enhanced by allowing users to control the possibility of distortion, in which case, settings Setting γ=0 will cause the system to correct radial distortion in the source video. The disclosed system can also be used to achieve so-called directional zoom, in which the angular offset γ is adjusted so that the principal point in the reconstructed image overlaps with the region of interest (ROI), and distortion is increased. To amplify the ROI while preserving the FoV and all visual content in the image ( Figure 15 As an alternative, ROI magnification can also be achieved by maintaining distortion and increasing the focal length. To achieve this, in this case, FoV is reduced and some image content will be lost. Other applications Figure 8 and Figure 11 The disclosed method of the scheme can be used to process a set of discrete images for reproducing images of virtual target cameras with different known calibrations and / or lens cutout angles, said images being used as input for training and / or testing machine learning or deep learning-based algorithms for automatically learning to identify and estimate different camera and / or lens characteristics, including but not limited to identifying lens cutouts, estimating calibration parameters, and learning to generate virtual images with different lens cutouts. The disclosed method can also be applied to other types of images, such as fundus images in ophthalmology. The disclosed method for selecting the focal length and position of the principal point in a target image ( Figure 11This method is not limited to endoscopic applications and can also be used to distort general images and videos. One possible embodiment is the use of the method in a visual surveillance system that implements electronic panning and tilting in images and videos acquired by a fisheye camera. Figure 16 This is an illustrative view of a computing system including a general-purpose computing system environment 1200, such as a desktop computer, laptop computer, smartphone, tablet computer, or any other such device capable of executing instructions (e.g., instructions stored in a non-transitory computer-readable medium). Furthermore, although described and shown in the context of a single computing system 1200, those skilled in the art will understand that the various tasks described below can be practiced in a distributed environment having multiple computing systems 1200 linked via a local or wide area network, wherein executable instructions can be associated with and / or executed by one or more of the multiple computing systems 1200. The computing system environment 1200 or portions thereof can be used in the processes, methods, and computational steps of this disclosure. In its most basic configuration, the computing system environment 1200 typically includes at least one processing unit 1202 and at least one memory 1204 that can be linked via a bus 1206. Depending on the exact configuration and type of the computing system environment, the memory 1204 may be volatile (e.g., RAM 1210), non-volatile (e.g., ROM 1208, flash memory, etc.), or some combination of both. The computing system environment 1200 may have additional features and / or functions. For example, the computing system environment 1200 may also include additional storage devices (removable and / or non-removable), including, but not limited to, hard disk or optical disk drives, tape drives, and / or flash memory drives. The computing system environment 1200 can access such additional storage devices through, for example, a hard disk drive interface 1212, a hard disk drive interface 1214, and / or an optical disk drive interface 1216. As will be understood, these devices, respectively linked to system bus 1206, allow reading from and writing to hard disk 1218, reading from or writing to removable disk 1220, and / or reading from or writing to removable optical disc 1222, such as CD / DVD ROM or other optical media. The drive interface and its associated computer-readable media allow for the non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computing system environment 1200. Those skilled in the art will further understand that other types of computer-readable media capable of storing data can be used for this same purpose. Examples of such media devices include, but are not limited to, magnetic cards, flash memory cards, digital video disks, Bernoulli cells, random access memory, nanodrives, memory sticks, other read / write and / or read-only memories, and / or any other methods or techniques for storing information such as computer-readable instructions, data structures, program modules, or other data. Any such computer storage medium may be part of computing system environment 1200. Multiple program modules may be stored in one or more memory / media devices. For example, a Basic Input / Output System (BIOS) 1224 containing basic routines that facilitate the transfer of information between components within the computing system environment 1200, for example, during startup, may be stored in ROM 1208. Similarly, RAM 1210, hard disk 1218, and / or peripheral device memory devices may be used to store computer-executable instructions including operating system 1226, one or more application programs 1228 (e.g., applications that execute the methods and processes of this disclosure), other program modules 1230, and / or program data 1232. Furthermore, the computer-executable instructions may be downloaded to computing environment 1200 as needed, for example, via a network connection. For example, end users such as customers and retail staff can input commands and information into the computing system environment 1200 via input devices such as keyboard 1234 and / or pointing device 1236. Although not shown, other input devices may include microphones, joysticks, gamepads, scanners, etc. These and other input devices are typically connected to the processing unit 1202 via peripheral interface 1238, which is coupled to bus 1206. Input devices may be connected directly or indirectly to the processing unit 1202 via interfaces such as parallel ports, game ports, FireWire, or Universal Serial Bus (USB). To view information from the computing system environment 1200, a display 1240 or other type of display device may also be connected to bus 1206 via an interface, such as via video adapter 1242. In addition to the display 1240, the computing system environment 1200 may also include other peripheral output devices, such as speakers and printers, not shown. The computing system environment 1200 may also utilize logical connections to one or more computing system environments. Communication between the computing system environment 1200 and remote computing system environments may be exchanged via another processing device, such as a network router 1252 responsible for network routing. Communication with the network router 1252 may be performed via a network interface component 1254. Therefore, in such networked environments, such as the Internet, the World Wide Web, a LAN, or other similar wired or wireless networks, it should be understood that program modules described relative to the computing system environment 1200 or a portion thereof may be stored in the memory storage device of the computing system environment 1200. The computing system environment 1200 may also include positioning hardware 1256 for determining the location of the computing system environment 1200. In embodiments, the positioning hardware 1256 may include, for example, a GPS antenna, an RFID chip or reader, a Wi-Fi antenna, or other computing hardware that can be used to capture or transmit signals that can be used to determine the location of the computing system environment 1200. In a first aspect of this disclosure, a method is disclosed that, given video acquired by a real source camera comprising a camera and a rigid endoscope having a lens notch β, the rigid endoscope is rotated about an azimuth angle about a mechanical axis intersecting the image at a point Q, for which the focal length f, radial distortion ξ, and principal point O at a certain azimuth angle α0 are known, the reconstruction will be performed by a camera positioned in the same 3D position as the source camera but with a lens notch β. Field of view is And distortion The video acquired by the virtual target camera, wherein for each corresponding source image I i Perform the following steps to reproduce a resolution of m x n with points Diameter of the center Each continuous target image of the circular boundary (i) The angular displacement δ of the azimuth angle is achieved by rotating O around Q. i =α i -α0, in source image I i Find the one with azimuth angle α i Main point O i (ii) Update the camera model of the source camera, which maps point x in the canonical image of the pinhole projection of point X in the 3D scene to point u in the pixel image, such that u = c s (x;f,O i (iii) By finding and back-projecting line n i Determine the vertical plane Π that contains or is close to the mechanical and optical axes of the source camera. i The 3D position of Π i With source image I i (iv) Define the 3D motion m between the target camera and the source camera as orbiting around a plane perpendicular to the vertical plane Π. i i direction Angle The rotation makes in (v) Calculate the principal points of the target camera in the canonical image; focal length and location, and according to Export the point Points mapped to pixel images The corresponding camera model; and (vi) generating the target image using image warping techniques. The mapping function w is used to... Each pixel in Mapped to source image I i For point u in the given information, the mapping function is function c. s m and c calculated in step (v) t The inverse combination Make interpolable color values In the first aspect of the embodiment, the source image I is processed by referencing the azimuth angle α0 from the specific notch position P. i To detect the central C i and notch position P i The boundary is used to estimate the angular displacement of the azimuth angle as... And from point O i and P i Limiting line n i . In a first aspect embodiment, the method further includes processing the reproduced target image. To generate a black frame so that meaningful image content is centered and diameter Within the circular region, and through the point on the circle's boundary. Visual markers are placed in the middle to create notches, where v i It is the image line n i 2D unit direction. In an embodiment of the first aspect, the area containing meaningful image content may take any desired geometric shape, such as, but not limited to, a conical shape, a rectangular shape, a hexagonal shape, or any other polygonal shape. In the first aspect of the embodiment, point P in consecutive frames is used. i and / or C i The detection dynamically determines the point Q, in which case it does not need to be known or determined a priori. In the embodiment of the first aspect, line n i Alternate location from point O i And Q or O i and C i limited. In the first aspect of the embodiment, the position of the principal point is given in a normalized reference frame attached to the circular boundary. In this case, it is not necessary to explicitly know the rotation center Q and the angular displacement δ of the azimuth angle. i In the case where, at each frame time i, its pixel position O is completed. i The calculation. In the first aspect of the embodiment, the source camera is equipped with an optical encoder or any other sensing device that measures the rotation of the observation instrument relative to the camera and estimates the angular displacement δ of the azimuth angle in step (i). i . In the embodiment of the first aspect, the principal point O coincides with the rotation center Q. In this case, it is not necessary to update the camera model of the source camera at every frame as in step (ii). In the first aspect of the embodiment, the distortion of the target camera The image is set to zero so that it can be reproduced. No distortion. In the first aspect of the embodiment, the reproduction of the target image in step (vi) is performed using any common method for image distortion or pixel value interpolation. The pixel value interpolation includes, but is not limited to, interpolation performed by the nearest neighbor, bilinear interpolation, or bicubic interpolation. In the embodiment of the first aspect, in step (v), the principal point is made to coincide with the center of the boundary. And select the focal length so that the FoV of the target camera is as desired. In this case, by solving get The value of Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray, all of which are dependent parameters. In the embodiment of the first aspect, in step (v), the principal point is made to coincide with the center of the boundary. Furthermore, the method involves selecting a focal length such that the reproduced image never contains areas without visual content (empty areas), and includes finding a distortion function in the source image that maps the principal point. Location The location can be determined by using camera model c. s and movement m The combined function g transitions from [0,0] to [0,0]. T Determine; determine limiting perspective It is The closest to the boundary of the circle The angle between the back projection rays of the point; if Then reduce the camera FoV to And by solving the equations get The value is determined by solving the equation; otherwise, the solution is obtained by... get The value of Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray, all of which are dependent parameters. In the embodiment of the first aspect, in step (vi), the focal length and the main point To ensure that the reconstructed image has no empty areas and the FoV of the target camera is a specified value. The method includes: finding a distortion function in the source image to map the principal point. Location The location can be determined by using camera model c. s and movement m The combined function g transitions from [0,0] to [0,0]. T Determine; determine limiting perspective It is The closest to the boundary of the circle The angle between the back projection rays of the point; if Then solve the following problem regarding focal length. The system of equations for λ: And obtain the main point, for Otherwise make And by solving turn up Where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray, all of which are dependent parameters. In the first aspect of the embodiment, the parameters of the target camera can be arbitrary, can be predefined to achieve a specific purpose, and can be equivalent to being equipped with a camera having a notch angle. The calibration of a specific real camera for a specific real endoscope can be selected by the user at the start of the procedure or changed during the operation based on specific events or user commands. In an embodiment of the first aspect, the method is used to connect to a system containing a source camera, so that the user can have and Electronic switching is performed between the lens cutout and the two virtual target cameras of FoV. In the first aspect of the embodiment, the source camera is selected such that the incision angle of the rigid endoscope is approximately And FoV is approximately This is to ensure that the principal point in the target camera is close to the center of the image. In the first aspect of the embodiment, the target camera's distortion Image resolution and diameter It is set to the same value as the source camera, and the cut angle The amount of angular offset γ added to the lens cutout β of the source camera, as set by the user, is such that... In the first aspect of the embodiment, distortion When both the angular offset γ and the angle offset γ are set to zero, the system will correct radial distortion in the source image. In the first aspect of the embodiment, the distortion Set to zero so that the target image can be reproduced without radial distortion, regardless of the selected angular offset γ. In the first aspect of the embodiment, an angular offset γ is selected such that the principal point is placed in the region of interest in the target image, and a radial distortion parameter is increased. This allows for magnification of the area of ​​interest while maintaining the FoV of the target camera and all visible content. In an embodiment of the first aspect, the source camera and / or the target camera have radial distortion described by any of the models known in the literature, including but not limited to learning-based models, Brownian polynomial models, rational number models, fisheye models, or segmentation models. In the embodiments of the first aspect, a general quadratic curve detection method is used, or machine learning or deep learning techniques or any other image processing techniques are used to detect curves with a center C. i and notch P i The boundary. In the first aspect embodiment, multiple notch locations are detected, and the angular displacement δ of the azimuth angle is transferred using specific notches of these multiple notches. i It is determined to be from point P i Angles limited by Q and P. In the first aspect of the embodiment, the field of view of the virtual target camera and distortion Take the same value as the source camera, but with a lens cutout. It is set by the user at each frame. In the first aspect of the embodiment, the field of view of the virtual target camera Take the same value as the source camera, but with a lens cutout. and distortion It is set by the user at each frame. In the first aspect of the embodiment, the field of view of the virtual target camera and lens cut Take the same value as the source camera, but with distortion. The user sets the scaling effect at each frame to produce a zoom-in or zoom-out effect without changing the FoV. In the first aspect of the embodiment, the field of view of the virtual target camera Take the same value as the source camera, but with a lens cutout. and distortion It is set by the user at each frame moment to adapt to DoV and produce a scaling up or down effect without changing FoV. In the first aspect of the embodiment, the field of view of the virtual target camera Take the same value as the source camera, but with a lens cutout. and focal length The focal length is set by the user at each frame time; in this case, the variable to be solved is not the focal length. Instead, it's a distortion. To produce a magnification or reduction effect without changing the FoV. While various embodiments have been described for the purposes of this disclosure, such embodiments should not be construed as limiting the teachings of this disclosure to those embodiments. Various changes and modifications can be made to the elements and operations described above to obtain results that remain within the scope of the systems and processes described in this disclosure. All patents, patent applications, and publications referenced herein are incorporated herein by reference in their entirety. It should be emphasized that the above embodiments of this disclosure are merely possible examples of implementations and are set forth solely for the purpose of clearly understanding the principles of this disclosure. Many variations and modifications can be made to the above embodiments without substantially departing from the spirit and principles of this disclosure. It should be understood that some and other features and functions disclosed above, or alternatives thereof, can be ideally combined into many other different systems or applications. All such modifications and variations are intended to be included within the scope of this disclosure, if they fall within the scope of the appended claims. The described embodiments are to be considered illustrative rather than restrictive in all respects, and therefore the scope of the embodiments disclosed herein is indicated by the appended claims rather than the foregoing description. All variations falling within the equivalent meaning and scope of the claims should be considered within their scope. Those skilled in the art may implement the described functionality in different ways for each particular application, but such implementation decisions should not be construed as resulting in a departure from the scope of the disclosed systems and / or methods.

Claims

1. A method for using source images I captured by a real source camera i Reproduce the target image from the virtual target camera The method, wherein the source camera includes a camera and a rigid endoscope having a lens notch β, the rigid endoscope rotating about an azimuth angle about a mechanical axis intersecting the image at a point Q, for which the focal length f, radial distortion ξ, and principal point O at a certain azimuth angle α0 are known, the method comprising: by rotating Q around O by an angular displacement δ of the azimuthal angle i = a i - a0, in the source image I i finding the position of the principal point O i with azimuthal angle a i . According to the focal length f, the radial distortion ξ and the position of the principal point O i , the camera model c of the source camera mapping a point x in the canonical image to a point u in the pixel image is updated s ; by finding and back-projecting the line n i determining the 3D position of a vertical plane Π i containing or close to the mechanical and optical axis of the source camera, where Π i intersects the source image I i ; defining a 3D motion m between the target camera and the source camera as a rotation around an axis ii oriented angularly to a direction wherein is a point in the canonical image of the target camera; calculating a principal point of the target camera a focal length of and a position of the principal point and deriving a camera model c mapping a point in a normalized image of the target image to a point in a pixel image of the target image t ; and by using a mapping function w to map a plurality of pixels in the source image I to points u in the target image by using a mapping function w to map a plurality of pixels in the source image I i to points u in the target image The mapping function is a combination of the camera model c s of the source camera, the 3D motion m and the inverse model of the camera model c t of the target camera.

2. The method of claim 1, wherein the azimuth angle a0 corresponds to a first notch position P, the method further comprising: processing said source image I i to detect a center C i and a second notch position P i of the boundary; wherein the angular displacement of the orientation angle is estimated from the first notch position P, the second notch position P i and the point Q. wherein the line n i is defined by the points O i and P i .

3. The method of claim 2, further comprising processing the reproduced target image to produce a black frame defining an image region having a center and a diameter and producing a notch by placing a visual marker at a point in the circular boundary, where v i is a 2D unit direction of the image line n i .

4. The method of claim 3, wherein the image region comprises one or more of a circular shape, a conical shape, a rectangular shape, a hexagonal shape, or another polygonal shape.

5. The method of claim 2, wherein the source image I i comprises two or more source image frames, wherein the point Q is determined by detecting one or more of the points P i or C i in consecutive ones of the two or more frames.

6. The method of claim 2, wherein processing the source image I i to detect a center C i and a second notch position P i includes detecting a plurality of second notch positions P i wherein one of the plurality of second notch positions is used to determine an angular displacement δ i of the azimuth angle.

7. The method of claim 1, wherein the azimuth angle a0 corresponds to a first notch position P, the method further comprising: processing said source image I i to detect a center C i and a second notch position P i of the boundary; wherein the angular displacement of the orientation angle is estimated from the first notch position P, the second notch position P i and the point Q. wherein the line n i is defined by: Point O i and Q; or Point O i and C i .

8. The method of claim 1, wherein the position of the principal point is given in a normalized reference frame disposed in or attached to the endoscope.

9. The method of claim 1, wherein the source camera is equipped with a sensor that measures the rotation of the endoscope relative to the camera and estimates the angular displacement δ of the azimuth angle i .

10. The method of claim 1, wherein the principal point O coincides with a center of rotation Q.

11. The method of claim 1, wherein the distortion of the target camera is set to zero.

11. The method of claim 1, wherein the distortion of the target camera is set to zero.​ 12. The method of claim 1, wherein generating the target image is performed using image warping or pixel value interpolation. The pixel value interpolation includes one or more of nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

13. The method of claim 1, wherein the principal point coincides with the center of the border of the target image and the focal length is calculated including solving where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray.

14. The method of claim 1, wherein the principal point coincides with a center of a boundary of the target image the method comprising: by a function g of the combination of the camera model c s and the motion m transforms the origin [0,0] of the source image T to find the position in the source image that maps to the principal point of the destination image determining and the limiting viewing angle between the back-projection ray of the point in the circular boundary that is closest to ​ When the camera field of view is reduced to and the value of is obtained by solving the equation where Φ is a mathematical expression relating the focal length, radial distortion, image distance and the angle between the back-projected ray. When the value of Φ is obtained by solving the equation where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray, all of which are dependent parameters.​ 15. The method of claim 1, further comprising: by a function g of the combination of the camera model c s and the motion m transforms the origin [0,0] of the source image T to find the position in the source image that maps to the principal point of the destination image determining and the limiting viewing angle between the back-projection ray of the point in the circular boundary that is closest to the point in the circular boundary that is closest to When the focal length and λ, the following system of equations is solved: and the principal point is obtained as When is set and by solving is determined where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray, all of which are dependent parameters.

16. The method of claim 15, wherein: The source camera is selected such that the cut angle of the rigid endoscope is and the field of view is where and are the lens cut and field of view of the first virtual target camera, respectively, and and are the lens cut and field of view of the second virtual target camera, respectively, wherein the virtual target camera is the first virtual target camera or the second virtual target camera.

17. The method of claim 15, wherein the target camera's distortion image resolution and diameter are set to the same respective values as the source camera's, and the target camera's cut angle is set by the user through a user selection of an amount of angular offset γ added to the source camera's lens cut β.

18. The method of claim 17, wherein the distortion is zero.

19. The method of claim 18, wherein the angular offset g is zero.

20. The method of claim 15, wherein the cut angle of the target camera is set by a user through a user selection of an amount of angular offset γ to the lens cut β added to the source camera, wherein the angular offset γ is selected by the user such that the principal point is placed in a region of interest in the target image, and the radial distortion parameter is increased relative to the radial distortion ξ of the source camera to magnify the region of interest.

21. The method of claim 1, wherein a field of view of the virtual target camera and the distortion of the virtual target camera take the same values as the source camera, and a lens cut of the virtual target camera is set by a user.

22. The method of claim 1, wherein a field of view of the virtual target camera takes the same value as that of the source camera, and the lens cut and the distortion of the virtual target camera are set by a user.

23. The method of claim 22, wherein the lens cut and the distortion are set by the user to adapt the depth of field and produce a zoom effect that does not change the field of view.

24. The method of claim 1, wherein a field of view of the virtual target camera and the lens cut take the same values as the source camera, and the distortion of the virtual target camera is set by a user.

25. The method of claim 1, wherein the field of view of the virtual target camera takes the same value as the source camera, and the lens cut and focal length of the virtual target camera are set by the user.

26. The method of claim 25, wherein the principal point coincides with the center of the border of the target image and the distortion is calculated comprises solving where Φ is a mathematical expression relating focal length, radial distortion, image distance and angle between back-projected rays.

27. The method of claim 25, wherein the principal point coincides with a center of a border of the target image the method comprising: by a function g of the combination of the camera model c s and the motion m transforms the origin [0,0] of the source image T to find the position in the source image that maps to the principal point of the destination image determining and the limiting view angle between the back-projection ray of the point in the circle boundary closest to the point in the circle boundary closest to When the camera field of view is reduced to and the equation is solved for the value of distortion where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray. When the value of the distortion is obtained by solving the equation where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray, all of which are dependent parameters.

28. The method of claim 25, further comprising: by a function g of the combination of the camera model c s and the motion m transforms the origin [0,0] of the source image T to find the position in the source image that maps to the principal point of the destination image determining and the limiting viewing angle between the back-projection ray of the point in the circle boundary closest to the point in the circle boundary closest to When the distortion and λ solve the following system of equations: and obtain the principal point, as When is set and the distortion is determined by solving where Φ is a mathematical expression relating focal length, radial distortion, image distance, and the angle between the back-projected ray, all of which are dependent parameters.

29. A method for using source images I captured by a real source camera i Reproduce the target image from the virtual target camera The system, wherein the source camera includes a camera and a rigid endoscope having a lens notch β, the rigid endoscope rotating about an azimuth angle about a mechanical axis intersecting the image at point Q, for which the focal length f, radial distortion ξ, and principal point O at a certain azimuth angle α0 are known, the system comprising: a non-transitory computer readable medium storing instructions; and a processor configured to execute the instructions to perform a method, the method comprising: by rotating Q about the origin by an angular displacement δ of the azimuthal angle of O i = a i - a0, in the source image I i find the position of the origin O i with the azimuthal angle a i . According to the focal length f, the radial distortion ξ and the position of the principal point O i , the camera model c of the source camera mapping a point x in the canonical image to a point u in the pixel image is updated s ; by finding and back-projecting the line n i , determining the 3D position of a vertical plane Π i containing or close to the mechanical and optical axis of the source camera, where Π i intersects the source image I i ; defining a 3D motion m between the target camera and the source camera as a rotation around a direction ii angularly angled to the vertical plane Π wherein is a point in the canonical image of the target camera; calculating a principal point of the target camera a focal length of and a position of the principal point and deriving a camera model mapping points in the pixel image to points in the scene according to the focal length and the position of the principal point. by using a mapping function w to map a plurality of pixels in the source image I to points u in the target image by using a mapping function w to map a plurality of pixels in the source image I i to points u in the target image The mapping function is a combination of the camera model c s of the source camera, the 3D motion m and an inverse model of the camera model c t of the target camera, wherein the camera model c t of the target camera maps points in a normalized image of the target camera to points in a pixel image of the target camera.

30. A method for reproducing target images from a virtual target camera. The method, the method comprising: obtaining a source image I captured by a real source camera, the source camera comprising a medical observation instrument i , the source camera comprising a medical observation instrument; The position of a main point O i with an azimuth angle a i is found in the source image I i , wherein the real source camera is associated with a camera model c i that maps a point x in a canonical image to a point u in a pixel image depending on the position of the main point O s . by finding and back-projecting the line n i , determining the 3D position of the vertical plane Π i close to the mechanical and optical axis of the source camera, where Π i intersects the source image I i ; defining as 3D motion m between the target camera and the source camera a rotation around a direction i normal to the vertical plane Π ; deriving a camera model c that maps points in a canonical image of the target image to points in a pixel image of the target image t ;​​​ and a non-transitory computer readable medium storing instructions; and a processor configured to execute the instructions to perform a method, the method comprising: by using a mapping function w to map a plurality of pixels in the source image I to points u in the target image by using a mapping function w to map a plurality of pixels in the source image I i to points u in the target image The mapping function is a combination of the camera model c s of the source camera, the 3D motion m and the inverse model of the camera model c t of the target camera.

Images