System and method for correcting distortion caused by optical element of image viewer of computer-assisted surgical system

Abstract

Systems and methods for correcting distortion caused by an optical element of an image viewer of a computer-assisted surgical system are disclosed. An illustrative system includes one or more processors configured to execute a process. The process may comprise: obtaining a distortion factor representative of an amount of distortion imparted by one or more optical elements of an image viewer of the computer-assisted surgical system to an image displayed by means of the image viewer; obtaining image data representing an image displayable by means of an image viewer; modifying the image data based on the distortion factor to correct for distortion imparted by one or more optical elements of the image viewer, the modification producing modified image data representative of the image; and instructing the computer-assisted surgical system to display an image by means of an image viewer based on the modified image data.

Description

Background Art

[0001] Computer-assisted surgical systems employing robotic and / or teleoperation technologies typically include a stereoscopic image viewer configured to provide images of an imaging space (e.g., a surgical space) captured by an imaging device (e.g., an endoscope) for display to a surgeon. When the surgeon's eyes are in front of the viewing lenses of the stereoscopic image viewer, the surgeon can view the images of the surgical space while remotely manipulating one or more surgical instruments located within the surgical space. The surgical instruments are attached to one or more manipulator arms of a surgical instrument manipulation system included as part of the computer-assisted surgical system.

[0002] The stereoscopic image viewer of a computer-assisted surgical system must display a precisely overlapping pair of images (one for each eye). Additionally, compact stereoscopic image viewers typically involve the use of one or more optical elements that apply optical power to position the image rendering plane at a specific distance from the user. However, such optical elements may create distortions (e.g., barrel distortion, pincushion distortion, etc.) in the displayed images for each eye. This distortion inhibits the overlap of the pair of images. If uncorrected, such distortion may degrade the user's experience when operating the computer-assisted surgical system (e.g., by causing nausea, headache, etc.). To correct such distortion, some computer-assisted surgical systems include a distortion correction lens element that is placed near the flat panel display of the stereoscopic image viewer. However, such distortion correction lens elements are complex, bulky, expensive, and difficult to fabricate / purchase. Thus, there is still room for improvement in distortion correction in the image viewers of computer-assisted surgical systems. Summary of the Invention

[0003] An example system includes one or more processors configured to execute a process that includes: obtaining a distortion factor representative of the amount of distortion imparted to an image displayed by an image viewer of a computer-assisted surgical system by one or more optical elements of the image viewer; obtaining image data representative of an image displayable by the image viewer; modifying the image data based on the distortion factor to correct for the distortion imparted by the one or more optical elements of the image viewer, the modification resulting in modified image data representative of the image; and instructing the computer-assisted surgical system to display the image based on the modified image data by means of the image viewer.

[0004] An example computer-assisted surgical system includes an image viewer that includes: a display device; and an optical assembly through which a user views an image displayed by the display device, the optical assembly being disposed along an optical path between the display device and the user's eyes, the optical assembly imposing distortion on the image displayed by the display device when viewed through the optical assembly; wherein the display device is configured to: receive an image to be displayed by means of the image viewer; and display the image based on a distortion factor that corrects the distortion imposed by the optical assembly.

[0005] An additional example computer-assisted surgical system includes one or more manipulator arms; and one or more processors configured to execute a process that includes: obtaining a distortion factor that represents an amount of distortion imposed by one or more optical elements of an image viewer; obtaining image data that represents an image displayable by means of the image viewer; modifying the image data based on the distortion factor to correct the distortion imposed by one or more optical elements of the image viewer, the modification resulting in modified image data representative of the image; and presenting the image by means of the image viewer based on the modified image data.

[0006] An example method includes: obtaining, by an image processing system, a distortion factor that represents an amount of distortion imposed by one or more optical elements of an image viewer of a computer-assisted surgical system on an image displayed by means of the image viewer; obtaining, by the image processing system, image data that represents an image displayable by means of the image viewer; modifying, by the image processing system and based on the distortion factor, the image data to correct the distortion imposed by one or more optical elements of the image viewer, the modification resulting in modified image data representative of the image; and directing, by the image processing system, the computer-assisted surgical system to display the image by means of the image viewer based on the modified image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The drawings illustrate various embodiments and are a part of the specification. The illustrated embodiments are merely examples and do not limit the scope of the disclosure. In all the drawings, the same or similar reference numerals denote the same or similar elements.

[0008] Figure 1 An example computer-assisted surgical system in accordance with the principles described herein is illustrated.

[0009] Figure 2 An example implementation of an image viewer configured in accordance with the principles described herein is illustrated.

[0010] Figure 3 An example image processing system in accordance with the principles described herein is illustrated.

[0011] Figure 4 Illustrates an example flowchart depicting various operations that can be performed by an Figure 3 image processing system as shown in accordance with the principles described herein.

[0012] Figure 5 Illustrates an example view of a modified image that has been modified based on a distortion factor in accordance with the principles described herein.

[0013] Figure 6 Illustrates a simplified version of a display panel configured to correct distortion in accordance with the principles described herein.

[0014] Figure 7 Illustrates an example method for correcting distortion caused by an optical element of an image viewer of a computer-assisted surgical system in accordance with the principles described herein.

[0015] Figure 8 Illustrates an example computing device in accordance with the principles described herein. DETAILED DESCRIPTION

[0016] Systems and methods for correcting distortion imparted by an optical element of an image viewer of a computer-assisted surgical system are described herein. As will be described in more detail below, an illustrative system includes one or more processors configured to perform a process. The process can include: obtaining a distortion factor that represents an amount of distortion imparted by one or more optical elements of an image viewer of a computer-assisted surgical system to an image displayed by means of the image viewer; obtaining image data that represents an image displayable by means of the image viewer; modifying the image data based on the distortion factor to correct for the distortion imparted by the one or more optical elements of the image viewer, the modification resulting in modified image data that represents the image; and instructing the computer-assisted surgical system to display the image by means of the image viewer based on the modified image data.

[0017] Numerous advantages and benefits are associated with the systems and methods described herein. For example, systems and methods such as those described herein can help correct for distortion caused by one or more optical elements of an image viewer in a manner that improves the user experience, is more cost-effective and / or more efficient than conventional image viewers. For example, systems and methods such as those described herein do not require the use of expensive and difficult-to-fabricate distortion correction lenses. Additionally, by eliminating the need for such distortion correction lenses, it is possible to reduce or eliminate unwanted stray light paths in the image viewer. These and other benefits that can be achieved by the systems and methods described herein will be apparent from the following disclosure.

[0018] The example systems described herein can be configured to operate as part of or in conjunction with a variety of different types of computer-assisted surgical systems. Different types of computer-assisted surgical systems can include any type of computer-assisted surgical system, which can serve a particular implementation, such as, for example, a computer-assisted surgical system designed for minimally invasive medical procedures. In some examples, one type of computer-assisted surgical system can include a system in which one or more surgical devices (e.g., surgical instruments) are manually controlled by a user (e.g., in a laparoscopic manner). In some examples, a computer-assisted surgical system can include a robotic surgical system configured to facilitate the operation of one or more intelligent instruments (e.g., an intelligent sub-surface imaging device), which can be manually and / or robotically controlled by a user. In some implementations, multiple different types of computer-assisted surgical systems can be different types at least in that they include different types of surgical instrument manipulation systems. For example, a first computer-assisted surgical system can include a first type of surgical instrument manipulation system, a second computer-assisted surgical system can include a second type of surgical instrument manipulation system, and a third computer-assisted surgical system can include a third type of surgical instrument manipulation system.

[0019] Each type of surgical instrument manipulation system can have a different architecture (e.g., manipulator arm architecture), have different kinematic profiles, and / or operate according to different configuration parameters. Now, reference will be made to Figure 1 describe an illustrative computer-assisted surgical system having a first type of surgical instrument manipulation system. The computer-assisted surgical system described is illustrative and not restrictive. Systems such as those described herein can operate as part of or in conjunction with the described computer-assisted surgical system and / or any other suitable computer-assisted surgical system.

[0020] Figure 1 An example computer-assisted surgical system 100 (“surgical system 100”) is illustrated. As shown, surgical system 100 includes a surgical instrument manipulation system 102 (“manipulation system 102”), a user control system 104, and an auxiliary system 106 communicatively coupled to each other. Additional or alternative components can be included in surgical system 100, which can serve a particular implementation.

[0021] A surgical team may utilize the surgical system 100 to perform a computer-assisted surgical procedure on a patient 108. As shown, the surgical team may include a surgeon 110-1, an assistant 110-2, a nurse 110-3, and an anesthesiologist 110-4, all of whom may be collectively referred to as "surgical team members 110". During a surgical period, there may be additional or alternative surgical team members, which may serve specific embodiments.

[0022] Although Figure 1 an illustration of a minimally invasive surgical procedure in progress is shown, the surgical system 100 may similarly be used to perform an open surgical procedure or other types of surgical procedures, which may similarly benefit from the accuracy and convenience of the surgical system 100. In addition, it will be understood that the surgical period during which the surgical system 100 may be used may include not only the surgical phase of the surgical procedure as Figure 1 shown, but may also include pre-operative, post-operative, and / or other suitable phases of the surgical procedure. A surgical procedure may include any procedure that uses manual and / or instrumental techniques (e.g., teleoperated instrumental techniques) on a patient to investigate, diagnose, or treat the patient's physical condition. In addition, a surgical procedure may include any procedure that is not performed on a living patient, such as a calibration procedure, a simulation training procedure, and an experimental or research procedure.

[0023] As Figure 1 shown, the surgical instrument manipulation system 102 includes a plurality of manipulator arms 112 (e.g., manipulator arms 112-1 to 112-4), to which a plurality of robotic surgical instruments ("robotic instruments") (not shown) may be coupled. As used herein, a "robotic instrument" refers to any instrument that may be directly attached to (e.g., inserted into, fixedly coupled to, mated to, etc.) a manipulator arm (e.g., manipulator arm 112-1) such that movement of the manipulator arm directly causes movement of the instrument. Each robotic instrument may be implemented by any suitable therapeutic instrument (e.g., a tool having tissue interaction capabilities), an imaging device (e.g., an endoscope), a diagnostic instrument, etc., which may be used in a computer-assisted surgical procedure (e.g., by being at least partially inserted into the patient 108's body and being manipulated to perform a computer-assisted surgical procedure on the patient 108). In some examples, one or more of the robotic instruments include force sensing and / or other sensing capabilities.

[0024] In Figure 1In the example shown, the manipulator arm 112 of the manipulation system 102 is attached to the distal end of a horizontally extending overhead boom. However, in some embodiments, the manipulator arm 112 may have other configurations. Additionally, although the manipulation system 102 is depicted and described herein as including four manipulator arms 112, it will be appreciated that the manipulation system 102 may include only a single manipulator arm 112 or any other number of manipulator arms, which may serve a particular embodiment.

[0025] The manipulator arm 112 and / or the robotic instrument attached to the manipulator arm 112 may include one or more displacement transducers, orientation sensors, and / or position sensors (hereinafter referred to as "surgical system sensors") for generating raw (e.g., uncorrected) kinematic information. One or more components of the surgical system 100 may be configured to use the kinematic information to track the robotic instrument (e.g., determine the position of the robotic instrument) and / or control the robotic instrument.

[0026] In addition, each manipulator arm 112 may include or otherwise be associated with a plurality of motors or actuators that control the movement of the manipulator arm 112 and / or the surgical instrument attached to the manipulator arm 112. For example, the manipulator arm 112-1 may include or otherwise be associated with a first internal motor (not explicitly shown) that is configured to yaw the manipulator arm 112-1 about a yaw axis. In a similar manner, the manipulator arm 112-1 may be associated with a second internal motor (not explicitly shown) that is configured to drive and pitch the manipulator arm 112-1 about a pitch axis. Similarly, the manipulator arm 112-1 may be associated with a third internal motor (not explicitly shown) that is configured to slide the manipulator arm 112-1 along an insertion axis. Each manipulator arm 112 may include a transmission system driven by one or more of these motors to control the pivoting of the manipulator arm 122 in any way that may serve a particular embodiment. Thus, if it is desired to mechanically move, for example, a robotic instrument attached to the manipulator arm 112-1, one or more of the motors coupled to the transmission system may be energized to move the manipulator arm 112-1.

[0027] The robotic instruments attached to the manipulator arm 112 can each be located within the imaging space. As used herein, "imaging space" can refer to any space or orientation in which imaging operations can be performed by an imaging device as described herein. In some examples, the imaging space can correspond to the surgical space. In some embodiments, "surgical space" can be entirely disposed within a patient and can include the region at or near the location within the patient where a surgical procedure is planned, being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissue within a patient, the surgical space can include the tissue, the anatomical structures beneath the tissue, and the space surrounding the tissue, such as the space in which robotic instruments and / or other instruments for performing the surgical procedure are located. In other examples, the surgical space can be at least partially disposed outside the patient, located at or near the location where a surgical procedure is planned, being performed, or has been performed on the patient. For example, the surgical system 100 can be used to perform an open surgical procedure such that a portion of the surgical space (e.g., the tissue being manipulated) is within the patient while another portion of the surgical space (e.g., the space around the tissue where one or more instruments can be disposed) is outside the patient. When at least a portion of a robotic instrument (e.g., the distal portion of the robotic instrument) is located within the surgical space, the robotic instrument can be referred to as being positioned at or within the surgical space. Example imaging spaces and / or images of the imaging space will be described herein.

[0028] The user control system 104 is configured to facilitate control of the manipulator arm 112 and the robotic instruments attached to the manipulator arm 112 by the surgeon 110-1. For example, the surgeon 110-1 can interact with the user control system 104 to remotely move, manipulate, or otherwise remotely operate the manipulator arm 112 and the robotic instruments. To this end, the user control system 104 can provide the surgeon 110-1 with one or more images of the surgical space associated with the patient 108 as captured by the imaging device. In some examples, the user control system 104 can include a stereoscopic image viewer having two displays, where the surgeon 110-1 can view a pair of stereoscopic images of the surgical space associated with the patient 108 generated by the stereoscopic imaging system. The surgeon 110-1 can utilize one or more images to perform one or more procedures with one or more robotic instruments attached to the manipulator arm 112.

[0029] To facilitate control of the robotic instrument, the user control system 104 may include a set of master controls (not shown). These master controls may be manipulated by the surgeon 110-1 to control the movement of the robotic instrument (e.g., by utilizing robotics and / or teleoperation techniques). The master controls may be configured to detect a variety of hand, wrist, and finger movements of the surgeon 110-1. In this way, the surgeon 110-1 may use one or more robotic instruments to intuitively perform a surgical procedure.

[0030] In some examples, the user control system 104 may also be configured to facilitate the surgeon 110-1 in controlling other components of the surgical system 100. For example, the surgeon 110-1 may interact with the user control system 104 to change the configuration or operating mode of the surgical system 100, change the display mode of the surgical system 100, generate additional control signals for controlling the surgical instruments attached to the manipulator arm 112, facilitate switching control from one robotic instrument to another, facilitate interaction with other instruments and / or objects within the surgical space, or perform any other suitable operation. To this end, the user control system 104 may also include one or more input devices (e.g., foot pedals, buttons, switches, etc.) that are configured to receive input from the surgeon 110-1.

[0031] In some examples, the auxiliary system 106 includes one or more computing devices that are configured to perform the main processing operations of the surgical system 100. The one or more computing devices included in the auxiliary system 106 may control and / or coordinate the operations performed by the various other components of the surgical system 100 (e.g., the manipulation system 102 and / or the user control system 104). For example, the computing devices included in the user control system 104 may transmit instructions to the manipulation system 102 by means of the one or more computing devices included in the auxiliary system 106. As another example, the auxiliary system 106 may receive and process image data that represents images captured by an imaging device attached to one of the manipulator arms 112.

[0032] In some examples, the assistance system 106 is configured to present visual content to a surgical team member 110 who may not have access to the images provided to the surgeon 110-1 at the user control system 104. To this end, as shown, the assistance system 106 includes a display monitor 114 that is configured to display one or more user interfaces, such as an image of the surgical space, information associated with the patient 108 and / or the surgical procedure, and / or any other visual content, which may serve a particular implementation. For example, the display monitor 114 may display an image of the surgical space and additional content (e.g., a representation of a target object, graphical content, context information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 114 is implemented by a touchscreen display with which the surgical team member 110 may interact (e.g., by means of touch gestures) to provide user input to the surgical system 100.

[0033] The manipulation system 102, the user control system 104, and the assistance system 106 may be communicatively coupled to each other in any suitable manner. For example, as Figure 1 shown, the manipulation system 102, the user control system 104, and the assistance system 106 are communicatively coupled by means of a control line 116, which may represent any wired or wireless communication link, which may serve a particular implementation. To this end, the manipulation system 102, the user control system 104, and the assistance system 106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0034] Figure 2 An exemplary embodiment 200 of an image viewer 202 is illustrated, which may be configured, in accordance with the principles described herein, as part of a computer-assisted surgical system, such as the surgical system 100. As Figure 2 shown, the image viewer 202 includes an optical assembly 204 and a display device 206 that are arranged relative to each other along an optical path. The image viewer 202 may include additional or alternative components, which may serve a particular implementation.

[0035] As Figure 2 shown, the optical assembly 204 is arranged in front of the user's eye 208 such that the image displayed by the display device 206 is viewable by means of the optical assembly 204. Although Figure 2 only one optical assembly 204 and display device 206 are shown, it should be understood that additional optical assemblies and display devices may be provided for the user's other eye.

[0036] Figure 2The depiction of the image viewer 202 therein is schematic to illustrate the basic components of the image viewer 202. Thus, the relative sizes, orientations, and arrangements of the optical assembly 204 and the display device 206 are provided for illustrative purposes only. It should be understood that Figure 2 the components shown may be configured differently in other embodiments and / or may include any suitable additional elements. For example, in some embodiments, the image viewer 202 may also include one or more reflectors disposed along the optical path between the display device 206 and the optical assembly 204. In such examples, the display device 206 may be disposed in a side-mounted position, and the reflector may redirect the image displayed by the display device 206 toward the eye 208.

[0037] The image viewer 202 may be implemented in any suitable manner as part of a computer-assisted surgical system. For example, the image viewer 202 may be part of a surgeon console (e.g., the user control system 104) that is communicatively coupled to one or more manipulator arms (e.g., the manipulator arm 112) and an imaging device 210, which are communicatively coupled to the computer-assisted surgical system.

[0038] As Figure 2 shown, the optical assembly 204 is disposed along the optical path between the display device 206 and the user's eye 208. The optical assembly 204 may include one or more optical elements that impart distortion to the image displayed by the display device 206 when viewed through the optical assembly 204. The optical assembly 204 may include a single optical element or lens as Figure 2 shown, or may include a lens group composed of multiple lenses or optical elements. For example, the optical assembly 204 may include an eyepiece and one or more optical elements. The optical assembly 204 may include any suitable type of optical element, which may serve a particular embodiment. For example, the optical assembly 204 may include convex lenses, concave lenses, Fresnel lenses, and / or any other suitable type of optical element or combination thereof. The optical assembly 204 may also include any suitable additional optical imaging elements, such as, for example, filters, gratings, etc.

[0039] The display device 206 may include any suitable display device, which may serve a particular implementation. For example, in some implementations, the display device 206 may correspond to a flat panel light emitting diode (LED) display device, a liquid crystal display (LCD) device, a liquid crystal on silicon (LCOS) display device, a microelectromechanical systems (MEMS) display device, a digital light processing (DLP) display device, an organic light emitting diode (OLED) display device, and / or any other suitable type of display device. In some examples, the display device 206 may correspond to a rectilinear display device. In some alternative implementations, the display device 206 may correspond to a non-rectilinear display device (e.g., a display device having a curved display architecture with a non-rectilinear pixel grid). Implementations using non-rectilinear display devices are also described herein.

[0040] The display device 206 is configured to receive the images captured by the imaging device 210 and display the images such that they are viewable along the optical path through the optical assembly 204. The imaging device 210 may include any suitable type of imaging device, which may serve a particular implementation. For example, the imaging device 210 may be implemented by an endoscope engaged with a manipulator arm of a computer-assisted surgical system (e.g., manipulator arm 112-2).

[0041] As Figure 2 shown, the user's eyes 208 may view the images displayed by the display device 206 by means of the optical assembly 204. The optical assembly 204 is configured to position the image rendering plane 212 at a particular distance from the user, which is suitable for viewing. However, due to the optical power and / or other parameters associated with the optical assembly 204 and / or the image viewer 202, the images displayed by the display device 206 may be distorted. For example, such distortion may include barrel distortion, pincushion distortion, and / or any other type of distortion or combination of distortions. Other parameters that may affect the amount and / or type of distortion may include, for example, the focal length of the optical assembly 204, the distance between the display device 206 and the optical assembly 204, the distance between the user's eyes 208 and the optical assembly 204, and / or any other suitable parameters.

[0042] As Figure 2As shown, the image viewer 202 does not include a distortion correction lens specifically configured to correct the distortion caused by the optical assembly 204. Additionally, the image viewer 202 does not include a distortion correction lens included as part of the optical assembly 204 that is configured to correct the distortion imparted by one or more other optical elements of the optical assembly 204. Instead, the display device 206 may be further configured to display an image captured by the imaging device 210 based on a distortion factor that corrects the distortion imparted by the optical assembly 204. As used herein, a "distortion factor" may indicate the amount of distortion imparted by one or more optical elements of an image viewer. The distortion factor may be due to the inherent characteristics of the optical assembly 204 and / or other parameters (such as those described herein) that may affect the amount of distortion.

[0043] The image viewer 202 may be configured in any suitable manner to correct the distortion imparted by the optical assembly 204. In some examples, the image viewer 202 may also include or otherwise be communicatively coupled to one or more processors configured to modify an image captured by the imaging device 210 based on the distortion factor. In such examples, a distortion correction process may be performed on the image to be displayed before the image is displayed by the display device 206. Such a distortion correction process is configured to counteract the distortion caused by the optical assembly 204 such that when the image is viewed through the optical assembly 204, it appears undistorted to the user.

[0044] Figure 3 An example image processing system 300 is shown that may be implemented in accordance with the principles described herein to correct distortion caused by one or more optical elements of an image viewer. As Figure 3 shown, the image processing system 300 ("system 300") includes a processor 302. The processor 302 may be implemented as one or more processors of any suitable type, which may serve a particular implementation. For example, in some implementations, all or part of the functions or processes described herein may be implemented as dedicated logic circuitry (e.g., a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC)).

[0045] System 300 may include additional or alternative elements, which may serve a particular implementation. For example, system 300 may also include a memory selectively and communicatively coupled to processor 302. Such memory and processor 302 may each include hardware and / or software components (e.g., processors, memories, communication interfaces, instructions stored in the memory for execution by the processor, etc.) or be implemented by hardware and / or software components. In some examples, such memory and processor 302 may be implemented by a single device (e.g., a single computing device). In certain alternative examples, the memory and processor 302 may be distributed among multiple devices and / or multiple locations, which may serve a particular implementation.

[0046] In certain examples, the memory may maintain (e.g., store) executable data used by processor 302 to perform any of the operations described herein. For example, the memory may store instructions that may be executed by processor 302 to perform any of the operations described herein. These instructions may be implemented by any suitable application, software, code, computer program, and / or other executable data instance.

[0047] The computer program may be written in any form of programming language (including compiled and / or interpreted languages) and may be deployed in any form (including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment). The computing program may be deployed to be executed by processor 302 at one location or distributed across multiple locations and interconnected by a network.

[0048] In certain examples, the memory may also maintain any data received, generated, managed, used, and / or transmitted by processor 302. For example, the memory may maintain any suitable data associated with correcting distortion caused by one or more optical elements. Such data may include, but is not limited to, data associated with distortion factors, factors associated with the use of an image viewer, image data of an imaging space (e.g., an endoscopic image), user interface content (e.g., graphical objects, notifications, etc.), and / or any other suitable data.

[0049] Processor 302 may be configured to perform (e.g., execute instructions stored in the memory) various processing operations associated with correcting distortion imparted by one or more optical elements. For example, processor 302 may modify the image data based on the distortion factor to correct the distortion imparted by one or more optical elements of an image viewer, the modification resulting in modified image data representative of the image. Processor 302 may also direct a computer-assisted surgical system to display the image via the image viewer based on the modified image data. These and other operations that may be performed by processor 302 are described herein.

[0050] Figure 4 FIG. 400 is illustrated, which depicts various operations that can be performed by system 300 (e.g., processor 302) to correct for distortion imparted by an optical assembly of an image viewer. Figure 4 One or more of the illustrated operations can be performed as part of an image processing pipeline executed by system 300.

[0051] At operation 402, system 300 obtains a distortion factor. This can be performed in any suitable manner. For example, system 300 can access the distortion factor from any suitable memory or storage device communicatively coupled to system 300. In some examples, system 300 can additionally or alternatively obtain the distortion factor by accessing a pre-programmed distortion factor from the logic of an FPGA.

[0052] In some examples, system 300 can obtain the distortion factor by generating the distortion factor based on one or more factors or parameters associated with the operation of the image viewer. For example, system 300 can generate the distortion factor based on the focal length of optical assembly 204, the distance between display device 206 and optical assembly 204, the distance between user's eye 208 and optical assembly 204, the inter-pupillary distance between user's eye 208 and the other eye, the distortion caused by imaging device 210, and / or any other suitable factor or parameter.

[0053] At operation 404, system 300 obtains image data that represents an image that can be displayed by means of the image viewer. In some examples, the image can be captured by an imaging device associated with a computer-assisted surgery system. This can be accomplished in any suitable manner. For example, system 300 can receive image data captured by imaging device 210 during a surgical procedure performed by the computer-assisted surgery system.

[0054] At operation 406, system 300 modifies the image data based on the distortion factor to correct for distortion imparted by one or more optical elements of the image viewer. This can be accomplished in any suitable manner. For example, system 300 can pre-distort the image data using at least some of the pixels included in a plurality of pixels of the image. By way of illustration, Figure 5 example illustration 500 is shown depicting a modified image 502 related to a pixel region 504 of a display device. Figure 5Illustrates the amount of distortion that can be applied to an image in certain examples. Pixel region 504 represents the size that would typically be displayed in the absence of any distortion of the image data. The modified image 502 can include repositioning and / or distortion of the per-pixel image data. This can be done in any suitable manner. For example, the pixel grid of the display device can be mapped back to the source image, and the image data can be interpolated from the source image. In such examples, whenever the image data is not perfectly aligned with a particular pixel orientation, system 300 can sample and interpolate from the source image. As a result, when the modified image 502 is viewed through the optical assembly of the image viewer, the modified image 502 appears undistorted to the user.

[0055] In certain examples, system 300 can modify the image data row by row for each pixel included in the image data.

[0056] In certain alternative examples, system 300 can modify the image data only for a subset of the pixels included in a plurality of pixels of the image data. In such examples, less distortion can be applied to certain portions of the image viewed through the optical assembly than to other portions. For example, pixels along the vertical centerline of the image can appear undistorted or can appear less distorted than pixels at the corners of the image. In such examples, system 300 can be configured to pre-distort only the pixel values of a subset of the pixels included in a plurality of pixels of the image. For example, the pre-distortion can be applied only to pixels that are more than a predefined threshold distance from the vertical centerline of the image.

[0057] In certain examples, system 300 can automatically switch from modifying each pixel to modifying only a subset of the pixels based on one or more factors associated with the operation of the computer-assisted surgical system. For example, system 300 can automatically switch to modifying only a subset of the pixels based on bandwidth constraints, the type of surgical procedure (e.g., some types of surgical procedures may be less complex and require less distortion), and / or any other suitable factor.

[0058] Return Figure 4 , at operation 408, system 300 directs the display device to display an image based on the modified image data. This can be done in any suitable manner. For example, system 300 can direct the display device to display the modified image, such as Figure 5 the modified image 502 that uses less of the pixel region 504 as shown in

[0059] In some examples, one or more factors associated with the operation of the image viewer can change in a manner that affects the amount and / or type of distortion imparted to an image viewed through the image viewer. For example, the distance between the user's eye 208 and the optical assembly 204 can change during use by the user physically moving the eye 208 away from the image viewer 202. Such a change can produce more or less distortion visible in the image viewed through the image viewer 202. Additionally or alternatively, the presence or absence of certain lenses (e.g., in a system where lenses are swapped in and out) can produce a change that affects the amount of distortion. Additionally or alternatively, an optical element that changes shape (e.g., a deformable mirror, an adjustable lens, etc.) can produce a change that affects the amount of distortion. Additionally or alternatively, a change in the thermal state of the image viewer can affect the amount of distortion. Additionally or alternatively, the relative distances of the elements in the optical assembly can change, which can affect the amount of distortion. Additionally or alternatively, the location at which the user views in a surgical scenario while using the image viewer can affect the amount of distortion that needs to be corrected. Thus, at operation 410, the system 300 determines whether a factor associated with the operation of the image viewer has changed. If the answer at operation 410 is "no", the process can return to operation 408. If the answer at operation 410 is "yes", the system 300 can obtain an updated distortion factor that takes into account the factor change at operation 412. The process can then return to operation 406, in which the system 300 can modify the image data based on the updated distortion factor.

[0060] Operation 406 can be repeated any suitable number of times to ensure that the distortion caused by one or more optical elements of the image viewer is dynamically corrected during the operation of the image viewer. In some examples, the system 300 can continuously monitor factors associated with the operation of the image viewer 202 to ensure that distortion is corrected as the factors change during use.

[0061] In some examples, the system 300 can be configured to minimize the amount of time used to process the image data to correct the distortion. For example, the system 300 can process the image data such that the latency from image capture to display of the image via the image viewer is less than 50 milliseconds.

[0062] In some additional or alternative examples, distortion caused by the optical assembly of the image viewer and / or other components associated with the image viewer can be corrected by the hardware configuration of a display device (e.g., display device 206). In such examples, the display device can include pixels that are arranged in an array based on a distortion factor to correct the distortion imparted by the optical assembly. This can be done in any suitable manner. For example, the arrangement of the pixels of the display device can be non-linear such that when viewing an image displayed by the display device through the optical elements of the image viewer, the image appears undistorted to the user. For illustration, Figure 6 a simplified example of a display device 600 is shown that includes a plurality of pixels 602 (e.g., pixels 602-1 to 602-3) arranged in a non-linear manner. In Figure 6 the example shown, the pixels are arranged in a barrel configuration, which can facilitate correction of the pincushion distortion that would otherwise be observed when viewing the image through the optical assembly.

[0063] Figure 6 The number, shape, and / or arrangement of the pixels 602 shown are provided for illustrative purposes only. It should be understood that any suitable number and / or arrangement of pixels can be used, which can serve a particular implementation. For example, in some implementations, the arrangement of the pixels can be a non-linear shape.

[0064] In some additional or alternative examples, distortion caused by the optical assembly of the image viewer and / or other components associated with the image viewer can be corrected in hardware by using a deformable optical device (deformable mirror, deformable lens, adjusting the laser scanning pattern, etc.). The system 300 can be configured in any suitable manner to control the deformation of such a deformable device to correct the distortion in the image viewable through the image viewer. For example, as one or more factors associated with the image viewer change during the operation of the image viewer, the system 300 can dynamically control the deformation of the deformable element.

[0065] Figure 7 An example method 700 for correcting distortion caused by the optical elements of an image viewer of a computer-assisted surgical system is illustrated. Although Figure 7 an example operation according to one embodiment is shown, other embodiments can omit, add, reorder, and / or modify Figure 7 any of the operations shown. Figure 7 One or more of the operations shown can be performed by a system such as system 300, any of the components included therein, and / or any of its implementations.

[0066] At operation 702, an image processing system (e.g., image processing system 300) obtains a distortion factor that represents the amount of distortion imparted to an image displayed by means of an image viewer of a computer-assisted surgical system by one or more optical elements of the image viewer. Operation 702 can be performed in any of the ways described herein.

[0067] At operation 704, the image processing system obtains image data that represents an image displayable by means of the image viewer. Operation 704 can be performed in any of the ways described herein.

[0068] At operation 706, the image processing system modifies the image data based on the distortion factor to correct for the distortion imparted by one or more optical elements of the image viewer, the modification resulting in modified image data representative of the image. Operation 706 can be performed in any of the ways described herein.

[0069] At operation 708, the image processing system directs the computer-assisted surgical system to display the image by means of the image viewer based on the modified image data. Operation 708 can be performed in any of the ways described herein.

[0070] Although the foregoing disclosure describes correcting for distortion imparted to an image captured by an imaging device by one or more optical elements of an image viewer, it should be understood that the distortion correction concepts described herein can be applied to any type of content or combination thereof that is displayable by means of an image viewer. For example, distortion correction can be applied to one or more images and / or other content that are not captured by an imaging device, such as computer-generated renderings, augmented reality content, virtual reality content, user interface elements, text, graphics, images, and / or video supplied by a third party (e.g., a customer), and / or any other type of content.

[0071] In some examples, a non-transitory computer-readable medium storing computer-readable instructions can be provided in accordance with the principles described herein. When executed by a processor of a computing device, the instructions can direct the processor and / or the computing device to perform one or more operations, including one or more of the operations described herein. These instructions can be stored and / or transmitted using any of a variety of known computer-readable media.

[0072] As used herein, a non-transitory computer-readable medium may include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of the computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Illustrative non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory ("RAM"), and optical disks (e.g., compact disks, digital video disks, Blu-ray disks, etc.). Illustrative volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0073] Figure 8 An example computing device 800 is illustrated, which may be specifically configured to perform one or more of the processes described herein. As Figure 8 shown, the computing device 800 may include a communication interface 802, a processor 804, a storage device 806, and an input / output ("I / O") module 808 that are communicatively coupled to each other via a communication infrastructure 810. Although Figure 8 the example computing device 800 is shown, Figure 8 the components shown are not intended to be limiting. In other embodiments, additional or alternative components may be used. The components of the computing device 800 shown will now be described in more detail. Figure 8 the components of the computing device 800 shown.

[0074] The communication interface 802 may be configured to communicate with one or more computing devices. Examples of the communication interface 802 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connector, and any other suitable interface.

[0075] The processor 804 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 804 may perform operations by executing computer-executable instructions 812 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 806.

[0076] The storage device 806 may include one or more data storage media, devices, or configurations, and may employ any type, form, and combination of data storage media and / or devices. For example, the storage device 806 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data (including the data described herein) may be stored temporarily and / or permanently in the storage device 806. For example, data representing computer-executable instructions 812 configured to direct the processor 804 to perform any of the operations described herein may be stored in the storage device 806. In some examples, the data may be arranged in one or more databases located within the storage device 806.

[0077] The I / O module 808 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 808 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 808 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0078] The I / O module 808 may include one or more devices for presenting output to the user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In certain embodiments, the I / O module 808 is configured to provide graphics data to a display for presentation to the user. The graphics data may represent one or more graphical user interfaces and / or any other graphical content, which may serve a particular implementation.

[0079] In some examples, any one of the systems, computing devices, and / or other components described herein may be implemented by the computing device 800. For example, the processor 302 may be implemented by the processor 804.

[0080] In the foregoing description, various example embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes can be made thereto, and additional embodiments can be implemented, without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substitute for features of another embodiment described herein. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A system, comprising: one or more processors configured to execute a process, the process including: obtaining a distortion factor representative of an amount of distortion imparted to an image displayed by an image viewer of a computer - assisted surgical system by one or more optical elements of the image viewer; obtaining image data representative of an image displayable by the image viewer; modifying the image data based on the distortion factor to correct the distortion imparted by the one or more optical elements of the image viewer, the modification resulting in modified image data representative of the image; and directing the computer - assisted surgical system to display the image based on the modified image data by the image viewer.

2. The system according to claim 1, wherein the modification of the image data includes pre - distorting pixel values of a subset of pixels included in a plurality of pixels of the image.

3. The system according to claim 1, wherein the one or more processors include a field - programmable gate array (FPGA).

4. The system according to claim 3, wherein the obtaining of the distortion factor includes accessing a pre - programmed distortion factor from the logic of the FPGA.

5. The system according to claim 1, wherein the obtaining of the distortion factor includes generating the distortion factor based on one or more factors associated with the operation of the image viewer.

6. The system according to claim 1, wherein the process further includes: detecting a change in one or more factors associated with the operation of the image viewer; obtaining an updated distortion factor based on the change in the one or more factors; and modifying the image data based on the updated distortion factor to correct the distortion imparted by the one or more optical elements of the image viewer.

7. The system according to claim 1, wherein: the image is captured by an imaging device communicatively coupled to the computer - assisted surgical system; and a waiting time from image capture to display of the image by the image viewer is less than 50 milliseconds.

8. A computer - assisted surgical system, comprising: an image viewer, including: a display device; and an optical assembly through which a user views an image displayed by the display device, the optical assembly being disposed along an optical path between the display device and the user's eyes and imparting distortion to the image displayed by the display device when viewed through the optical assembly; wherein the display device is configured to: receive an image to be displayed by the image viewer; and display the image based on a distortion factor that corrects the distortion imparted by the optical assembly.

9. The computer - assisted surgical system according to claim 8, wherein the image viewer further comprises one or more processors configured to modify the image to be displayed by means of the image viewer device based on the distortion factor.

10. The computer - assisted surgical system according to claim 9, wherein the latency associated with the one or more processors modifying the image and the display of the image is less than 50 milliseconds.

11. The computer - assisted surgical system according to claim 9, wherein the one or more processors include a field - programmable gate array (FPGA).

12. The computer - assisted surgical system according to claim 8, wherein the display device includes pixels arranged in an array based on the distortion factor to correct the distortion imparted by the optical assembly.

13. The computer - assisted surgical system according to claim 12, wherein the arrangement of the pixels of the display device is non - linear.

14. The computer - assisted surgical system according to claim 8, further comprising one or more reflectors disposed along the optical path between the display device and the optical assembly, the one or more reflectors changing the direction of the optical path.

15. The computer - assisted surgical system according to claim 8, further comprising: one or more manipulator arms configured to hold instruments; and one or more actuators for controlling the one or more manipulator arms.

16. The computer - assisted surgical system according to claim 15, wherein: the image is captured by an imaging device communicatively coupled to the computer - assisted surgical system; and the imaging device is engaged with a manipulator arm included in the one or more manipulator arms.

17. The computer - assisted surgical system according to claim 15, wherein the image viewer is part of a surgeon's console communicatively coupled to the one or more manipulator arms and the imaging device.

18. A method, comprising: obtaining, by an image processing system, a distortion factor representing the amount of distortion imparted to an image to be displayed by means of an image viewer of a computer - assisted surgical system by one or more optical elements of the image viewer; obtaining, by the image processing system, image data representing an image displayable by means of the image viewer; modifying, by the image processing system and based on the distortion factor, the image data to correct the distortion imparted by the one or more optical elements of the image viewer, the modification resulting in modified image data representing the image; and guiding, by the image processing system, the computer - assisted surgical system to display the image by means of the image viewer based on the modified image data.

19. The method according to claim 18, wherein the modification of the image data includes pre-distorting pixel values of a subset of pixels included in a plurality of pixels of the image.

20. The method according to claim 18, further comprising: detecting, by the image processing system, a change in one or more factors associated with an operation of the image viewer; obtaining, by the image processing system and based on the change in the one or more factors, an updated distortion factor; and modifying, by the image processing system and based on the updated distortion factor, the image data to correct the distortion imposed by the one or more optical elements of the image viewer.

Images