Medical device, system, and method using gaze tracking

Abstract

To provide a teleoperational medical system for performing a medical procedure in a surgical field.SOLUTION: The teleoperational medical system comprises an eye tracking unit 200 and a control unit. The eye tracking unit includes an image display 202a, 202b configured to display to a user an image of a surgical field, at least one eye tracker 204a, 204b configured to measure data about a gaze point of the user, and a processor 206 configured to process the data to determine a viewing location in the displayed image at which the gaze point of the user is directed. The control unit is configured to control at least one function of the teleoperational medical system on the basis of the determined viewing location.SELECTED DRAWING: Figure 2B

Description

Technical Field

[0001] This patent application claims the benefit of the filing date and priority of U.S. Provisional Patent Application No. 61 / 955,314, filed on March 19, 2014, entitled "Medical Devices, Systems, and Methods Using Eye Gaze Tracking", and U.S. Provisional Patent Application No. 61 / 955,355, filed on March 19, 2014, entitled "Medical Devices, Systems, and Methods Using Eye Gaze Tracking for Secondary Imaging", the entire contents of which are hereby incorporated by reference in their entirety.

Background Art

[0002] Surgery can be performed in a minimally invasive manner using a teleoperated medical system. The advantages of minimally invasive surgery are well known and include less patient trauma, less blood loss, and faster recovery times when compared to traditional open incision surgery. In addition, the use of teleoperated medical systems, such as the DA VINCI® surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California, is known. Such teleoperated medical systems may allow a surgeon to operate with intuitive control and increased precision when compared to manual minimally invasive surgery.

[0003] The teleoperation medical system may include one or more instruments coupled to one or more robotic arms. When the system is used for performing minimally invasive surgery, the instruments may access the surgical area through one or more small incisions or through one or more small openings of the patient, such as natural openings like the mouth, urethra, or anus. In some cases, instead of directly inserting the instrument through the opening(s), a cannula or other guiding element can be inserted into each opening, and the instrument can be inserted through the cannula to access the surgical area. An imaging tool such as an endoscope can be used to view the surgical area, and the images captured by the imaging tool can be displayed on an image display for the surgeon to view during the surgery.

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is desirable to provide a teleoperation medical system that can be effectively controlled and monitored for various uses during minimally invasive medical procedures. The systems and methods disclosed herein overcome one or more of the deficiencies of the prior art.

Means for Solving the Problems

[0005] In one exemplary aspect, the present disclosure is directed to a teleoperation medical system having an eye tracking unit and a control unit. In one aspect, the eye tracking unit includes an image display configured to display an image of the surgical field to the user, and at least one eye tracker configured to measure data regarding the gaze point of the user. In one aspect, the eye tracking unit includes a processor configured to process the data to determine the viewing location in the displayed image at which the user's gaze point is directed. In one aspect, the control unit is configured to control at least one function of the telemedical system based on the determined viewing location.

[0006] In other exemplary embodiments, the present disclosure is directed to a method for operating a telemedicine system. In one embodiment, the method includes displaying an image, including an operative field image, on an image display. In one embodiment, the method includes measuring a user's gaze point on the image display. In one embodiment, the method includes determining a viewpoint in the displayed image at which the user's gaze point is directed. In one embodiment, the method includes actuating at least one function of the telemedicine system based on the determined viewpoint.

[0007] In other exemplary embodiments, the present disclosure is directed to a telemedicine system having a first visual target tracking unit and a second visual target tracking unit. In one embodiment, the first visual target tracking unit includes one or more first image displays, one or more first eye trackers, and a first processor coupled to the one or more first eye trackers and configured to calculate a first gaze point of a first user when the first user views a first image displayed by the one or more first image displays. In one embodiment, the second visual target tracking unit includes one or more second image displays, one or more second eye trackers, and a second processor coupled to the one or more second eye trackers and configured to calculate a second gaze point of a second user when the second user views a second image displayed by the one or more second image displays. In one embodiment, the one or more first image displays are coupled to the second processor. In one embodiment, the one or more second image displays are coupled to the first processor.

[0008] In other exemplary embodiments, the present disclosure is directed to a method for operating a remotely operated medical system. In one embodiment, the method includes tracking eye gaze dynamics in a 3D image display of a surgical site. In one embodiment, the method includes determining a state of a user while the user is viewing the 3D image display.

[0009] In other exemplary embodiments, the present disclosure is directed to a method for operating a surgical system. In one embodiment, the method includes determining a 3D fixation point for a first user looking at a 3D image on a first display and displaying the 3D fixation point on a 3D image on a second display. In one embodiment, the method includes receiving a command from a second user looking at the 3D fixation point of the first user on the second display.

[0010] In other exemplary embodiments, the present disclosure is directed to a method for operating a surgical system having an instrument and a 3D display. In one embodiment, the method includes displaying a 3D image on the 3D display and determining the position of a 3D fixation point of a user looking at the 3D image. In one embodiment, the method includes comparing the 3D image and the position of the 3D fixation point.

[0011] These and other embodiments are described in further detail below with respect to the following figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying drawings. It is emphasized that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily enlarged or reduced for clarity of discussion. Also, the present disclosure may repeat reference numerals and / or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself define a relationship between the various embodiments and / or configurations being discussed.

Figure 1A

Figure 1B

Figure 1C

Figure 1D

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 3A

Figure 3B

Figure 3C

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 7A

Figure 7B

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 8E

Figure 9

Figure 10A

Figure 10B

Figure 10C

DETAILED DESCRIPTION OF THE INVENTION

[0013] For the purpose of facilitating an understanding of the principles of the present disclosure, reference is now made to the embodiments illustrated in the drawings, and a specific language is used to describe them. Nevertheless, it will be understood that no limitation of the scope of the present disclosure is intended. In the following detailed description of aspects of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the present invention.

[0014] Any alternatives and further modifications to the apparatus, devices, methods, and any further applications of the principles of this disclosure are fully contemplated as would be normally envisioned by one of ordinary skill in the art to which this disclosure pertains. Specifically, it is fully contemplated that the features, components, and / or steps described for one embodiment can be combined with the features, components, and / or steps described for other embodiments of this disclosure. Numerous repetitions of these combinations are not separately described. Additionally, the dimensions provided herein are for specific examples, and it is contemplated that different sizes, dimensions, and / or ratios may be used to implement the concepts of this disclosure. To avoid unnecessary repetition in the description, one or more components or operations described in accordance with one exemplary embodiment can be used as needed in other exemplary embodiments or omitted as needed from other exemplary embodiments. For simplicity, in some cases, the same reference numbers are used throughout the drawings to refer to the same or similar parts.

[0015] The following embodiments describe various devices and parts of devices from the perspective of their states in three-dimensional space. As used herein, the term "position" refers to the location of an object or a part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational arrangement of an object or a part of an object (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or a part of an object in at least one degree of freedom of translational degrees of freedom and the orientation of an object or a part of an object in at least one degree of freedom of rotational degrees of freedom (up to a total of six degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an elongated object.

[0016] The terms "proximal" and "distal" are understood in this specification to be used in relation to a clinician who manipulates the end of an instrument extending from the clinician to a surgical site. The term "proximal" refers to the portion of the instrument closer to the clinician, and the term "distal" refers to the portion of the instrument farther from the clinician and closer to the surgical site. For brevity and clarity, terms related to space such as "horizontal", "vertical", "above", and "below" may be used in this specification with respect to the drawings. However, surgical instruments are used in many directions and positions, and these terms are not intended to be limiting and absolute.

[0017] The present disclosure generally relates to a foveal tracking system for observing and measuring (e.g., gaze tracking) the characteristics of a user's eyes during the use of a teleoperated medical system and / or instrument, including, but not limited to, diagnostic, surgical, and / or therapeutic procedures. In particular, in some embodiments, the foveal tracking systems disclosed herein rely on the ability to track the exact position (e.g., 2D or 3D position) of a user's gaze to a surgical console, display system, or other medical or surgical system component. In some embodiments, the foveal tracking system can be used to control a teleoperated system by directly manipulating the system instrument and / or by affecting the characteristics of the system to cause a change in the overall system. In particular, some embodiments of the present disclosure relate to system and instrument control, particularly system and instrument control by tracking an operator's gaze while the operator uses a teleoperated medical system during a minimally invasive procedure. In some embodiments, multiple foveal tracking systems (e.g., for trainers / test supervisors and students) can be used together to enable testing and training supervision through a given procedure. In some embodiments, the foveal tracking system can be used to obtain performance metrics in operating a teleoperated system during a given procedure or to evaluate user skill. In particular, in some embodiments, a foveal tracking system incorporated into a telemedicine system can track a surgeon's gaze to evaluate the surgeon's skill level, consistency, physical state, and / or any other performance measure during surgery. Those skilled in the art will recognize that the foveal tracking systems disclosed herein can be used in similar (e.g., non-teleoperated) implementations that benefit from system / instrument control, training / test supervision, and / or performance evaluation. By using the foveal tracking systems and methods disclosed herein, a user can experience a more intuitive and efficient interaction with a teleoperated medical system.

[0018] According to various embodiments, minimally invasive medical procedures can be performed using a teleoperation system to guide instrument delivery and manipulation. Referring to FIG. 1A of the drawings, for example, a teleoperation medical system for use in a medical procedure including a diagnosis, treatment, or surgical procedure is generally indicated by reference numeral 10. As described, the teleoperation medical system of the present disclosure is under the remote control of a surgeon. In an alternative embodiment, the teleoperation medical system may be under partial control of a computer programmed to perform a procedure or sub-procedure. In yet another alternative embodiment, a fully automated medical system under full control of a computer programmed to perform a procedure or sub-procedure may be used to perform the procedure or sub-procedure. As shown in FIG. 1, the teleoperation medical system 10 generally includes a teleoperation assembly 12 located near or attached to an operating table O on which a patient P is positioned. The teleoperation assembly 12 may be referred to as a patient-side manipulator (PSM). A medical instrument system 14 is operably coupled to the teleoperation assembly 12. An operator input system 16 enables a surgeon or other type of clinician S to view an image of the surgical site or a display of the surgical site and to control the operation of the medical instrument system 14. The operator input system 16 may be referred to as a master console or a surgeon's console. One example of a teleoperation surgical system that can be used to implement the systems and techniques described in this disclosure is the da Vinci® surgical system manufactured by Intuitive Surgical, Inc. of Sunnyvale, California.

[0019] The remote operation assembly 12 supports the medical instrument system 14 and may include one or more non-servo control links (e.g., one or more links manually positioned and locked in a predetermined position, generally referred to as a setup structure) and the kinematic structure of a remote operation manipulator (see, e.g., FIG. 2). The remote operation assembly 12 includes a plurality of motors that drive the input portion of the medical instrument system 14. These motors move in response to commands from the control system 22. The motors include a drive system that can advance the medical instrument into an anatomical orifice, natural or surgically created, when coupled to the medical instrument system 14. Other motor drive systems can move the distal end of the medical instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian coordinate axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian coordinate axes). Further, the motors can be used to operate the articulatable end effector of the instrument.

[0020] The remote operation medical system 10 also includes an image capture system 18, including an image capture device such as an endoscope, and associated image processing hardware and software. The remote operation medical system 10 also includes a control system 12 operably coupled to the sensors, motors, actuators, and other components of the remote operation assembly 12, the operator input system 16, and the image capture system 18.

[0021] The operator input system 16 can be located on the surgical medical console, which is typically located in the same room as the operating table O. However, it should be understood that the surgeon S can be located in a different room from the patient P or even in a completely different building. The operator input system 16 generally includes one or more control devices for controlling the medical instrument system 14. More specifically, in response to the input commands of the surgeon, the control system 22 causes the servo mechanism movement of the medical instrument system 14. The control device(s) may include one or more of any number of various input devices such as a handgrip, a joystick, a trackball, a data glove, a trigger gun, a manual operation control device, a foot operation control device, a voice recognition device, a touch screen, a body movement or presence sensor, etc. In some embodiments, the control device(s) has the same degree of freedom as the medical instrument of the remote operation assembly in order to provide the surgeon with a perception such that the control device(s) is integrated with the instrument so that the surgeon has a strong sense of directly controlling the instrument as if being at the surgical site. In other embodiments, the control device(s) can have more or fewer degrees of freedom than the associated medical instrument and still provide telepresence to the surgeon. In some embodiments, the control device(s) is a manual input device that moves in six degrees of freedom and may also include an actuable handle for actuating the instrument (for example, closing the grasping jaws, applying a potential to an electrode, delivering drug therapy, etc.).

[0022] The system operator views an image captured by the image capture system 18 that is presented for viewing on a display system 20 that is operably coupled to or incorporated into the operator input system 16. The display system 20 displays an image or representation of the surgical site and the medical instrument system(s) 14, such as may be generated by a subsystem of the image capture system 18. The display system 20 and the operator input system 16 can be oriented so that the operator can control the medical instrument system 14 and the operator input system 16 with a sense of telepresence. The display system 20 can include multiple displays, such as separate right and left displays for presenting separate images to each of the operator's eyes, thereby enabling the operator to view a stereoscopic image.

[0023] Alternatively or additionally, the display system 20 can present images of the surgical site recorded and / or imaged before or during surgery using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube x-ray imaging, etc. The presented pre-operative or intra-operative images can include two-dimensional, three-dimensional, or four-dimensional (e.g., including time-based or velocity-based information) images and associated image datasets for reconstructing the images.

[0024] The control system 22 includes at least one memory and at least one processor (not shown), typically a plurality of processors, for performing control among the remote operation system 12, the medical instrument system 14, the operator input system 16, the image capture system 18, and the display system 20. The control system 22 also includes programmed instructions (e.g., a computer-readable medium storing the instructions) for implementing some or all of the described methods according to the aspects disclosed herein. Although the control system 22 is shown as a single block in the schematic diagram of FIG. 1, the system may include two or more data processing circuits where some parts of the processing are optionally executed in or adjacent to the remote operation assembly 12 and other parts of the processing are executed in, for example, the operator input system 16. Any of a variety of centralized or distributed data processing architectures may be used. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be incorporated into a number of other aspects of the remote operation system described herein. In one embodiment, the control system 22 supports wireless communication protocols such as Bluetooth®, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0025] In some embodiments, the control system 22 may include one or more servo controllers that receive force feedback and / or torque feedback from the medical instrument system 14. In response to the feedback, the servo controller transmits a signal to the operator input system 16. The servo controller(s) may also transmit a signal instructing the remote operation assembly 12 to move the medical instrument system(s) 14 that extends into an internal surgical site within the body through an opening of the patient's body. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separated from the remote operation assembly 12 or may be integrated with the remote operation assembly 12. In some embodiments, the servo controller and the remote operation assembly are provided as part of a remote operation arm cart positioned adjacent to the patient's body.

[0026] In this embodiment, the teleoperation medical system 10 also includes a visual target tracking unit 24 that can be operably coupled to or incorporated within the operator input system 16. The visual target tracking unit 24 is operably coupled to a control system 22 for detecting, measuring, recording, and transmitting information related to the operator's eyes while the operator is looking at the display 20 and / or operating the operator control unit of the operator input system 16.

[0027] The teleoperation medical system 10 may further include operating and support systems (not shown) such as a lighting system, a steering control system, a fluid management system such as a cleaning system, and / or a suction system. In an alternative embodiment, the teleoperation system may include more than one teleoperation assembly and / or more than one operator input system. The exact number of manipulator assemblies depends, among other factors, on the surgical procedure and the spatial constraints within the operating room. The operator input systems may be co-located or they may be located at separate positions. Multiple operator input systems allow more than one operator to control one or more manipulator assemblies in various combinations.

[0028] Figure 1B is a front view of a remote operation assembly 100 (e.g., the remote operation assembly 12 shown in Figure 1A) according to one embodiment. The assembly 100 includes a base 102 placed on the floor, a support tower 104 attached to the base 102, and several arms 106 that support a surgical instrument (including a portion of the image capture system 18). As shown in Figure 1B, arms 106a, 106b, 106c are instrument arms that support and move surgical instruments used to manipulate tissue, and arm 108 is a camera arm that supports and moves an endoscope. Figure 1B further shows replaceable surgical instruments 110a, 110b, 110c attached to each of the instrument arms 106a, 106b, 106c, and an endoscope 112 attached to the camera arm 108. The endoscope can be a stereoscopic endoscope for capturing a stereoscopic image of the surgical site and providing a separate stereoscopic image to the display system 20. Those skilled in the art will recognize that the arms that support the instruments and the camera can also be supported by a base platform (fixedly or movably) attached to the ceiling or wall, or in some examples, other parts of the surgical suite equipment (e.g., the operating table). Similarly, they will recognize that two or more separate bases can be used (e.g., one base supports each arm).

[0029] As further shown in FIG. 1B, instruments 110a, 110b, 110c, and endoscope 112 each include instrument interfaces 150a, 150b, 150c, and 150d and instrument shafts 152a, 152b, 152c, and 152d, respectively. In some embodiments, the remote operation assembly 100 may include a cannula support for fixing instruments 110a, 110b, 110c, and endoscope 112 relative to the cannula. In some embodiments, respective portions of instrument arms 106a, 106b, 106c, and 108 may be adjustable by operating room personnel to position instruments 110a, 110b, 110c, and endoscope 112 relative to the patient. Other portions of arms 106a, 106b, 106c, and 108 may be actuated and controlled by an operator of the operator input system 120 (as shown in FIG. 1C). Surgical instruments 110a, 110b, 110c, and endoscope 112 may also be controlled by an operator of the operator input system 120.

[0030] FIG. 1C is a front view of the operator input system 120 (e.g., the operator input system 16 shown in FIG. 1A). The operator input system 120 includes a console 121 with left and right multi-degree-of-freedom (DOF) control interfaces 122a and 122b, which are kinematic chains used to control the surgical instruments 110a, 110b, 110c, and the endoscope 112. The surgeon typically grasps the respective pincher assemblies 124a, 124b of the control interfaces 122 with the thumb and index finger and can move the pincher assemblies to various positions and orientations. When the tool control mode is selected, each of the control interfaces 122 is configured to control the corresponding surgical instrument and instrument arm 106. For example, the left control interface 122a can be coupled to control the instrument arm 106a and the surgical instrument 110a, and the right control interface 122b can be coupled to control the instrument arm 106b and the surgical instrument 110b. If a third instrument arm 106c is used during the surgical procedure and is located on the left side, the left control interface 122a can be switched from controlling the arm 106a and the surgical instrument 110a to controlling the arm 106c and the surgical instrument 110c. Similarly, if a third instrument arm 106c is used during the surgical procedure and is located on the right side, the right control interface 122b can be switched from controlling the arm 106b and the surgical instrument 110b to controlling the arm 106c and the surgical instrument 110c. In some examples, the control assignment between the control interfaces 122a, 122b, the combination of the arm 106a / surgical instrument 110a, and the combination of the arm 106b / surgical instrument 110b may be exchanged. This can be done, for example, when the endoscope is rotated 180 degrees so that the instrument moving in the endoscope field of view appears to be on the same side as the control interface the surgeon is moving. The pincher assembly is typically used to operate the jawed surgical end effector (e.g., pliers, grasping retractor, etc.) at the distal end of the surgical instrument 110.

[0031] The additional control device includes a foot pedal 128. Each of the foot pedals 128 can activate one specific selected function of the instrument 110. For example, the foot pedal 128 can activate a drill or a cautery tool or can operate a cleaning, suction, or other function. A plurality of instruments can be activated by pressing a plurality of the pedals 128. The specific function of the instrument 110 may also be activated by other control devices.

[0032] The surgical console 120 also includes a stereoscopic image viewer system 126 (e.g., the display system 20 shown in FIG. 1A). The stereoscopic image viewer system 126 includes a left eyepiece 125a and a right eyepiece 125b so that a surgeon can view left and right stereoscopic images in the stereoscopic image viewer system 126 using the surgeon's left and right eyes respectively. The left and right images captured by the endoscope 112 are output to corresponding left and right image displays, and the surgeon recognizes a three-dimensional image on the display system (e.g., the display system 20 shown in FIG. 1A). In an advantageous configuration, the control interface 122 is arranged below the stereoscopic image viewer system 126 such that the image of the surgical tool shown on the display appears to be located near the surgeon's hand below the display. This feature enables the surgeon to intuitively control various surgical instruments in the three-dimensional display as if directly looking at the hand. Thus, the associated instrument arm and instrument servo control are based on an endoscopic image reference frame.

[0033] The endoscopic image reference coordinate system is also used when the control interface 122 is switched to the camera control mode. In some cases, when the camera control mode is selected, the surgeon may move the distal end of the endoscope 122 by moving it together with one or both of the control interfaces 122. The surgeon can then intuitively move the stereoscopic image displayed by moving the control interface 122 as if the image were in his or her hand (e.g., panning, tilting, zooming).

[0034] As further shown in FIG. 1C, the headrest 130 is disposed over the stereoscopic image viewer system 126. When the surgeon is looking through the stereoscopic image viewer system 126, the surgeon's forehead is positioned relative to the headrest 130. In some embodiments of the present disclosure, the operation of the endoscope 112 or other surgical instrument can be accomplished through the operation of the headrest 130 instead of the utilization of the control interface 122. In some embodiments, the headrest 130 can include a pressure sensor, a rocker plate, an optically monitored slip plate, or other sensors capable of detecting the movement of the surgeon's head. Further details regarding the use of a sensing method for operating a headrest to control an endoscopic camera can be found, for example, in U.S. Application No. 61 / 865,996 entitled "ENDOSCOPE CONTROL SYSTEM", which is incorporated herein by reference.

[0035] FIG. 1D is a front view of the vision cart component 140 of the surgical system. For example, in one embodiment, the vision cart component 140 is part of the medical system 10 shown in FIG. 1A. The vision cart 140 can house a central electronic data processing unit 142 (e.g., all or part of the control system 22 shown in FIG. 1A) of the surgical system and a vision device 144 (e.g., part of the image capture system 18 shown in FIG. 1A). The central electronic data processing unit 142 includes much of the data processing used to operate the surgical system. However, in various implementations, the central electronic data processing can be distributed among the surgeon console 120 and the remote operation assembly 100. The vision device 144 can include camera control units for the left and right image capture functions of the endoscope 112. The vision device 144 can also include an illumination device (e.g., a xenon lamp) for providing illumination to image the surgical site. As shown in FIG. 1D, the vision cart 140 includes an optional touch screen monitor 146 (e.g., a 24-inch monitor), which may be attached elsewhere, such as on the assembly 100 or on the patient-side cart. The vision cart 140 further includes a space 148 for optional auxiliary surgical devices, such as an electrosurgical unit, insufflators, suction irrigation instruments, or a third-party cautery device. The remote operation assembly 100 and the surgeon console 120 are coupled to the vision cart 140, for example, via an optical fiber communication link, such that the three components together function as a single remote operation minimally invasive surgical system that provides an intuitive telepresence to the surgeon.

[0036] In some embodiments, some or all of the assembly 100 of the remote surgical system can be implemented within a virtual (simulated) environment, and note that some or all of the images viewed by the surgeon at the surgical console 120 can be artificial images of the instruments and / or anatomical structures. In some embodiments, such artificial images can be provided by the vision cart component 140 and / or can be generated directly at the surgical console 120 (e.g., by a simulation module).

[0037] During a typical minimally invasive surgery in the remote surgical system described with reference to FIGS. 1A - 1D, at least two incisions are made in the patient's body (usually using a trocar to place the associated cannula). One incision is for the endoscopic camera instrument and the other is for the surgical instruments. In some surgical procedures, several instruments and / or camera ports are used to provide access and imaging of the surgical site. The incisions are relatively small compared to the large incisions used for conventional open surgery, but the minimum number of incisions is desired to further reduce patient trauma and for improved cosmesis. In other embodiments, the remote medical system 10 can be used with a single incision access to the patient's anatomical structure or with access through natural openings such as the nose, mouth, anus, vagina, etc.

[0038] During typical remote surgery, a surgeon often needs to physically operate various control devices to control a surgical system, an imaging device, and / or other surgical instruments associated with the system. For example, the surgeon may need to adjust the field of view of the imaging device by physically operating a control unit to direct and affect the device. The surgeon may use his or her hands to manually control a joystick or mouse, or his or her feet to lightly tap a foot pedal on the surgeon console to log into the surgical system, to search for a target surgical site within the field of view of an endoscope, to manipulate the movement of a surgical instrument such as a clamp, and / or to adjust system or display settings. Conventional methods require the surgeon to free up one hand from the surgical action or use one foot to lightly tap the foot pedal, both of which may unnecessarily delay or interrupt the surgical action. For example, the hand or foot movement may redirect the surgeon's line of sight and attention from the target surgical site to the surgeon console, which can delay or interrupt the operation. After performing the necessary manual adjustments, the surgeon may need to spend additional time refocusing his or her attention and point of gaze back on the target surgical site.

[0039] The embodiments disclosed herein utilize gaze detection to improve the way one or more users (e.g., surgeons and / or trainees) interface with a surgical system. By converting a user's gaze (e.g., the 3D position of the user's gaze relative to a surgical console, display system, or other medical or surgical system component) into a command directed to the surgical system, the embodiments disclosed herein may enable faster and more efficient control of the telemedical system 10 than provided by conventional control methods. Gaze tracking, or eye tracking, is the process of measuring either the point-of-gaze (POG) (i.e., typically in 3D space, where the user is looking) or the movement of the eyes relative to the head. In other words, the POG is the point in space towards which a person's gaze is directed and is also defined as the point in space that is imaged at the center of the highest acuity region of the fovea of each eye.

[0040] FIG. 2A schematically shows a user U (e.g., surgeon S or test supervisor) with respect to an image display 151 (e.g., the image display system 20 shown in FIG. 1A) and a surgical field 155 (e.g., a region of the internal anatomical structure of patient P). The user (and his or her eyes) are present in a user 3D Cartesian reference coordinate system 160 (i.e., the user frame). For ease of understanding and to avoid redundancy, the term "Cartesian reference coordinate system" is simply referred to as "frame" for the remainder of this specification. The image display 151 is present in a two-dimensional or 3D image frame 165, and the surgical field is present in a surgical frame 170. Each frame 160, 165, 170 includes different dimensions and characteristics from the others. When the user, while in the first frame 160, moves his or her gaze with respect to the image display 151 in the second frame 165, the embodiments disclosed herein can convert the movement of his or her eyes into a control signal so as to correspondingly affect the telemedical system 10 that includes surgical instruments visible in the frame 165 of the display and present in the frame 170 of the surgical field.

[0041] In one aspect, eye tracking and the observation of other eye characteristics can be used generally to communicate with and / or affect the behavior of the teleoperated medical system 10. For example, the eye characteristics and dynamics observed by the eye tracking unit 24 shown in FIG. 1A can be used for surgeon recognition and login (e.g., in a manner similar to a retinal scan). This feature is described in further detail below in connection with FIGS. 4A and 4B. In some examples, the user's line of sight can be used to calibrate the 3D position of the surgical instrument in the surgical field frame 170 and to account for possible inaccuracies in the remote robot arm kinematic chain. In some examples, the teleoperated medical system 10 can be configured to prevent the movement of the surgical instrument in the surgical frame 170 when the user's line of sight is not directed towards the image display frame 165 or a specific viewpoint within the frame 165 (i.e., to lock out the user). In some examples, the teleoperated medical system 10 can be configured to prevent the movement of the surgical instrument in the surgical frame 170 when the user's eye is not detected by the eye tracker (i.e., to lock out the user). In some embodiments, a user interface (e.g., a menu) can be overlaid on the image of the surgical field shown on the image display. The user's line of sight in the user frame 160 can be used to determine the viewpoint in the image displayed on the image display 151 in the frame 165 and to identify the user's selection among the user-selectable options of the user interface corresponding to the determined viewpoint. In some examples, the 3D position of the user's line of sight relative to the image frame 165 of the image display can determine the depth, position, and size of the user interface displayed on the image display 151. For example, the user interface can be automatically displayed at a depth, position, and size that best corresponds to the current 3D position of the user's line of sight, thereby minimizing the need for him or her to refocus his or her eyes to interact with the user interface.In some examples, the 3D position of the user's line of sight can be used to measure whether the user is viewing a stereo based on the observed dynamics between the two eyes. In some examples, the 3D position of the user's line of sight can be used to adjust the ergonomic settings (e.g., height, orientation, etc.) of the stereoscopic viewer so that the user can view the entire display (and conversely the user's line of sight can be determined across the entire screen), or to center the user's line of sight on the center of the screen.

[0042] In other embodiments, real-time gaze tracking can be used to control, for example, a separate surgical instrument coupled to the remote medical system 10, such as an imaging device and / or an energy supply device, at the surgical frame 170 to activate, deactivate, and otherwise control it. For example, the system 10 can be configured to activate the surgical instrument when the control system determines that the viewpoint with respect to the image on the image display coincides with the position of the surgical instrument at a predetermined time. In one embodiment, gaze detection can be used to determine where the user desires to direct the imaging device to define the field of view. The embodiments disclosed herein can be configured to automatically move the imaging device in the direction of the user's gaze so as to continuously maintain the user's desired field of view (e.g., the target surgical site) on the display without the user having to manually change the position or viewing angle of the imaging device. For example, in some embodiments, the user's gaze can be used to automatically center the field of view of the imaging device so as to correspond to the direction of the user's gaze. In some embodiments, the user's gaze can be used to switch the instrument from one modality to another. For example, in one example, the user's gaze can be interpreted to change the operating mode of the imaging device (e.g., switch between imaging modes such as color imaging, black and white imaging, fluorescence imaging, ultrasonic imaging, and / or any other imaging modality). Similarly, in other examples, the user can perform a specific pattern of blinking or other eye movements to change the operating mode of the imaging device (e.g., switch between imaging modes such as color imaging, black and white imaging, fluorescence imaging, ultrasonic imaging, and / or any other imaging modality).

[0043] In other examples, the line of sight direction may assist in labeling or otherwise marking an image displayed in real time of the surgical field. The surgeon can view the 3D position of the surgical field and confirm it with a secondary action (e.g., by pressing a separate button, by maintaining an extended gaze, or by blinking in a specific pattern) so as to place a virtual label on the surgical field and / or on the displayed image 150 to identify an area of interest of the anatomical structure.

[0044] In other examples, a particular surgical instrument may be actuated only when a visual marker tracking unit confirms that the surgeon's line of sight has been focused on that particular instrument for a predetermined period of time. For example, the teleoperation medical system 10 may be configured to require that the surgeon's line of sight be focused on the stapler instrument for a predetermined period of time before the stapler instrument is permitted to fire the stapler's needle. This can facilitate the intended activation of the instrument within the surgeon's field of view and prevent inadvertent activation of instruments that were out of the field of view and / or not being attended to. When the surgeon's line of sight is directed elsewhere for a particular period of time, the surgical instrument may be deactivated. In other embodiments, control over a particular surgical instrument may be transferred from a first user to a second user when the visual marker tracking unit confirms that the line of sight of the second user has been focused on that particular instrument for a predetermined period of time or in a manner focused on that particular instrument. Some of these embodiments are further described below with reference to FIG. 3C.

[0045] In other aspects, real-time gaze tracking can be used to facilitate the training or test supervision of a surgeon during a procedure. In one example, as shown in FIG. 2D, the teleoperated medical system 10 may include a set of separate surgical consoles and separate gaze tracking units for the surgeon and test supervision, and each set of gaze tracking units is configured to recognize and communicate the gaze movements of either the surgeon or the test supervisor so as to affect the operation of the telemedical system 10 and / or the surgical instruments. Exemplary methods of test supervision or training are described below with reference to FIGS. 3A and 3B. The 3D position of the surgeon's eyes can be displayed on an external 3D image display for the test supervisor (e.g., the surgeon performing the test supervision) to view and evaluate in order to provide real-time feedback and guidance. Similarly, the 3D position of the test supervisor's eyes can be displayed on an external 3D image display of the surgical console for the surgeon to view and be guided in real time. For example, during the operation of the teleoperated system 10 (e.g., during training or an actual surgery), it may be desirable to ensure that the surgeon is focusing on the correct part of the surgical site. By viewing the 3D position of the test supervisor's gaze on the image display within the surgical console, the surgeon can know where in the surgical field to look (e.g., by seeing where the test supervisor is looking within the surgical field). In some examples, the test supervisor's gaze can be captured from the image displays of various possible devices, including, by way of non-limiting example, vision carts 140, surgical consoles 120, dual or shared consoles, touch screen displays, and / or remote devices such as tablet devices. It may be desirable to ensure that the surgeon is focusing on the correct part of the surgical site. By viewing the 3D position of the test supervisor's gaze on the image display within the surgical console, the surgeon can know where in the surgical field to look (e.g., by seeing where the test supervisor is looking within the surgical field). In some examples, the test supervisor's gaze can be captured from the image displays of various possible devices, including, by way of non-limiting example, vision carts 140, surgical consoles 120, dual or shared consoles, touch screen displays, and / or remote devices such as tablet devices.

[0046] In some examples, the surgeon's image display can be changed in real time to reflect the 3D position of the examiner's line of sight. In one embodiment, when the system 10 detects that the examiner is looking at a particular region of his or her image display corresponding to a particular 3D position within the surgical frame 170, the system 10 can highlight or otherwise indicate the corresponding region of the surgeon's image display (e.g., the region of the surgeon's image display corresponding to the same 3D position within the surgical frame 170). For example, the surgeon's image display 151 can sharpen (e.g., increase the resolution) or brighten the region of the image display corresponding to the 3D position of the examiner's line of sight. In additional or alternative embodiments, the surgeon's image display can dim or blur the region of the image display corresponding to 3D positions not being looked at by the examiner's line of sight.

[0047] In other aspects, real-time gaze tracking can be used to evaluate and score a surgeon's skill during and / or after a procedure. The gaze tracking embodiments disclosed herein can be used to measure and quantify the skill level of a surgeon operating the telemedicine system 10 based on various eye characteristics including, but not limited to, fixation, saccades, and / or which part of the screen the gaze occupies. Additionally, tracking a surgeon's eye gaze dynamics and / or pupil diameter fluctuations in real time can be used to monitor the surgeon's state (e.g., stress level and / or workload). In some examples, the system 10 can be configured to provide a warning if a decline in that state is determined based on detected changes or patterns in the gaze. This feature is described in further detail below in connection with FIGS. 4A and 4B.

[0048] FIG. 2B shows some examples of a fixation tracking unit 200 that can be used by the telemedicine system 10 of FIGS. 1A, 1B, and 1C according to some embodiments of the present disclosure. As described above, gaze tracking, or fixation tracking, is the process of measuring either the POG (e.g., where the user is looking) or the movement of the eyes relative to the head. Thus, the fixation tracking unit 200 has devices for measuring user eye characteristics such as eye position and eye movement. There are several methods for measuring eye movement and gaze direction. Some methods use video images from which eye position is extracted, other methods use search coils, or are based on electrooculograms. In yet other methods, infrared light is emitted by a device having or communicating with an infrared camera or detector. The infrared light reflects from the user's retina and returns to the infrared camera or detector, and the amount of reflected infrared light is based on the direction of the person's gaze relative to the emitter. The user's fixation point in 3D space can be determined once the reflected IR light reaches a certain threshold during a period of time. Small movements of the gaze can be interpreted as blinks and are typically ignored.

[0049] In the described embodiment, the eye gaze tracking unit 200 includes left and right eyepieces 125a and 125b, left and right eye image displays 202a and 202b, left and right eye trackers 204a and 204b, and a processor 206. In other embodiments, the eye gaze tracking unit 200 may include a single eye tracker configured to track both the left and right eyes simultaneously (e.g., where the left and right eyes have separate image displays 202a, 202b). In some embodiments, the eye gaze tracking unit 200 further includes a reflection system and / or a light emitter for irradiating the surgeon's eyes for the eye tracker to track the point of fixation. In some embodiments, the reflection system may include a plurality of mirrors arranged to reflect light from the light emitter to the surgeon's eyes and to reflect the point of fixation of the surgeon's eyes to the eye tracker. Further details of various embodiments of the stereoscopic viewer 126 can be found, for example, in U.S. Provisional Application No. 61 / 955,334, filed Mar. 19, 2014, entitled "MEDICAL DEVICES, SYSTEMS, AND METHODS INTEGRATING EYE GAZE TRACKING FOR STEREO VIEWER", which is hereby incorporated by reference in its entirety.

[0050] In some embodiments, the endoscope 112 disposed in the remote operation assembly 100 can be operated to capture images of the surgical field during surgery, and these images are shown on the left and right image displays 202a and 202b. The images captured by the endoscope 112 can then be processed by the processor 206 to generate left and right stereoscopic images. In some examples, the processor 206 can be disposed, for example, on a vision cart as part of the central electronic data processing unit 142. Alternatively, the processor 206 can be disposed in the remote operation assembly 100 and / or the surgical console 120. In some embodiments, the visual target tracking unit 200 can also be integrated with a simulation module, such as the da Vinci® Skills Simulator®, used in the surgical console 120, where virtual images can be shown on the left and right image displays 208a and 208b.

[0051] Referring to FIG. 2B, the generated left and right stereoscopic images can be shown on the left and right image displays 202a and 202b respectively. The left and right eyepieces 125a and 125b include lenses, and the surgeon can view the right and left image displays 202a and 202b through the left and right eyepieces 125a and 125b with the surgeon's left and right eyes respectively. A 3D stereoscopic image of the surgical field can be perceived by the surgeon through the visual target tracking unit 200. In some embodiments, the distance between the left and right eyepieces 125a and 125b can be adjusted to accommodate different inter-pupillary distances of different users. In some embodiments, the left and right eyepieces 125a and 125b can be adjusted independently based on the requirements of the surgeon's left and right eye vision respectively. The left and right eye image displays can be 2D or 3D display screens. In some embodiments, the left and right eye image displays are liquid crystal display (LCD) screens.

[0052] Still referring to FIG. 2B, the left eye tracker 204a can be used to track the fixation point of the surgeon's left eye, and the right eye tracker 204b can be used to track the fixation point of the surgeon's right eye. In some embodiments, the visual target tracking unit 200 may also include a light emitter that can emit light to irradiate the surgeon's eyes so that the fixation points of the surgeon's left and right eyes can be captured by the left and right eye trackers 204a and 204b, respectively. The light emitter may or may not be integrated with the left and / or right eye trackers 204a and 204b. In some embodiments, the light emitter may be an infrared (IR) light emitter, such as an infrared light emitting diode (IR LED). In some embodiments, the left and right eyepieces 125a and 125b may include a suitable optical coating configured to minimize reflection and maximize transmission of light from the light emitter and / or the left and right eye image displays 202a and 202b. In some embodiments, the left and right eye trackers 204a and 204b may include a stereo camera. In some embodiments, the left and right eye trackers 204a and 204b are charge-coupled device (CCD) cameras. In some embodiments, the left and right eye trackers 204a and 204b are infrared (IR) cameras that are sensitive to IR light and can capture the IR light emitted from the IR light emitter. The left and right eye trackers 204a and 204b can be disposed in the stereo image viewer 126 and can be attached to the base portions of the left and right eye image displays 202a and 202b. The left and right eye trackers 204a and 204b and the left and right image displays 202a and 202b can be arranged in any suitable arrangement, as discussed in U.S. Provisional Application No. 61 / 955,334.

[0053] In some embodiments, the processor 206 is coupled to the left and right eye trackers 204a and 204b and is configured to calculate the 3D position of the surgeon's point of gaze with respect to the image frame 165 of the image display 151 and to transform that 3D position to the corresponding 3D position of the surgical frame 170 of the surgical field 155 (shown in FIG. 2A). For example, the points of gaze captured by the left and right eye trackers 204a and 204b can be rectified and the disparity between the left and right points of gaze of the surgeon can be determined. The 3D position of the surgeon's point of gaze can then be calculated using the distance between the left and right eye trackers 204a and 204b, parameters related to the respective focal lengths of the left and right eye trackers 204a and 204b, and the determined disparity. In some embodiments, the processor 206 is included in the visual target tracking imaging system 200 of the surgeon console 120. In some embodiments, the processor 206 is included in the vision cart 140 shown in FIG. 1D, for example, as part of the central electronic data processing unit 142. In some embodiments, the processor is part of the control system 22. In some embodiments, the processor 206 can also be coupled to a memory for storing 3D point of gaze measurements, alignment, and calibration data. In some embodiments, the processor 206 can be used to calculate the 2D position of the surgeon's point of gaze. In some embodiments, the calculated 2D or 3D position of the surgeon's point of gaze can be displayed in any of a variety of suitable representations, such as dots, flags, or vectors indicating changes in the surgeon's point of gaze, and the surgeon's point of gaze can be displayed in combination with an image of the surgical field 155 on the left and right image displays 202a and 202b.

[0054] In some embodiments, the movement of the surgeon's head / face during surgery can be tracked using the left and right eye trackers 204a and 204b. The calculated viewpoint, which can be located at the 2D or 3D fixation point position, can be further adjusted or compensated based on the tracked movement of the surgeon's head / face. Further details of the process of tracking and compensating for head / face movement can be found, for example, in U.S. Provisional Application No. 61 / 955334, and U.S. Provisional Application No. 61 / 865,996 entitled "ENDOSCOPE CONTROL SYSTEM", which are hereby incorporated by reference in their entirety.

[0055] The fixation tracking unit 200 can be coupled to an image display or a fixation point display 210 as shown in FIG. 2B. In some embodiments, the fixation point display 210 is a 3D fixation point display. In some embodiments, the fixation point display 210 is the same as the image display 151 shown in FIG. 2A. The generated 2D or 3D position of the surgeon's fixation point can be output to the fixation point display 210 as any suitable representation, such as a point, flag, or vector indicating the change in the surgeon's fixation point. An image of the surgical field 155 can be displayed on the fixation point display 210 in combination with the 3D position of the surgeon's fixation point. In some embodiments, the fixation point display 210 can be an external 3D image display. For example, the fixation point display 210 can be the 3D touch screen monitor 146 located on the vision cart 140 shown in FIG. 1D. In some examples, the fixation point display 210 can be attached to the remote operation assembly 100 shown in FIG. 1B. In some examples, the fixation point display 210 can be a portable display device, such as a tablet. In some examples, the fixation point display 210 can be presented simultaneously on multiple display screens or devices.

[0056] Accordingly, the visual target tracking unit 200 includes a fixation point display 210 configured to display an image to the user, at least one eye tracker configured to measure data reflecting the user's fixation point, and a processor 206 configured to process data to determine the viewing point of the image on the fixation point display 210 at which the user's fixation point is directed and to control at least one function of the teleoperated medical system 10 based on the determined viewing point. For example, in some embodiments, the processor 206 of the visual target tracking unit 200 may be coupled to an instrument control unit 212 configured to control the movement and energy emission of at least one surgical instrument. The instrument control unit 212 may be a component of the control system 22. The instrument control unit 212 may include a separate processor and one or more actuators for controlling the functions of one or more instruments. In some embodiments, the instrument control unit 212 is configured to control the activation, deactivation, and movement of one or more surgical instruments. In some embodiments, the processor of the instrument control unit 212 supplies control signals to one or more motors. For example, in one embodiment, the one or more motors may include a firing motor configured to release or fire a surgical instrument such as a stapler.

[0057] In some embodiments, the processor 206 of the visual target tracking unit 200 may be coupled to a system control unit 214 configured to adjust various system parameters and characteristics of the teleoperated medical system 100. The system control unit 214 may be a component of the control system 22. The system control unit 214 may include one or more separate processors. The system control unit 214 may include one or more user interfaces for providing interaction between the surgeon and the teleoperated medical system 100. In some examples, the user interface includes a surgical console, a display, a touch screen, or other suitable input device. The user interface may also include one or more software applications.

[0058] Figure 2C shows a flowchart 215 depicting an exemplary method of using the eye tracking unit 200 to control the teleoperation medical system 100 and / or any associated medical instrument and to affect the teleoperation medical system 100 and / or any associated medical instrument. Any of the method steps described herein may be implemented in the form of executable code stored in a non-transitory, tangible, machine-readable medium that is at least partially executed by one or more processors. In step 216, user U fixates on a particular viewpoint (i.e., 3D position) of the image shown on the image display 151 in the image frame 165 within the user frame 160 shown in FIG. 2A. In process 218, the left and right eye trackers 204a, 204b of the eye tracking unit 200 observe and measure the characteristics of user U's eyes (e.g., characteristics that reflect the line of sight). In some embodiments, the eye trackers 204a, 204b measure the line of sight of each of the user's eyes with respect to the image frame 165. In step 220, the processor 206 uses the measured line of sight data from the eye trackers 204a, 204b to determine the 3D viewpoint in the image on the image display 151 that the user's eyes are directed at (within the image frame 165). In some embodiments, the processor 206 may determine the position seen by tracking the angle of incidence of the light received by the eye trackers 204a, 204b from the reflection exiting the eye. In some embodiments, the processor 206 may first perform a calibration process (e.g., the calibration process 302 described in FIG. 4A) to determine the baseline angle of incidence when the user looks at a target indicia displayed at a known location on the image display 151 and generate a functional relationship between the detected angle and the location seen on the image display 151. The processor 206 can then track the angle of incidence when the user looks at other locations on the image display 151 and use the generated functional relationship to determine the corresponding location seen (extrapolating from the calibrated angles and locations).

[0059] In step 222, the processor 206 determines whether one of the marks (e.g., menu options) displayed on the image display 151 is being viewed by the user in a manner that satisfies a defined condition for selection of that mark (e.g., the mark is within the field of view and / or within the field of view for a predetermined period). If so, in step 224, the user's selection of the mark causes the processor 206 to activate a function corresponding to the displayed mark. For example, in some embodiments, the user's line of sight may indicate selection of a mark associated with logging onto the teleoperation medical system 100, or illuminating the image display 151, or various other system settings.

[0060] If not, in step 206, the processor 206 co-registers the viewed 3D position in the image frame 165 with the corresponding 3D position of the surgical field 155 in the surgical frame 170. In step 228, the processor determines whether the user is viewing the surgical field in a manner that satisfies a defined condition for operating another surgical instrument visible in the imaging device or on the image display 151. If so, in step 230, the user's line of sight to a particular region of the surgical field or a particular surgical instrument in the surgical field causes the processor 206 to affect the associated instrument in a manner corresponding to the characteristics of the user's line of sight. For example, in some embodiments, as described above, when the user gazes at a particular region of the surgical field 155, the imaging device may "follow" the user's line of sight and re-center its field of view (e.g., by placing the center of its field of view at the user's point of fixation). In other embodiments, when the user gazes at a particular surgical instrument for a predetermined time, the surgical instrument may be activated automatically or by a second user event (e.g., by pressing a pedal, footswitch, finger switch, etc.). If not, in step 232, the eye tracker continues to evaluate the user's line of sight for possible commands.

[0061] In some implementations, as shown in FIG. 2D, two or more surgical consoles 120 (either co-located or separated from each other) can be networked together so that two users can view the surgical site and control the tools simultaneously. In some embodiments, two different surgical consoles 120 can be used by an examiner and a trainee during a training process, so that each user can view a separate stereoscopic image displaying gaze data obtained from two separate gaze tracking units 200 (e.g., one gaze tracking unit 200 for each console 120). FIG. 2D shows a training / examining system that includes an examiner's gaze tracking unit 200 coupled to a trainee's gaze tracking unit 240, according to some embodiments of the present disclosure. The trainee's gaze tracking unit 240 can have substantially the same design and functionality as the examiner's gaze tracking unit 200. As shown in FIG. 2D, the processor 206 of the examiner's gaze tracking unit 200 is coupled to the left and right image displays 204a and 204b of the trainee's gaze tracking unit 240. Similarly, the processor 248 of the trainee's gaze tracking unit 240 is coupled to the left and right image displays 204a and 204b of the examiner's gaze tracking unit 200. Thus, the examiner's point-of-gaze display can be displayed to the trainee, and the trainee's point-of-gaze display can be displayed to the examiner.

[0062] For example, in some embodiments, the 3D fixation point of the test supervisor can be demonstrated on the 3D fixation point display 250 of the trainee's visual target tracking unit 240 so that the trainee can have a direct view of the test supervisor's fixation point in real time during the procedure. In some embodiments, the 3D fixation point of the test supervisor can be shown as a stereoscopic image on the left and right image displays 244a and 244b of the trainee's visual target tracking unit 240 so that the trainee can be assisted in real time to complete the surgery using the test supervisor's fixation point as a visual guide to follow. In some embodiments, during the training process, the 3D fixation point of the trainee can be shown on the display 210 or the left and right image displays 202a and 202b of the test supervisor's visual target tracking unit 200 so that the surgeon's ability can be monitored and evaluated in real time by the test supervisor. In other embodiments, both the test supervisor's and the trainee's fixation points can be shown on both the test supervisor's and the trainee's displays, thereby enabling one or both users to see any differences between their fixation points. In this way, the test supervisor can provide timely instructions to the trainee so that the trainee can focus on the correct surgical site in real time and avoid incorrect movements.

[0063] In some embodiments, the control of the endoscope 112 and / or other surgical instruments 110 can be switched between the test supervisor and the trainee. Alternatively, the endoscope 112 and / or other surgical instruments 110 can be operated simultaneously by both the test supervisor and the trainee. In FIG. 2D, the processors 206 and 248 are shown separately in the visual target tracking units 200 and 240, but those skilled in the art will recognize other variations. For example, the training / test supervision system can include one processor for both users' visual target tracking units.

[0064] Figure 3A shows a method 300 for determining a 3D fixation point of a surgeon in an image frame using the visual target tracking system 200 of FIG. 2B according to an embodiment of the present disclosure. The viewpoint of the surgeon can be a 3D fixation point as described. However, in alternative embodiments, a 2D fixation point can be used. Method 300 includes three processes: a calibration process 302, a measurement process 304, and an output process 306. In some embodiments, the calibration process 302 is a 3D calibration process, where the fixation point of the surgeon in the 3D space of the image frame is compared with a predetermined target in the 3D space of the image frame having known 3D position parameters.

[0065] The calibration process 302 begins at step 312 by indicating a target in the image frame. In some examples, the target can be a surgical tool icon. The target can be a moving target or a target that can dynamically change size. Alternatively, the target can be an actual surgical tool in the surgical field, and this position can be tracked and identified using any suitable tool tracking technique. For example, the calibration process may incorporate features disclosed in U.S. Patent Application Publication No. 2006 / 0258938, entitled "Methods and system for performing 3D tool tracking by fusion of sensor and / or camera derived data during minimally invasive robotic surgery," filed on May 16, 2005, which is hereby incorporated by reference in its entirety into this application. The image of the target shown in the 3D image frame can be separated into left and right stereo images and displayed on the left and right image displays 202a and 202b shown in FIG. 2B, respectively. During the calibration process 302, the 3D position of the target is predetermined such that, for example, using known 3D position parameters in the 3D image frame, the measured data can be compared with the known position parameters of the target to determine various models in subsequent steps.

[0066] In the described embodiment, calibration process 302 proceeds to step 314 by receiving the 2D position of the left and right pupils and the 2D corneal reflection data of the left and right eyes respectively captured by the left and right eye trackers 204a and 204b. In some embodiments, the 2D pupil position and 2D corneal reflection data may include coordinate values, displacements, and / or angles. In some embodiments, the movement of the surgeon's head / face may also be captured by the left and right eye trackers 204a and 204b.

[0067] Calibration process 302 proceeds to process 316 by using the 2D position of the surgeon's left and right pupils and the 2D corneal reflection data to determine the surgeon's pupil position and corneal reflection data. In some embodiments, the left and right eye trackers include stereo cameras, and a stereo image including the 2D pupil position and 2D corneal reflection data can be captured and processed by processor 206 to calculate the difference between the two stereo images. In some embodiments, the determined position data may include the 2D position of the surgeon's pupil and the 2D corneal reflection data. The determined 2D data for each eye can then be combined to estimate the surgeon's 3D line-of-sight position. In some embodiments, the determined position data may include the 3D position of the pupil and the 3D corneal reflection data. The 3D data including the depth of the surgeon's pupil and corneal reflection can be estimated using the difference. For example, the 3D data of the surgeon's left eye can be calculated using the distance between the left eye tracker 204a, the respective focal lengths of the left eye tracker 204a, and the calculated difference. The difference with respect to the depth conversion map can be obtained during the calibration process using this method. In some embodiments, the pupil position and corneal reflection data may be compensated for the captured head / face movement.

[0068] In step 318 of the calibration process 302, the determined 3D position of the pupil and the 3D corneal reflection data are compared with the predetermined 3D position parameters of a predetermined target to determine a gaze point determination model. In some embodiments, the gaze point determination model may include a function that can be used to map a 3D line-of-sight position using the determined position of the pupil and the corneal reflection data. In some embodiments, multiple calibration targets are used for the calibration process, and the parameters of the function can be determined using the pupil position and corneal reflection data collected from the calibration process. In some examples, methodologies such as least squares optimization or maximum likelihood estimation can be used to determine the parameters of the function. In some embodiments, the gaze point determination model may also include a matrix that represents the transformation from the 3D position of the pupil and the 3D corneal reflection data to the 3D position of the target T in the coordinate system in 3D space. In some embodiments, the gaze point determination model can be saved in a memory coupled to the processor 206. Further details on how the 3D position of the pupil and the 3D corneal reflection data are determined, as well as details regarding the gaze point determination model, can be found, for example, in U.S. Provisional Application No. 61 / 955,334, filed on March 19, 2014, which is hereby incorporated by reference in its entirety.

[0069] In some embodiments, the calibration process 302 can be repeated multiple times such that the accuracy of the gaze point determination model can be improved to meet a predetermined system requirement. In some embodiments, after a first gaze point determination model is formed, a real target can be used to estimate the accuracy of the first gaze point determination model. For example, by re-executing the mapping optimization using the real target, the first gaze point determination model can be updated to form a second gaze point determination model. The accuracy between the first and second models is compared and evaluated such that a more accurate gaze point determination model can be formed.

[0070] After the calibration process 302 is completed, the method 300 proceeds to the measurement process 304. The measurement process 304 can be performed during a surgical or training process when the endoscope 112 is capturing an image of the surgical site.

[0071] The measurement process 304 begins at step 320 by receiving 2D pupil position and 2D corneal reflection data for the surgeon's left and right eyes while the surgeon is viewing the surgical site displayed on the left and right image displays 202a and 202b. The image of the surgical site can be captured by the endoscope 112, processed, and separated into left and right stereoscopic images that are respectively displayed on the left and right image displays 202a and 202b. The 2D pupil position and 2D corneal reflection data for the surgeon's left and right eyes are captured by the left and right eye trackers 204a and 204b respectively. In some embodiments, the movement of the surgeon's head / face can also be captured by the left and right eye trackers 204a and 204b.

[0072] The measurement process 304 proceeds to step 322 by determining the 3D gaze point position of the surgeon using the gaze point determination model obtained during the calibration process 302. The 3D position of the pupil and the 3D corneal reflection data can first be determined using the 2D position of the pupil and the 2D corneal reflection data using substantially the same method as for step 316 of the method 300 described above. In some embodiments, the head / face movement captured at step 320 can also be used to compensate for the pupil position and corneal reflection data or the 3D gaze point of the surgeon. In some embodiments, during the calibration process 302, the corners of the surgeon's eyes can be tracked while the surgeon focuses on the calibration target using his or her pupil and rotates his or her head. A function between the head / face movement and the tracked movement of the eye corners can be formed during the calibration process. During the measurement process 304, the movement of the eye corners can also be tracked and the movement of the surgeon's head / face can be estimated using the function formed from the calibration process 302. The 3D position of the pupil and the 3D corneal reflection data can then be converted by the processor 206 to the 3D gaze point position of the surgeon by using the gaze point determination model obtained at step 318.

[0073] In step 324 of the output process 306, the determined 3D fixation point position can be shown on the fixation point display 210 shown in FIG. 2B. The 3D fixation point position can be represented in any of various suitable representations, such as, but not limited to, a point, a line, a vector, an arrow, and a translucent circle. In some embodiments, the 3D fixation point of the test supervisor (e.g., determined by the processor 206 of the test supervisor's eye tracking unit 200) can be shown on the 3D fixation point display 250 of the surgeon's eye tracking unit 240 (as shown in FIG. 2D) so that the trained surgeon can have a direct view of the test supervisor's fixation point. In some embodiments, the 3D fixation point of the test supervisor can be shown on the left and right image displays 244a and 244b of the surgeon's eye tracking unit 240 so that the surgeon can be guided to intuitively use the test supervisor's fixation point to complete the surgery. In some embodiments, the 3D fixation point of the surgeon (e.g., determined by the processor 2486 of the surgeon's eye tracking unit 240) can be shown on the display 210 of the test supervisor's eye tracking unit 200 so that the surgeon's fixation point can be monitored in real time by the test supervisor during the surgery. The fixation point measured by the teleoperation medical system 10 as described above can be used for various purposes.

[0074] Figure 3B shows a training / test supervision method 340 that uses the dual visual target tracking units 200 and 240 of Figure 2D. Method 340 begins at step 342 by monitoring the 3D gaze point of a first user by a second user on a 3D gaze point display during surgery. The 3D gaze point of the first user can be determined using a first visual target tracking unit (e.g., visual target tracking unit 200) as shown in method 300 shown in Figure 3A. The determined 3D gaze point can be displayed on a 3D gaze point display (e.g., image display 244 or gaze point display 250) of a second visual target tracking unit (e.g., visual target tracking unit 240) so that it can be monitored and evaluated by the second user. Method 340 proceeds to step 344 when the second user sends an instruction to the first user. In some embodiments, the first user may be a surgeon, and the second user may be a test supervisor who trains the surgeon during surgery. For example, in one example, when the test supervisor notices that the surgeon is looking at an area within a surgical site different from the target area within the surgical site, or that the surgeon has selected the wrong surgical instrument, the test supervisor can send an instruction to the surgeon to correct the surgeon's movement and / or the point being looked at in real time. For example, the test supervisor can instruct the surgeon by gaze-activated menu selection, manually pressing a button, or lightly tapping a foot pedal on the test supervisor's console. In one example, the test supervisor's instruction appears on the surgeon's image display in any of various ways to guide the surgeon towards the correct approach. In one embodiment, the surgeon's image display (e.g., left and right image displays 202a and 202b of visual target tracking unit 200) can be darkened. In other embodiments, the gaze point area of the surgeon's image display can present a visual (e.g., flashing red), audible (e.g., buzzing), or tactile (e.g., vibrating a part of the manual control unit) warning to alert the surgeon of proceeding with an incorrect action. In some embodiments, the test supervisor can also send an instruction using a telestrator or video marker on a touch screen or a 3D pointer on the surgeon's console or a separate console.

[0075] As described above, in some embodiments, a surgeon may use the surgeon's 3D line of sight to label a surgical site and identify the location of the surgical site. For example, when a surgeon wishes to label a location of an anatomical structure, the surgeon may fixate on that location such that the visual tracking unit 200 captures the surgeon's 3D line of sight and uses method 300 to determine 3D coordinate values. Next, the surgeon may further press a button on the surgeon console 120, or lightly tap the foot pedal 128, to label the location at the current line of sight on the image display using an icon, such as a flag. When the endoscope 112 focuses on other quadrants within the anatomical environment, the labeled 3D position may be compensated for by being referenced from an external position sensor for the next camera movement or to hold the labeled position. The labeled position may be used as a reference position that will help the surgeon to efficiently and effectively identify the desired surgical site.

[0076] Regarding method 300 shown in FIG. 3A, in some embodiments, the 3D fixation point position captured by the eye tracking unit 200 can be used to adjust the endoscope 112 such that the surgeon's 3D fixation point is at the center of the field of view. For example, when it is confirmed that the surgeon's 3D fixation point is at the surgical site, the surgeon can send a confirmation instruction, for example, by pressing a button and / or by lightly tapping a foot pedal. The control unit of the endoscope 112 can then receive the position data of the 3D fixation point from the processor 206 such that the endoscope 112 can be adjusted to capture an updated image having the surgeon's 3D fixation point located at the center of the image. Further details regarding using the 3D fixation point to adjust the endoscope 112 can be found in U.S. Provisional Application No. 61 / 955,355, filed on March 19, 2014, entitled "MEDICAL DEVICES, SYSTEMS, AND METHODS USING EYE GAZE TRACKING FOR SECONDARY IMAGING", which is hereby incorporated by reference in its entirety.

[0077] Figure 3C shows a fixation point activation method 350 for activating a surgical instrument (e.g., instrument system 14) attached to the remote operation assembly 100 using the surgeon's fixation point during surgery. The surgeon is first required to focus on an image of a surgical instrument to be used in the surgery, such as a fastener delivery tool (e.g., stapler) or an energy application tool (e.g., ablation instrument), through the fixation point tracking unit 200. The 3D position data of the surgical instrument in the image frame is stored in a computer-readable medium such as a memory. The 3D position data of the 3D image of the surgical instrument in the image frame is referenced to the 3D position of the surgical instrument in the surgical frame. In step 351, an image of the surgical instrument captured by the endoscope 112 is displayed on the display (e.g., on the image displays 202a and 202b of the fixation point tracking unit 200) for the surgeon to view. In step 352 of method 350, the fixation point of the surgeon in the image frame is determined using a method substantially similar to that shown in method 300 when the surgeon is required to focus on the image of the surgical instrument on the display. In some embodiments, the surgeon may focus the fixation point of his or her line of sight on the image of the surgical instrument or on an area of the image near the surgical instrument, such as tissue near the tip of the surgical instrument. In step 354, the determined position of the surgeon's 3D fixation point is compared with the 3D position data of the surgical instrument stored in the computer-readable medium. If the determined position of the surgeon's 3D fixation point coincides with the 3D position of the surgical instrument, or if the determined position of the surgeon's 3D fixation point is within a predetermined area near the surgical instrument, step 354 proceeds to step 356 to activate the surgical instrument. For example, the processor 206 of the line of sight tracking system may send a signal to the instrument control unit 212 to enable the energy release of a surgical instrument such as a stapler. The surgeon can then control and fire the stapler using the corresponding control interface 122. In some embodiments, the processor 206 can also activate and control the control interface 122 to deliver to the surgical instrument to be used by the surgeon for the surgery.If the determined position of the surgeon's 3D gaze point is not within a predetermined region near the surgical instrument, at step 358, the surgical instrument is deactivated. At step 358, since the surgeon is not looking at the instrument, the instrument control unit 212 cannot be activated by any of the surgeon's physical commands (e.g., this could be inadvertent), and the surgical instrument is locked. Thus, the surgical instrument cannot perform any firing action or movement without verification of the surgeon's gaze point as discussed in method 350.

[0078] During the training process, the current 3D gaze point of the surgeon in training can be monitored and evaluated by the test supervisor from various perspectives such as skill level and / or stress level. In some embodiments, the surgeon's skill level can be characterized by tracking gaze point movements during the surgical process. FIG. 4A shows a method 400 for evaluating a surgeon during surgery using the visual target tracking unit 200 of FIG. 2B. The evaluation can include the surgeon's skill level, stress level, fatigue, and / or any other gaze-indicated performance or behavioral metric. FIG. 5 is an example of a 3D image display 450 showing a 3D image of the surgical site in a 3D image frame coordinate system.

[0079] Method 400 begins with optional step 402 by obtaining baseline data as a criterion for evaluating and quantifying a surgeon's skill level. In some embodiments, the baseline data can be obtained prior to starting a training process by measuring the gaze point movement of a surgeon with experience during standard operations. In some embodiments, the baseline data may be obtained during a simulation exercise, which may include camera targeting, needle driving, manipulation exercises, suturing, cauterization / energy application, and / or any other exercises. In some embodiments, the baseline data may include information regarding the time ratio (T) and / or displacement ratio (D) of the time and / or displacement of the gaze point moving back and forth between the target surgical area and the surgical instrument, and the time and / or displacement of the gaze point moving from the initial position to the target surgical area. In some embodiments, the baseline data may also include the time of fixation of the gaze point to each tracking point. In some embodiments, the baseline data can be incorporated in combination with system kinematic and event data. In some embodiments, there may be different values of the time ratio (T) and / or displacement ratio (D) representing various levels of skill. In some embodiments, the baseline data can be saved to a computer-readable medium for future repeated use.

[0080] Method 400 proceeds to optional step 404 by identifying target P on 3D image display 450 shown in FIG. 5. A first signal indicating the start of the identification process can be output by a surgeon in training by pressing a button to select the identification mode, or by lightly tapping the foot pedal of the surgical console. As shown in FIG. 5, after target area P has been identified as the area requiring surgery, target P can be identified on 3D image display 450 by focusing the surgeon's line of sight on target P of image display 450. In some embodiments, target P can also be identified and marked by pressing a button on surgical console 120, or by lightly tapping foot pedal 128. In some embodiments, target P can be in a predetermined position or area within display 450, or can even be established in situ by the actual line-of-sight position(s) accessed by the surgeon.

[0081] After receiving the identification, control unit of endoscope 112 can optionally receive position data of the line of sight (e.g., target P) from processor 206 so that, by receiving other instructions such as pressing a button and / or lightly tapping the foot pedal, endoscope 112 can be adjusted to have target P located at the center of image display 450. In some embodiments, target P can be automatically assigned coordinates of (0, 0, 0) as the origin of the current 3D coordinate system.

[0082] Method 400 proceeds to optional step 406 by verifying the initial position of the tip of surgical instrument 110a, 110b, or 110c on 3D image display 450. For example, as shown in FIG. 5, the initial position of the tip of the surgical instrument can be identified at position Q0. Position Q0 can then be verified on 3D image display 450 by focusing the surgeon's line of sight on position Q0 on 3D image display 450. After vision tracking unit 200 receives the verification, position Q0 can be automatically assigned 3D coordinates of (q x0 , q y0 , q z0 ) with respect to the corresponding 3D coordinate system defined in step 404. In some embodiments, position Q0 can also be verified by pressing a button on surgeon console 120 or by lightly tapping foot pedal 128. In other embodiments, the initial position of the tip of the surgical instrument may be established by any other method, such as system kinematics, position sensors, and / or optional tool tracking.

[0083] Method 400 proceeds to step 408, for example, by measuring the movement of the surgeon's line of sight with respect to the instrument tip and target T. In other embodiments, upon completion of step 406, a signal indicating the start of the measurement process can be generated by any suitable method, such as the surgeon pressing a button on surgeon console 120 or lightly tapping a foot pedal. In other embodiments, the measurement process can be ongoing or can be initiated by a particular state of the instrument. In some embodiments, the surgeon's line of sight can start from target area P on 3D image display 450. In some embodiments, the surgeon's line of sight can start from the initial position Q0 of the surgical instrument on 3D image display 450.

[0084] Referring to FIG. 5, in some embodiments, a surgeon's skill level can be reflected by the surgeon's gaze point movement. For example, during surgery, an inexperienced surgeon may move his or her gaze point multiple times back and forth between the surgical instrument (e.g., Q0, Q1, Q2...) and the target point (e.g., P) to confirm when he or she moves the surgical instrument from the initial position to the target point or performs an operation from the initial position to the target point. In contrast, an experienced surgeon may have a smoother and less frequent gaze point movement between the initial gaze point (e.g., Q0) and the target point (e.g., P) when he or she moves the surgical instrument or performs an operation. Thus, in one embodiment, the displacement and / or time of the surgeon's gaze point movement can be used as one or more factors for evaluating the surgeon's skill level.

[0085] In step 408, the position data of the surgeon's gaze point is captured in real time by the left and right eye trackers 204a and 204b as described above in connection with the method shown in FIG. 3A, processed by the processor 206 to receive the 3D position data of the gaze point, and can be saved in the memory. The position data of the surgeon's gaze point is (q x0 , q y0 , q z0 ) when the surgeon's gaze point moves from the initial position Q0 to Q1, Q2,... Qn on the 3D image display 450, (q x1 , q y1 , q z1 ) when the surgeon's gaze point moves from the initial position Q0 to Q1, Q2,... Qn on the 3D image display 450, (q x2 , q y2 , q z2 ) when the surgeon's gaze point moves from the initial position Q0 to Q1, Q2,... Qn on the 3D image display 450... (q xn , q yn , q zn ) when the surgeon's gaze point moves from the initial position Q0 to Q1, Q2,... Qn on the 3D image display 450, and may include the 3D coordinate values.

[0086] In step 408, the time of the surgeon's gaze point movement from one point to the next is also In real time, it can be captured by the left and right eye trackers 204a and 204b, processed by the processor 206, and saved in the memory. In some embodiments, the time of the surgeon's fixation point movement may include information related to the movement direction. In some embodiments, at step 408, the fixation time at each fixation point may also be tracked.

[0087] Method 400 proceeds to step 410 by determining an evaluation factor. The evaluation factor may include a displacement ratio factor D. The displacement ratio factor D can be used to quantify the surgeon's skill level. For example, the displacement ratio factor D is D = (|Q0P| + |PQ1| + |Q1P| + |PQ2| + |Q2P| +... |PQ n | + |Q n P|) / |Q0P|, where |AB| represents the displacement of the fixation point A towards the fixation point B using the coordinate values of points A and B on the 3D image display 450. For example, |PQ1| represents the displacement from the fixation point P(0,0,0) to the fixation point (q x1 ,q y1 ,q z1 ), and |Q1P| represents the displacement from the fixation point Q1(q x1 ,q y1 ,q z1 ) to the fixation point P(0,0,0).

[0088] In some embodiments, the evaluation factor may include a time ratio factor T. The time ratio factor T can also be used to quantify the surgeon's skill level. For example, the time ratio factor T can be calculated as T = (t1 + t2 +... + t n ) / t i , where t1, t2,..., t n represent the time of the surgeon's fixation point required to move from one point to the next until the measurement process ends. During the measurement process, both the displacement (e.g., |PQ1|) and the time (e.g., t1) are captured between any two moving points, and the speed (v i) can also be calculated. The average speed (v) can further be calculated at the completion of the measurement process. The average speed (v) is then used to calculate the time t i for the current user to move his or her line of sight from an initial point (e.g., Q0) to a target point (e.g., P).

[0089] In some embodiments, the evaluation factors may include factors related to a fixed time measured at each line of sight. For example, the average fixation time of a surgeon's line of sight can be calculated after measuring a plurality of fixation times at a plurality of lines of sight in step 408. The average fixation time of the line of sight can then be compared with the line of sight fixation information stored in the baseline data to evaluate the surgeon's skill level.

[0090] Method 400 proceeds to step 412 by determining the surgeon's skill level by comparing evaluation factors (e.g., displacement ratio factor D, time ratio factor T, and / or line of sight fixation time) against baseline data (e.g., obtained in step 402). In some embodiments, the surgeon's skill level can also be determined using evaluation factors combined with system kinematic data and / or event data, such as data measured from the movement of the surgeon's hand, instrument, and / or camera. The movement of the surgeon's hand, instrument, and / or camera can be tracked and analyzed using appropriate methods.

[0091] In some embodiments, as described above, the 3D fixation point position of the surgeon captured by the fixation tracking unit 200 can be used to monitor the surgeon's ability and / or condition and to provide a warning that a decline in that ability and / or condition should be indicated. For example, the 3D fixation point can be used to determine the surgeon's stress or fatigue level, which, in some embodiments, can be reflected in changes in the surgeon's eye dynamics. Eye dynamics can include pupil diameter fluctuations and / or eye saccades. In some examples, prior to the measurement process (e.g., step 402 of method 400 in FIG. 4A), a baseline including information on the surgeon's eye dynamics when the surgeon is working in a normal state can first be obtained. For example, the baseline can include the frequency of the surgeon's eye saccades and / or the frequency and magnitude of the surgeon's pupil diameter fluctuations. During the measurement process (e.g., step 408 of method 400), the eye dynamics can be monitored using the left and right eye trackers 204a and 204b and compared to the baseline obtained under normal working conditions. When the eye dynamics appear abnormal compared to the normal state (e.g., step 412 of method 400), a notification or alarm can be given to prevent the surgeon from proceeding with any action. Abnormal eye dynamics can be indicated by various eye characteristics, including but not limited to sudden pupil diameter fluctuations or more frequent eye saccades than normal. In some embodiments, the surgeon's eye dynamics can also be monitored on a touch screen disposed on the teleoperation assembly 100, and a notification or alarm informing of an abnormal state of the surgeon's eye dynamics can be sent to the operating room (OR) staff so that the OR staff can react in a timely manner.

[0092] In some embodiments, in addition to monitoring the surgeon's stress level during a medical procedure, the tracked eye dynamics information can also be collected and analyzed after the procedure. In some examples, the analysis results can be used to understand why the surgeon is having difficulty with a particular training exercise or a particular part of a surgery.

[0093] In some embodiments, the 3D gaze point of the surgeon being tracked can be used for system login / logout and user identification. For example, in one embodiment, the surgeon is required to focus on a target point before logging into the system. The target point may include a surgical instrument whose 3D position data is stored in a computer-readable medium. After the position of the surgeon's 3D gaze point is tracked by the visual target tracking system using the method 300 described above, the position of the surgeon's 3D gaze point is compared with the 3D position of the target point. When the surgeon's 3D gaze point coincides with the 3D position target point, the surgeon can be automatically logged into the telemedical system 10. In some embodiments, if the surgeon's 3D gaze point does not match the 3D position of the target point, the surgeon cannot log into the telemedical system 10. In some embodiments, the telemedical system 10 is locked down or stopped from operating when the surgeon's eyes (and / or the surgeon's gaze point) cannot be detected.

[0094] In some embodiments, stored eye information of the user, such as iris characteristics, eye dynamics, or saccade velocity, can be used for user recognition and system login. For example, before starting the surgery, the surgeon may be required to perform some saccade exercises. The visual target tracking unit 200 can capture the eye dynamics of the surgeon and compare them with profile data associated with the corresponding eye dynamics. In some embodiments, the visual target tracking unit 200 can observe or measure various iris characteristics of the user and compare these iris characteristics with a database of stored iris profiles to enable iris recognition of different users. Once the surgeon is identified as a reoccurring user having a profile saved in the memory, the surgeon can be automatically logged into his or her own profile with customized settings.

[0095] In some embodiments, the 3D fixation point position of the surgeon captured by the fixation tracking unit 200 can be used to adjust the system characteristics of the teleoperated medical system 10 using the system control unit 214 shown in FIG. 2B. For example, the measured line of sight of the surgeon includes 3D position data of the surgical site (which can be a live patient, cadaver, animal, model, or partial / complete computer simulation). The 3D position data can be used to align the user interface (e.g., the control interfaces 122a and 122b of FIG. 1B) to an appropriate position (e.g., appropriate depth) with respect to the surgical site on the 3D image display so that the surgeon can view the surgical site on the 3D image display during surgery and intuitively control the surgical instrument. In some embodiments, the measured line of sight of the surgeon can be used to adjust the ergonomic settings of the surgeon console, such as the position and orientation of the image display relative to the user's head, so that the user can view the entire image display without obstruction.

[0096] In some embodiments, to more accurately calibrate the 3D position in 3D space, the fixation point of the surgeon can be focused on the surgical instrument on the image display, and the fixation tracking unit 200 can capture the 3D fixation point of the surgeon and provide accurate 3D position information of the surgical instrument based on the fixation point data received by the fixation tracking unit 200.

[0097] In an exemplary embodiment, a first teleoperated medical system includes a first visual target tracking unit having one or more first image displays, one or more first eye trackers, and a first processor coupled to the one or more first eye trackers and configured to calculate a first gaze point of a first user when the first user is viewing a first image displayed by the one or more first image displays. The system also includes a second visual target tracking unit having one or more second image displays, one or more second eye trackers, and a second processor coupled to the one or more second eye trackers and configured to calculate a second gaze point of a second user when the second user is viewing a second image displayed by the one or more second image displays. The one or more first image displays are coupled to the second processor, and the one or more second image displays are coupled to the first processor.

[0098] In other embodiments of the first teleoperated medical system, the first image and the second image include images of a surgical site.

[0099] In other embodiments of the first teleoperated medical system, the first image further includes a representation of the second gaze point.

[0100] In other embodiments of the first teleoperated medical system, the second image further includes a representation of the first gaze point.

[0101] In other embodiments of the first teleoperated medical system, the first image further includes a representation of the first gaze point.

[0102] In other embodiments of the first teleoperated medical system, the one or more first eye trackers include a left eye tracker and a right eye tracker.

[0103] In other embodiments of the first teleoperated medical system, the one or more second eye trackers include a left eye tracker and a right eye tracker.

[0104] In an exemplary embodiment, a first method of operating a teleoperated medical system includes tracking the movement of a 3D fixation point on a 3D image display of a surgical site and determining an evaluation factor from the movement of the 3D fixation point.

[0105] In another embodiment of the first method of operating a teleoperated medical system, the method further includes identifying a target spot and an initial spot on the 3D image display, and the movement is between the target spot and the initial spot.

[0106] In another embodiment of the first method of operating a teleoperated medical system, the step of identifying the target spot and the initial spot includes obtaining coordinate values of the target spot and the initial spot.

[0107] In another embodiment of the first method of operating a teleoperated medical system, the obtained coordinate values of the target spot are used to re-center the endoscope.

[0108] In another embodiment of the first method of operating a teleoperated medical system, the step of identifying the target spot and the initial spot includes pressing a button or lightly tapping a foot pedal.

[0109] In another embodiment of the first method of operating a teleoperated medical system, the step of identifying the target spot includes focusing on the target spot on the 3D image display.

[0110] In another embodiment of the first method of operating a teleoperated medical system, the step of identifying the initial spot includes focusing on the initial spot on the 3D image display.

[0111] In another embodiment, the first method of operating the teleoperated medical system includes determining a skill level by comparing baseline data regarding the movement of the point of regard with an evaluation factor.

[0112] In another embodiment of the first method of operating the teleoperated medical system, the movement of the 3D point of regard between the target spot and the initial spot includes a plurality of segmental movements of the 3D point of regard, each segmental movement being from the instrument spot to the target spot, and the instrument spot being between the initial spot and the target spot.

[0113] In another embodiment of the first method of operating the teleoperated medical system, the step of measuring the movement of the 3D point of regard includes measuring the 3D coordinate values of the instrument spot.

[0114] In another embodiment of the first method of operating the teleoperated medical system, the evaluation factor includes a displacement ratio factor, which is the ratio between the total displacement of the segmental movement of the 3D point of regard divided by the displacement between the target spot and the initial spot.

[0115] In another embodiment of the first method of operating the teleoperated medical system, the step of measuring the movement of the 3D point of regard includes measuring the time required for each segmental movement of the 3D point of regard between the target spot and the instrument spot.

[0116] In another embodiment of the first method of operating the teleoperated medical system, the evaluation factor includes a time ratio factor, which is the ratio between the total time required for the segmental movement divided by the time required for the 3D point of regard to move from the initial spot to the target spot calculated using the average 3D point of regard movement speed.

[0117] In another embodiment of the first method of operating a teleoperated medical system, the average 3D fixation point movement speed is the average value of all 3D fixation point movement speeds, and each of the 3D fixation point movement speeds is calculated using the displacement and time of each segment movement.

[0118] In another embodiment of the first method of operating a teleoperated medical system, the step of measuring the movement of the 3D fixation point includes the step of measuring the fixation time of the 3D fixation point at the instrument spot between the initial spot and the target spot.

[0119] In another embodiment of the first method of operating a teleoperated medical system, the evaluation factor includes the average fixation time calculated by using a plurality of fixation times measured at a plurality of instrument spots between the initial spot and the target spot.

[0120] In an exemplary embodiment, there is a second method of operating a teleoperated medical system, the method including the steps of tracking the gaze dynamics in a 3D image display of the surgical site and determining the state of the user when the user is looking at the 3D image display.

[0121] In another embodiment of the second method of operating a teleoperated medical system, the step of tracking the gaze dynamics includes the step of tracking at least one of the pupil diameter variation or saccades of the user using one or more eye trackers.

[0122] In another embodiment of the second method of operating a teleoperated medical system, the state of the user includes at least one of the stress level or fatigue level of the user.

[0123] In another embodiment of the second method of operating a teleoperated medical system, the step of determining the stress level includes the step of comparing the gaze dynamics with a baseline regarding the gaze dynamics.

[0124] In another embodiment of the second method of operating a teleoperated medical system, the baseline data includes the gaze dynamics measured when the surgeon is working in a normal state.

[0125] In another embodiment of the second method of operating a teleoperated medical system, the gaze dynamics are used for user recognition during the system login process.

[0126] In an exemplary method, the first method of operating a surgical system includes determining a 3D fixation point for a first user looking at a 3D image on a first display, displaying the 3D fixation point on a second display, and receiving an instruction from a second user looking at the 3D fixation point of the first user on the second display.

[0127] In another embodiment of the first method of operating a surgical system, the first user is being trained by the second user.

[0128] In another embodiment of the first method of operating a surgical system, the 3D fixation point of the first user is determined by a visual target tracking unit.

[0129] In another embodiment of the first method of operating a surgical system, the instruction from the second user is transmitted by using a telescriptor on the second display.

[0130] In an exemplary method, the second method of operating a surgical system, including an instrument and a 3D display, includes displaying a 3D image on the 3D display, determining the position of a 3D fixation point for a user looking at the 3D image, and comparing the 3D image with the position of the 3D fixation point.

[0131] In another embodiment of the second method of operating a surgical system, the position of the 3D fixation point is determined by using a visual target tracking unit.

[0132] In another embodiment of the second method of operating the surgical system, the 3D image includes a 3D image of the instrument displayed on one or more image displays of the visual target tracking unit.

[0133] In another embodiment of the second method of operating the surgical system, when the position of the 3D fixation point is within a predetermined region around the 3D image, the method includes the step of activating the instrument.

[0134] In another embodiment of the second method of operating the surgical system, the step of activating the instrument includes the step of activating the control interface to transmit to the instrument that is to be used by the user.

[0135] In another embodiment of the second method of operating the surgical system, the step of activating the instrument includes the step of controlling the instrument control unit to enable energy emission of the instrument.

[0136] In another embodiment, the second method of operating the surgical system includes the step of deactivating the instrument when the position of the 3D fixation point is outside a predetermined region around the 3D image.

[0137] In another embodiment, the second method of operating the surgical system includes the step of enabling automatic login to the surgical system when the position of the 3D fixation point is within a predetermined region around the 3D image.

[0138] In another embodiment, the second method of operating the surgical system includes the step of adjusting system characteristics.

[0139] In another embodiment of the second method of operating the teleoperation medical system, the 3D image includes a 3D image of the surgical site.

[0140] In another embodiment of the second method of operating the surgical system, the step of adjusting system characteristics includes the step of aligning the user interface with respect to the surgical site displayed on the 3D display.

[0141] In various embodiments, the visual target tracking unit 200, the control unit 210, and the imaging system (e.g., the imaging device 112) cooperate to operate so as to provide the user with a primary image and, optionally, a variety of secondarily adjusted images. The adjusted images can include, by way of non-limiting examples, enlarged images, brightened images, sharpened images, colored images, labeled images, and / or images including different wavelength ranges (e.g., the near-infrared range as contrasted with the visible light range). The prepared images of the surgical area can be at least partially controlled by the surgeon's line of sight. To provide the primary and secondary images, the imaging system can use a single imaging module and post-imaging operations (e.g., digital operations) for providing the adjusted images. Alternatively or additionally, as described below in various embodiments, the imaging system includes a plurality of imaging modules. The plurality of imaging modules can be provided on a common instrument deployed through a single patient aperture or on a plurality of instruments deployed through a plurality of patient apertures. In some embodiments, the imaging system can include a rigid endoscope (i.e., having a rigid shaft), with respect to which the pose (e.g., position, orientation, roll) is determined by the movement of a particular camera or the instrument arm to which it is attached. In other embodiments, the imaging device 112 can additionally or alternatively include integrated articulation capabilities.

[0142] FIG. 6A shows an exemplary imaging system 112a that can be used as the imaging device 112 of the remote operation assembly 100 of FIG. 1B. The imaging system 112a includes two imaging modules 714, 715 as components of a single imaging device. The imaging modules 714, 715 can be controlled independently of each other to create a primary image and a secondary image, such as an enlarged image overlay, of regions of interest that indicate different information or views of interest to the user. The imaging system 112a further includes an instrument interface 150d (e.g., coupled to the camera arm 108 shown in FIG. 1B), a shaft 152d connected to the interface 150d, a group 710 and 711 of cables, rods, and / or optical fibers passing through the shaft 152d, a list portion (e.g., a joint motion section) 712 and 713 connected to the shaft 152d, and the imaging modules 714 and 715 at the distal end of the imaging device 112a.

[0143] In some embodiments, the instrument interface 150d can receive commands from the control unit 210 shown in FIG. 2C. Each or both of the imaging modules 714, 715 can include optics and mechanisms for illuminating the surgical area. In some embodiments, the imaging module 714 and / or the imaging module 715 can also include a device capable of capturing an image (e.g., a stereoscopic image) of the surgical area. In some embodiments, the imaging device has a stereoscopic camera.

[0144] An image of the surgical area of interest can be captured by the auxiliary imaging module 715, and the adjusted image can be displayed on a display (the image displays (202a, 202b, the fixation point display 207, and / or the display 416 shown in FIG. 4) shown in FIG. 2C).

[0145] As shown in FIG. 6A, the shaft 152d of the imaging device 112a is configured to hold cables, rods, and / or optical fibers 710 and 711. In some embodiments, a first group 710 of cables, rods, and / or optical fibers may be coupled to the imaging module 714 through the list portion 712, and a second group 711 of cables, rods, and / or optical fibers may be coupled to the imaging module 715 through the list portion 713. Such a configuration provides independent control and operation between the imaging modules 714 and 715 by providing different instructions via two different sets 710, 711 of cables, rods, and / or optical fibers and by controlling the list portions 712, 713 with different motion controls. The list portions 712, 713 are connected to the shaft 152d at the distal end of the shaft 152d and are respectively coupled to the imaging modules 714, 715. The list portion 712 enables the movement of the imaging module 714 in at least two degrees of freedom and can be controlled by a first set of cables or rods 710 passing through the shaft 152d. Similarly, the list portion 713 enables the movement of the imaging module 715 in at least two degrees of freedom and can be controlled by a second set of cables or rods 711 passing through the shaft 152d. The optical fibers 710, 711 may be optically coupled to the imaging modules 714, 715 for both providing illumination and transmitting images.

[0146] In some embodiments, the instrument interface 150d couples the actuating motors of the camera arm 108 to shafts 152d within cables and rods 710 and 711, respectively. In some embodiments, the instrument interface 150d may include mechanisms that can be driven by the actuating motors of the camera arm 108 to control list portions 712 and 713 to operate imaging modules 714 and 715, respectively. For example, in one embodiment, when imaging module 714 is used as a primary imaging module and imaging module 715 is used as an auxiliary or secondary imaging module, the instrument interface 1520d may send instructions to the actuating motors such that the auxiliary imaging module 715 can be driven by the actuating motors to move the imaging module 715 to the surgeon's line of sight (e.g., based on a detected point of regard for image displays 202a, 202b as described above in connection with FIGS. 2B and 2C).

[0147] Imaging module 714 and / or imaging module 715 can adjust the type of image obtained by the imaging device. For example, in one embodiment, imaging modules 714, 715 can "zoom" in or "zoom" out on the surgical area to obtain more detailed and less detailed images of the surgical area. In some embodiments, imaging modules 714, 715 can change their position along the longitudinal axis LA of shaft 152d relative to distal end 716, thereby providing a physical, distance-based zoom function. Imaging module 714 and / or imaging module 715 can also be reoriented in multiple dimensions (e.g., within a spherical space) by operating respective list portions 712, 713. Software operating in the teleoperated medical system 10 can control the zoom function by controlling the physical position and orientation of list portions 712, 713 and / or imaging modules 714, 715, or by controlling the digital manipulation of the images obtained by imaging modules 714, 715 (e.g., through digital zooming or image adjustment). Additionally or alternatively, the zoom function can be controlled by imaging module 714 and / or imaging module 715 itself.

[0148] FIG. 6B shows an imaging system 112b that includes imaging devices 112c and 112d. In one embodiment, imaging devices 112c, 112d can be used with two separate camera arms 108 of the remote operation assembly 100 shown in FIG. 1B, and each imaging device 112c, 112d is independently attached to camera arms 108a, 108b, respectively. In the depicted embodiment, imaging devices 112c, 112d each include independent imaging modules 720, 722, respectively. For example, in the depicted embodiment, imaging module 720 is attached to the distal end of imaging device 112d, and imaging module 722 is attached to the distal end of imaging device 112c. Imaging devices 112c, 112d can be aligned close to each other within the surgical field to assist in aligning both images obtained independently from imaging modules 720, 722 to a single display (i.e., display 416 shown in FIG. 4). Imaging devices 112c, 112d are substantially similar to imaging module 112a described above in connection with FIG. 6A, except for the differences described herein. For example, imaging device 112d includes an instrument interface 150e, a shaft 152e connected to interface 150e, a wrist portion 724 connected to shaft 152e, an imaging module 720 connected to wrist portion 724, and cables, rods, and / or optical fibers 726 passing through shaft 152e. Similarly, imaging device 112c includes an instrument interface 150f, a shaft 152f connected to interface 150f, a wrist portion 727 connected to shaft 152f, an imaging module 722 connected to wrist portion 727, and cables, rods, and / or optical fibers 728 passing through shaft 152f. The functions of the components of imaging device 112c and / or imaging device 112d are substantially similar to the corresponding components of imaging device 112a described above.

[0149] FIG. 6C is a schematic diagram of a stereo camera 730 according to one embodiment of the present disclosure. The stereo camera 730 can be used as any of the imaging modules described above, including the imaging modules 714, 715 of FIG. 6A and the imaging modules 720, 722 of FIG. 6B. The stereo camera 730 can be used to capture a stereo image that can be displayed and perceived by a human left and right eye so as to form a 3D image.

[0150] The imaging module (e.g., imaging modules 714, 715, 720, 722) can be independently moved within the 3D space to provide different images of the surgical area. For example, the imaging module 714 of the imaging device 112a and / or the imaging module 720 of the imaging device 112b can be used to provide a primary image of the surgical area. The imaging module 715 of the imaging device 112a and / or the imaging module 722 of the imaging device 112c can be used to provide a secondary or adjusted image, such as an enlarged image of the surgical area corresponding to the current viewing point of the surgeon, shown as image 960 in FIG. 8C. In some examples, the imaging module 715 and the endoscopic effector 732 can also be used to provide other imaging modalities, such as fluorescence or ultrasound imaging, which can be used to identify and / or highlight structures within the enlarged image area. In some embodiments, as described in more detail below, the imaging module 715 of the imaging device 112a and / or the imaging module 722 of the imaging device 112c can be used to capture images of different regions arranged in a non-overlapping manner when the viewing point of the surgeon is scanned across the region (e.g., the patient's abdominal cavity) to identify the region of interest (e.g., the lesion). The image corresponding to the region of interest can be further enlarged or processed in other imaging modalities (e.g., by optical manipulation and / or digital processing).

[0151] A teleoperated medical system 10 that includes a visual target tracking unit 200 and one or more imaging devices 112 can be used to apply various image modifications to a surgeon's gaze region and non-gaze region. For example, the image modifications can include any of a variety of imaging effects, including, but not limited to, magnification, zoom in or out, highlighting, recoloring, decolorizing, labeling, brightening, blurring, and sharpening. In some embodiments, the image modifications can emphasize or vary the gaze region while suppressing the emphasis of the non-gaze region. For example, in some embodiments, the image modifications can include blurring the focus of the non-gaze region, decolorizing the non-gaze region, or blurring the non-gaze region to emphasize the gaze region by contrast. In some embodiments, the image modifications include magnifying the surgeon's gaze region in response to the surgeon's instructions. In some embodiments, the image modifications include overlaying the magnified gaze region of the surgeon over the primary view of the surgical site. In some embodiments, the image modifications include applying various image modalities to the surgeon's gaze region. In some embodiments, the teleoperated medical system 10 requires additional user input (in addition to user gaze) to initiate the image modifications. In some embodiments, the teleoperated medical system and the visual target tracking unit 200 interpret the user's gaze to determine which imaging device acquires the primary image and which imaging device acquires or creates auxiliary or secondary images.

[0152] FIG. 7A shows a block diagram of an image processing system 800 that can be coupled to any of the imaging devices 112a, 112b, 112c, 112d shown in FIGS. 6A-6B to achieve a desired image modification. The image processing system 800 includes a control unit 802 that can be the same as the control unit 210 shown in FIG. 2C. The control unit 802 can include one or more processors for processing input data and issuing commands. The processor can be coupled to a memory (volatile, non-volatile, or a combination) for holding data and program instructions. The control unit 802 can receive an indication signal from the gaze processor 206 shown in FIG. 2C. The control unit 802 can also receive an indication of other forms of input, such as an input signal from the control interface 122 shown in FIG. 1C. The program instructions can include instructions for converting, for example, a data signal received from the gaze processor 206 into a command signal indicative of a requested modification of the original image generated by the imaging device 112. Using these signals, the control unit 802 can determine the type and / or extent of the image change intended by the surgeon. For example, the control unit 802 can determine whether the surgeon is requesting a zoom or magnify function or whether the surgeon is requesting a different imaging modality display. In some embodiments, the control unit 802 can receive an indication to apply different image modalities to a region corresponding to the surgeon's point of regard (e.g., the gaze region) and the remainder of the image field (i.e., the non-gaze region).

[0153] After receiving a command, the control unit 802 may send a signal to a modifier calculation unit 804. In some embodiments, the modifier calculation unit may also be known as an endoscopic operation calculation unit. In some embodiments, the modifier calculation unit 804 may be included in the instrument interface 150 (e.g., 150d, 150e, 150f) of the imaging device 112 (e.g., 112a, 112b, 112c, 112d). The modifier calculation unit 804 may include a processor for converting the signal received from the control unit 802 into a command or an operation signal. In some embodiments, the command signal is directly processed by the endoscopic processor to digitally modify the image as indicated by the command signal. Additionally or alternatively, the command signal may be sent to an actuator 806 that can affect the movement of the imaging device 112. In some embodiments, the intended image modification by the surgeon (e.g., magnification, zoom in, or zoom out) may be realized when the actuator 806 causes the appropriate movement of the imaging module. The modifier calculation unit 804 may include one or more processors coupled to a memory (volatile, non-volatile, or a combination) that holds data and programming (e.g., fixation point data, motion-related command data, fixation point tracking algorithm, image modification algorithm). In some embodiments, the above-described functions of the control unit 802, the modifier calculation unit 804, and / or the endoscopic processor 808 may be performed by a single processor.

[0154] Actuator 806 can be mechanically coupled to the instrument interface 150 of the imaging device 112. For example, actuator 806 can be mechanically coupled to the instrument interface 150d of imaging device 112a as shown in FIG. 6A. Actuator 806 can be, for example, a rotary motor housed in the camera arm 108 (e.g., camera arms 108, 108a, and / or 108b) of the remote operation assembly 100, and a corresponding imaging device 112 is attached onto this camera arm. As described above, in some embodiments, the modifier calculation unit 804 of the instrument interface 150 can convert the mechanical input of actuator 806 into the movement of the corresponding list unit and the imaging module of the imaging device.

[0155] The control unit 802 can also control the light output of the illumination unit 810. The illumination unit 810 can provide light through one or more optical fibers, such as optical fibers 710 and / or 711 of the imaging device 112a in FIG. 6A, optical fiber 726 of the imaging device 112d in FIG. 6B, and optical fiber 728 of the imaging device 112c, to illuminate the surgical area.

[0156] The image processing system 800 can include a plurality of endoscope processors 808 configured to perform image capture, image processing, and / or image display functions for corresponding imaging modules, such as imaging modules 714, 715, 720, or 722. The endoscope processor 808 can be disposed in the vision cart 140, for example, as part of the central electronic data processing unit 142. The endoscope processor 808 can also be disposed in the remote operation assembly 100 and / or the surgeon console 120. The endoscope processor 808 can be coupled to a memory (volatile, non-volatile, or a combination) that holds data and programming.

[0157] The images processed by the endoscope processor 808 can be output on the display 816 shown in FIG. 4A. In some embodiments, the display 816 may include the image displays 202a, 202b, and / or the fixation point display 207 of the visual target tracking unit 200 shown in FIG. 2C. In some embodiments, the display 816 is the touch screen monitor 146 attached to the vision cart 140 shown in FIG. 1D. In some embodiments, the display 816 can show a secondary or modified image that overlays the primary image, for example, as displayed on the displays 202a and 202b in the surgeon console 120. In some embodiments, the display 816 can show a plurality of images (e.g., the primary image and the modified image) arranged side by side in an overlaid or non-overlaid manner.

[0158] The imaging module, for example, imaging modules 714, 715, 720, or 722, may have a stereoscopic camera that can capture a stereoscopic image of the surgical area. The stereoscopic image of the surgical area can be sent to the endoscope processor 808 by an optical fiber, such as optical fibers 710, 711, 726, or 728. The endoscope processor 808 can digitize the stereoscopic image captured by the imaging module and provide the stereoscopic image to the display 816. The stereoscopic image displayed on the display 816 can be perceived by the surgeon as a 3D stereoscopic image through the visual target tracking unit 200 (shown in FIG. 2C). The surgeon's instructions can be recognized and initiated using the surgeon's gaze data captured by the eye trackers 204a, 204b. The gaze processor 206 can process these instructions and transmit them to the control unit 802. The eye trackers 204a, 204b and the gaze processor 206 can capture the surgeon's 3D gaze and generate an image of the surgeon's 3D gaze, as detailed in connection with FIGS. 2A-2C.

[0159] With respect to FIG. 6A, in some embodiments, when the imaging device 112a is used in the remote operation medical system 10, the two imaging modules 714, 715 may share the same instrument interface 150d. As described above, in some embodiments, the imaging module 714 is used to capture a primary image of the surgical area, and the imaging module 715 is used to capture a secondary (e.g., modified) image based on the surgeon's instructions. While the imaging module 714 is capturing the primary image of the surgical site, the line-of-sight processor 206 can interpret the surgeon's instructions and send the surgeon's instructions to the control unit 802 in order to instruct the imaging module 715 to capture and display a prepared image (e.g., as a non-limiting example, an enlarged image, a sharpened image, a colorized image, a decolorized image) of the area in the surgical field corresponding to the surgeon's line of sight. The instructions can be processed by one or more processors of the control unit 802 and the corrector calculation unit 804.

[0160] In some embodiments, the resulting command signal is transmitted to the endoscope processor 808 to digitally affect (digitally magnify or otherwise modify) the images received by the imaging modules 714, 715. In some embodiments, the resulting command signal is transmitted to the endoscope processor 808 to change the imaging mode of the imaging module 715 (e.g., as a non-limiting example, change the wavelength range to be imaged, the optical magnification, or the width of the field of view). In some embodiments, the resulting command signal can be transmitted via the cable, rod or optical fiber 711 to control the movement of the imaging module 715, and the imaging module 715 can move or be adjusted based on the instructions of the surgeon. For example, when the surgeon desires to view an enlarged image of the area specified by his or her point of gaze, the imaging module 715 can move along the longitudinal axis LA of the shaft 152d to "zoom in" or obtain an enlarged image. In some embodiments, the imaging module 715 can be controlled to capture and display a primary image of the surgical area, and the imaging module 714 can capture and display a secondary image based on the instructions of the surgeon. The secondary image can include a visually adjusted portion of the primary image.

[0161] In some embodiments, as shown in FIG. 6B, imaging device 112b and imaging device 112c can be used in the remote operation medical system 10. As shown in FIG. 6B, imaging device 112b includes instrument interface 150d, and imaging device 112c includes instrument interface 150f. In some embodiments, there may be more than one control unit 802, and each control unit 802 is coupled to one of instrument interfaces 150d and 150f. In an alternative embodiment, there may be one control unit 802 that is coupled to both instrument interfaces 150d, 150f and can send independent instructions to instrument interfaces 150d, 150f. For example, in some embodiments, imaging device 112b may be used to capture a primary image of the surgical area based on instructions received by instrument interface 150d, and imaging device 112c may be used to capture a secondary or modified image based on instructions received by instrument interface 150f.

[0162] In one embodiment, while the imaging module 620 captures a primary image of the surgical site, the line-of-sight processor 206 interprets the surgeon's instructions and sends the surgeon's instructions to the control unit 802 to capture and display a prepared image (e.g., as a non-limiting example, a magnified image, a sharpened image, a colored image, a decolorized image) of the area of the surgical field corresponding to the surgeon's line of sight. The instructions can be processed by one or more processors of the control unit 802 and the modifier calculation unit 804.

[0163] In some embodiments, the resulting command signal is transmitted to the endoscope processor 808 to digitally affect (e.g., digitally magnify or otherwise modify) the images received by the imaging modules 720, 722. In some embodiments, the resulting command signal is transmitted to the endoscope processor 808 to change the imaging mode of the imaging module 722 (e.g., as a non-limiting example, change the wavelength range to be imaged, the optical magnification, or the width of the field of view). In some embodiments, the resulting command signal can be transmitted via a cable, rod, or fiber optic 728 to control the movement of the imaging module 722, and the imaging module 722 can move or be adjusted based on the surgeon's instructions. For example, when the surgeon desires to view a magnified image of the area identified by his or her point of gaze, the imaging module 722 can move along the longitudinal axis of the shaft 152f to obtain a "zoomed-in" or magnified image. In an alternative embodiment, the imaging module 722 can be controlled to capture and display a primary image of the surgical area, and the imaging module 720 can capture and display a secondary image based on the surgeon's instructions.

[0164] FIG. 7B is a flowchart showing a method 850 for controlling an imaging device(s) of the teleoperated medical system 10 to apply various imaging corrections according to an embodiment of the present disclosure. In process 852, a first imaging module captures a first or primary image of the surgical area. The first imaging module can be a stereoscopic camera configured to capture a stereoscopic image of the surgical area. The position within the surgical field on which the first imaging module focuses can be adjusted by someone other than the user, such as a person in the operating room next to the patient. Additionally or alternatively, the first imaging module can be actuated and controlled by the surgeon console 120 or other components of the teleoperated medical system 10 (as shown in FIG. 1C). In some embodiments, as described above in connection with FIGS. 2A-2C, the first imaging module can be controlled by the surgeon's line of sight. In particular, the surgeon's line of sight can be tracked and interpreted by the visual target tracking unit 200 shown in FIG. 2C so as to affect the first imaging module. In some embodiments, the first image is an overview image of the surgical area, such as the patient's abdomen, at zero or low magnification. In some embodiments, the first image includes a primary image of the surgical area (e.g., the surgical area where the surgeon is working) at zero or low magnification.

[0165] In process 854 of method 850, the first image is processed for display by an endoscopic processor 808 shown in FIG. 7A, which is coupled to a first imaging module. The endoscopic processor 808 receives data representing the first image and digitizes various characteristics of the image data, such as the position, boundaries, and / or magnification of the first image. The digitized information can be saved in one or more memories coupled to the endoscopic processor 808. The endoscopic processor 808 processes the imaging information of the first image and displays the first image on a display 816 shown in FIG. 7A. When the first imaging module is a stereoscopic camera, the first image is a stereoscopic image. The stereoscopic image can be displayed on the left and right eye image displays 202a, 202b of the surgeon console 120. A 3D primary image of the surgical area can then be viewed by the surgeon.

[0166] In process 856, the eye gaze tracking system 200 tracks the surgeon's point of gaze with respect to the display of the first image. For example, when the first image is displayed on the left and right eye image displays 202a, 202b, the eye gaze tracking system of FIG. 2C can determine the surgeon's point of gaze with respect to the first image. The gaze processor 206 can digitize the gaze information, for example, to provide the 3D position of the surgeon's point of gaze. The digitized information can be saved in one or more memories 208 (shown in FIG. 2C) coupled to the gaze processor 206. When the surgeon sends an instruction through the surgeon's gaze, the gaze processor 206 can provide the digitized information to the control unit 210. Further details of devices, systems, and methods for controlling a teleoperated medical system by eye gaze tracking can be found, for example, in U.S. Provisional Application No. 61 / 955334, filed on March 19, 2014, entitled "MEDICAL DEVICES, SYSTEMS, AND METHODS INTEGRATING EYE GAZE TRACKING FOR STEREO VIEWER", which is hereby incorporated by reference in its entirety.

[0167] In process 858, the teleoperated medical system 10 receives instructions from the surgeon to provide various image corrections. In some embodiments, the instructions from the surgeon include providing an enlarged image of the surgeon's line of sight area. In some embodiments, the instructions from the surgeon also include overlaying a second, modified image on the first image. The instructions from the surgeon may also include displaying the second image in any suitable imaging modality.

[0168] In process 860, the teleoperated medical system 10 captures a second image based on the support received in process 858 and the surgeon's point of gaze tracked in process 856. In some embodiments, the second image can be captured by the second imaging module using the gaze information received by the instruction and control unit 210. In some embodiments, the position and orientation of the second imaging module can be adjusted as appropriate by the image processing system 800 shown in FIG. 7A to capture the second image. In other embodiments, the desired second image can be obtained through digital processing without optical manipulation of the second imaging module. In some embodiments, both optical and digital manipulations are used to obtain the desired second image.

[0169] In process 862, the second image is displayed on the display 816 shown in FIG. 7A. For example, the second image can be an enlarged image of the surgeon's point of gaze or an image displayed in a different imaging modality requested by the surgeon.

[0170] FIGS. 8A, 9, and 10A are flowcharts showing various methods 900, 1000, and 1100 that illustrate different image corrections that can be achieved using the teleoperated medical system 10. It should be understood that these methods are merely illustrative in nature and are not intended to be limiting. Other image corrections are envisioned.

[0171] FIG. 8A is a flowchart showing a method 900 for displaying an enlarged image that overlaps a primary image of the remote operation medical system 10 of the present disclosure. In particular, the method 900 targets the capture and display of a primary image using a primary imaging module, and the capture and display of an enlarged image of a region corresponding to the surgeon's line of sight using an auxiliary imaging module. FIG. 8B shows a predetermined visual target tracking threshold region 935 displayed within the enlarged region according to some embodiments of the present disclosure. FIG. 8C shows an example of displaying an enlarged image 960 over a primary image 950. FIG. 8D is a schematic diagram showing the use of the imaging device 112a of FIG. 6A for capturing and generating the primary image 950 and the enlarged image 960 shown in FIG. 8C. FIG. 8E is a schematic diagram showing the use of the imaging devices 112b and 112c of FIG. 6B for capturing and generating the primary image 950 and the enlarged image 960 shown in FIG. 8C. FIGS. 8A-8E depict an auxiliary or secondary image as the enlarged image for simplicity, and it should be understood that other imaging effects (non-limiting examples include color / grayscale imaging, sharp / blurred imaging, and bright / dull imaging) may be envisioned.

[0172] The method 900 is described below with reference to FIGS. 8A and 8C-8E. In process 902, a primary image 950 of the surgical area is captured by the primary imaging module 714 of FIG. 8D or the primary imaging module 720 of FIG. 8E. In some embodiments, the primary imaging module may be the first imaging module discussed in process 952 of method 950 shown in FIG. 8B.

[0173] In process 904 of method 900, the captured primary image 950 is processed by an endoscope processor 908 that is coupled to the primary imaging module 714 or the primary imaging module 720, and the primary image is displayed on a display 816.

[0174] In process 906, the eye gaze tracking unit 200 of FIG. 2C tracks and interprets the surgeon's point of gaze. The gaze information may include the 3D position of the surgeon's point of gaze. The gaze information can be provided to the control unit 210 as discussed in relation to process 956 of FIG. 8B.

[0175] In some embodiments, the surgeon may wish to examine a region of interest within the surgical area in more detail. For example, the surgeon may wish to examine an enlarged view of the microstructure of the surgical area, such as nerves, blood vessels, and lesions. In some examples, the surgeon may also wish to initiate a microsurgery application on or overlapping the primary image 950. In process 908 of method 900, the teleoperation medical system 10 receives an instruction from the surgeon to view a virtually enlarged image of the region of interest. In some embodiments, the surgeon may use his or her point of gaze to identify the location of the region of interest and may enter a confirmation instruction using any of a variety of other input methods. For example, the surgeon may press a button on the surgeon console, lightly tap a foot pedal, send a voice message, or wink. The confirmation instruction can be sent to the control unit 210 of an auxiliary imaging module, such as imaging module 715 of FIG. 8D or imaging module 722 of FIG. 8E. In some embodiments, the confirmation instruction also includes more detailed information regarding the characteristics or extent of the selected image modification. For example, in some embodiments, the confirmation instruction includes information regarding the surgeon's preferred level of magnification. Additionally, the position information regarding the surgeon's point of gaze abstracted by the gaze processor 206 can also be sent to the control unit 210 coupled to the corresponding auxiliary imaging module.

[0176] As described above, FIG. 8B shows a predetermined visual target tracking threshold region 935 displayed within the virtual magnification region 930. The virtual magnification region 930 can be generated using digitized position information regarding the surgeon's line of sight (e.g., provided by the line of sight processor 206) and the desired level of magnification included in the surgeon's instructions (e.g., obtained in process 908). The predetermined visual target tracking threshold region reflects the characteristic eye movements of the current user of the teleoperated medical system 10. Generally, human eye movements can include frequent saccades, and eye movement behavior varies from person to person. Thus, the visual target tracking threshold region 935 includes a region within which the surgeon's line of sight frequently saccades and can correspond to the observed eye movement behavior of the surgeon. Since different surgeons can have different saccade frequencies, speeds, and amplitudes, the shape and size of the visual target tracking threshold region 935 can vary from surgeon to surgeon. The visual target tracking threshold region 935 for a particular surgeon can be determined during the visual target tracking calibration process.

[0177] The surgeon's predetermined visual target tracking threshold region 935 can reflect the range of accuracy and precision of the surgeon's line of sight measurements. In some embodiments, the visual target tracking threshold region 935 can be used to confirm the central position of the surgeon's fixation point and the consistency of repeated measurements at that point. The predetermined visual target tracking threshold region 935 can be arranged to be co-centered with the magnification region 930 to facilitate comparison between the visual target tracking threshold region 935 and the magnification region 930.

[0178] The circular shapes of the enlarged region 930 and the target tracking threshold region 935 are merely exemplary for the purposes of the description of the present disclosure, and it should be understood that the regions 930, 935 can be formed in any of a variety of suitable shapes including, by way of non-limiting example, rectangles, ellipses, or squares. In some embodiments, since a surgeon's predetermined target tracking threshold region 935 can change due to accumulated stress or fatigue, the surgeon may undergo another calibration session to update the target tracking threshold region 935. In some embodiments, a surgeon's eye movement behavior can be monitored in real time by the target tracking system 200 to alert the surgeon and / or other healthcare providers when the surgeon's abnormal eye behavior is detected. For example, if the target tracking system 200 detects abnormal and / or uncharacteristic eye movement behavior of the surgeon, a warning requesting a recalibration process can be sent to the surgeon by the system 200 (e.g., via the display 816) or the surgeon can be advised to stop the current work session.

[0179] In process 910 of method 900, the size of a predetermined foveal tracking threshold region 935 of a surgeon (e.g., the current user) is compared to the size of a virtual magnification region 930 (e.g., by the image processing system 800 shown in FIG. 7A) to avoid inadvertent jittering in the magnified view. This comparison can be performed by a processor of the gaze processor 206 and / or the control unit 210. If the magnification region 930 is smaller than the predetermined foveal tracking threshold region 935, the surgeon's current fixation point may not be within the desired range of accuracy and precision, and the magnified image 930 of the surgeon's current fixation point may be displayed with inadvertent jittering or saccades. In other words, when the surgeon's fixation point is located outside the predetermined foveal tracking threshold region 935, the surgical region shown by the magnified image 930 changes as the surgeon's gaze moves relative to the image display 816 shown in FIG. 7A (e.g., the image displays 202a, 202b shown in FIG. 2C). In some embodiments, an average fixation point (e.g., a time-averaged weighting of fixation points over a particular period or a particular number N of image frames) can be calculated by a processor of the gaze processor 206 and / or the control unit 210, and the average fixation point can be compared to the predetermined foveal tracking threshold region 935. For example, if the average fixation point (e.g., over the last N image frames) moves outside the predetermined foveal tracking threshold region 935, the position of the magnification region 930 moves relative to the image display 816 shown in FIG. 7A (e.g., the image displays 202a, 202b shown in FIG. 2C). Thus, the predetermined foveal tracking threshold region 935 enables natural saccades or movement of the user's gaze without inadvertent jittering of the position of the magnified view (e.g., without changing the magnified image 930 to track or reflect the natural saccades of the surgeon's gaze).

[0180] In some embodiments, if the surgeon's line of sight is located outside the predetermined target tracking threshold region 935, the surgeon's current line of sight may not need to be magnified, and method 900 returns to process 906 to continue tracking the surgeon's line of sight. In some embodiments, if the surgeon desires to change the primary image by changing the focal position of the primary imaging module, method 900 returns to process 902 to capture a different primary image.

[0181] However, if it is determined that the magnified region 930 reflecting the surgeon's line of sight is larger than the predetermined target tracking threshold region 935 (e.g., by the image processing system 800 shown in FIG. 7A), the surgeon's current line of sight (e.g., measured by the target tracking system 200) may be considered to be within the desired range of accuracy and precision of the current surgeon's line of sight measurement. In other words, if it is determined that the surgeon's line of sight is within the predetermined target tracking threshold region 935, the magnified image 930 remains stationary. The magnified image 930 of the surgeon's current line of sight can be displayed (e.g., as the magnified region 930) without reflecting accidental jittering or eye saccades.

[0182] In process 911, the processor of the line of sight processor 206 and / or the control unit 210 queries whether the magnified region 930 is within the primary image. In some examples, the surgeon's line of sight can move either intentionally or inadvertently outside or to the edge of the primary image captured by the primary imaging module 714 or the primary imaging module 720. Generally, it may be easier and more accurate to place and control the magnified vision of the user (e.g., the magnified region 930 of the region of interest) closer to the central region of the primary image than outside or to the edge of the primary image. Thus, in process 911, the virtual magnified region 930 of the surgeon's line of sight is further compared with the primary image 950 captured by the primary imaging module to confirm whether the surgeon's line of sight is within the primary image.

[0183] If the enlarged area 930 is outside or at the edge of the primary image 950, method 900 proceeds back to process 902 by adjusting the position and orientation of the primary imaging module and / or the auxiliary imaging module to obtain a new primary and enlarged image. In some examples, the primary imaging module may "follow" the movement or direction of the auxiliary imaging module within the surgical field. For example, the image processing system 800 shown in FIG. 7A can move the auxiliary imaging module 715 or 722 (shown in FIGS. 8D and 8E respectively) in coordination with the movement of the surgeon's line of sight, and can move the primary imaging module (shown in FIGS. 8D and 8E respectively) to "follow" the enlarged area 930, so as to maintain the enlarged area within the central portion of the primary image. In some embodiments, the digitized surgeon's line of sight information can be used by the control unit 200 of the corresponding primary imaging module to appropriately adjust the position of the corresponding primary imaging module.

[0184] When the remote operation medical system 10 (e.g., the image processing system 800 shown in FIG. 7A) determines that the virtual magnification region 930 of the surgeon's enlarged line of sight is within the primary image 950, the method 900 proceeds to process 912. In process 912, the control unit 210 receives and processes the surgeon's instructions based on the surgeon's line of sight and / or other inputs. For example, the control unit 210 receives the digitized position information of the surgeon's fixation point from the line of sight processor 206. The control unit 210 may also receive a preferred magnification input from the surgeon. The control unit 210 may transmit the position and magnification data to the instrument interface 150 (e.g., the instrument interface 150d or the instrument interface 150f connected to the auxiliary imaging module 715 or the auxiliary imaging module 722 respectively). As described above, in some embodiments, the instrument interface 150d or the instrument interface 150f may transmit the position and magnification information to the actuator 806 coupled to the auxiliary imaging module 715 or the auxiliary imaging module 722, and the actuator 806 can drive the auxiliary imaging module to move the corresponding imaging module to the position of the surgeon's fixation point as shown in FIGS. 8D-8E. The shaft 152d / list portion 713 or the shaft 152f / list portion 727, and the corresponding auxiliary imaging module 715 or imaging module 722 can each extend along the longitudinal axis of the shaft towards the surgeon's fixation point to a certain distance corresponding to the desired level of magnification, and the magnified image 960 (as shown in FIGS. 8D and 8E) can be captured by the corresponding auxiliary imaging module 715 or 722.

[0185] In process 914, an auxiliary imaging module (e.g., 715 or 722) captures a secondary or modified image of the region of interest indicated by the surgeon's line of sight after the actuator 806 adjusts the auxiliary imaging module in response to the surgeon's instructions. The auxiliary imaging module captures an enlarged image 960 of the region of interest indicated by the surgeon's line of sight. In some embodiments, the auxiliary imaging module includes a stereoscopic camera 730 shown in FIG. 6C that can capture stereoscopic images.

[0186] In process 916, an image processing system 800 (shown in FIG. 7A) processes and displays the enlarged image 960 of the surgeon's line of sight as an image superimposed on the primary image 950. In some embodiments, the enlarged image 960 may be captured by an optical magnification method (e.g., a physical adjustment of the auxiliary imaging module with respect to the region of interest indicated by the surgeon's line of sight). Additionally or alternatively, the enlarged image 960 may be created by the endoscope processor 808 (e.g., by digital manipulation). The endoscope processor 808 can digitize the enlarged image 960 captured by the auxiliary imaging module and then align the enlarged image 960 to overlay it on the primary image 950 of the display 816.

[0187] In some embodiments, a common feature-based method (e.g., a common landmark method) may be used to align the enlarged image 960 so as to overlay it on top of the primary image. For example, in some embodiments, the endoscope processor 808 can identify common features, such as the feature 955 of FIG. 8C, that are present in both the primary image 950 and the enlarged image 960. The endoscope processor 808 can then spatially align the common feature 955 of the enlarged image 960 with the common feature 955 of the primary image 950. Although a common feature-based method is described herein as a way to align the enlarged image 960 with the primary image 950, it should be understood that any suitable method, such as an intensity-based method or a feature-based method, can be used as a non-limiting example to align the enlarged image 960 so as to overlay it on top of the primary image 950.

[0188] In some embodiments, the display 816 can be the same as the image displays 202a and 202b. When the auxiliary imaging module includes the stereo camera shown in FIG. 6C, the stereo camera can capture a stereo image at the required magnification, and the endoscope processor 808 can generate left and right stereo images for display and for overlaying the stereo image of the primary image 950 on each of the image displays 202a and 202b. The surgeon can view the stereo image as a 3D image of the enlarged point of fixation. Additionally or alternatively, the display 816 can include an external display, such as the touch screen 146 shown in FIG. 1D.

[0189] In process 918, the teleoperation medical system 10 queries whether the magnified image is desired by the surgeon. During the procedure, the surgeon can turn the magnified image on and off, or "toggle" the magnified image on and off. After the surgeon views the magnified image 960, the surgeon can send an end instruction to turn off, hide, or close the magnified image 960. The surgeon may send the end instruction using any method, for example, the surgeon can press a button on the surgeon console, lightly tap a foot pedal, or send a voice message, or even wink.

[0190] Upon receiving an instruction from the surgeon to turn off, hide, or close the magnified image 960, method 900 proceeds to process 920. In process 920, the image processing system 800 turns off, hides, or closes the magnified image from the image display 816. When the magnified image 960 is turned off or hidden from view, the display 816 displays only the primary image 950. The end instruction may be sent to the control unit 210 coupled to the corresponding auxiliary imaging module to deregister the magnified image 960 of the image display 816 so that the superimposed appearance of the magnified image 960 over the primary image disappears. In some embodiments, the auxiliary imaging module continues to obtain the magnified image 960 although the magnified image 960 is not displayed on the display 816. In other embodiments, the auxiliary imaging module is stopped. The control unit 210 can turn off the auxiliary imaging module and can move the auxiliary imaging module out of the position of the surgeon's line of sight by controlling the actuator 806. In some embodiments, the auxiliary imaging module can be stopped only after a predetermined time has elapsed without the surgeon sending an instruction to turn on the magnified image 816 (e.g., by a resume instruction) or otherwise indicate the magnified image on the display 816.

[0191] In some examples, a surgeon may want to view an enlarged image of another region of interest within the surgical field. Method 900 may return to process 908 if the image processing system 900 does not receive an instruction to end the zoom or if the system 900 receives an instruction from the surgeon to resume zooming. In some embodiments, the surgeon may change his or her point of gaze, and the new instruction may include capturing and displaying an enlarged image of the surgeon's new point of gaze (as outlined, for example, in processes 910-918). In some embodiments, when the current magnification is insufficient to adequately examine the microstructure, the new instruction may instruct a further magnification or "zoom in" to the current enlarged image of the surgeon's point of gaze using processes 910-918.

[0192] FIG. 9 is a flowchart showing a method 1000 for using an image processing system 800 to control imaging devices 112a, 112b, and / or 112c to capture and display primary and secondary images using different imaging modalities, according to one embodiment of the present disclosure. When the imaging device 112a of FIG. 6A is used, a primary image in a first imaging modality can be captured and displayed using the primary imaging module 714, and an auxiliary or secondary image can be captured using an auxiliary imaging module 715 that uses a different imaging modality, such as fluoroscopy or ultrasound. The auxiliary image can assist the surgeon in characterizing and / or identifying structures within the region of interest corresponding to the surgeon's point of gaze. In some embodiments, the auxiliary imaging module 715 can be configured or adjusted to provide an enlarged image of the surgeon's point of gaze in a different imaging modality.

[0193] In some embodiments, when the imaging device 112b of FIG. 6B is used, a primary image can be captured and displayed using the primary imaging device 112b having the primary imaging module 720, and a secondary or auxiliary image can be captured in a different imaging modality, such as fluoroscopy or ultrasound, using the auxiliary imaging module 722 of the auxiliary imaging device 112c to characterize and / or identify the structure at the surgeon's point of regard. In some embodiments, the auxiliary imaging module 722 can be adjusted to provide a magnified image of the surgeon's point of regard in a different imaging modality.

[0194] Prior to the start of the surgery, the image processing system 800 configured to provide the auxiliary image can be changed based on the surgeon's request. In some embodiments, for example, if the surgeon desires an auxiliary image by fluoroscopy, the illumination unit 810 can use an X-ray source and the display 816 can be a fluorescent screen. In some embodiments, the image processing system 800 can include any known technique for converting X-rays into a visible light output and can couple the display 816 to an auxiliary imaging module having a CCD video camera. The image processing system 800 for fluoroscopy can enable the image to be recorded and shown on the display 816.

[0195] In some embodiments, if the surgeon desires an auxiliary image by ultrasound, the illumination unit 810 can use a piezoelectric transducer configured to generate ultrasound, the auxiliary imaging module can be an ultrasound scanner operating in an appropriate frequency range, and the endoscope processor 808 can process and convert the received sound waves into a digital image to be displayed on the display 816. Thus, the image processing system 800 can be changed and configured to accommodate any suitable arrangement of the required components and any suitable technique so as to be able to capture and display the desired imaging modality based on the surgeon's request.

[0196] In process 1002 of method 1000, the primary imaging module is used to capture a primary image of the surgical field. In process 1004, the image processing system 800 processes and displays the primary image on the display 816. In process 1006, the eye tracking system 200 (shown in FIG. 2C) tracks the surgeon's point of gaze. In some embodiments, the primary image may include a stereoscopic image and the 3D position of the surgeon's point of gaze is tracked. Processes 1002, 1004, and 1006 of method 1000 may be substantially similar to processes 902, 904, and 906 of method 850 shown in FIG. 8A.

[0197] In process 1008 of method 1000, the surgeon may direct his or her point of gaze to the region of interest and may enter an instruction using any suitable method (e.g., for viewing a secondary image). For example, the surgeon may send an instruction by pressing a button on the surgeon console, lightly tapping a foot pedal, sending a voice message, or winking. The instruction may be sent to the control unit 802 (shown in FIG. 6A) coupled to the auxiliary imaging module. In some embodiments, the instruction may also include information such as a preferred type of imaging modality and / or a preferred magnification. In addition, the position information of the surgeon's point of gaze provided by the eye line processor 206 (shown in FIG. 2C) may also be sent to the control unit 802 of the auxiliary imaging module.

[0198] In process 1010, similar to process 911 of FIG. 8A, the image processing system 400 determines whether the surgeon's line of sight is directed to a region outside the region shown by the primary image captured by the primary imaging module. If it is determined that the surgeon's point of gaze is outside the region shown by the primary image, method 1000 may proceed back to process 1002 by adjusting the position and orientation of the primary imaging module to form an updated primary image that includes the region corresponding to the surgeon's point of gaze. When the surgeon's point of gaze is within the primary image, method 1000 proceeds to process 1012.

[0199] In process 1012, the control unit 802 processes the surgeon's instructions received in process 1008 and prepares the auxiliary imaging module to capture the desired secondary image. In some embodiments, the control unit 802 may instruct the actuator 806 to physically position or align the auxiliary imaging module to capture the requested image.

[0200] In process 1014, the auxiliary imaging module may capture an image in the requested modality according to the surgeon's instructions. Additionally or alternatively, the control unit 7802 instructs the endoscope processor 808 to digitally prepare the secondary image in the desired modality.

[0201] In process 1016, the endoscope processor 808 processes and displays the requested image on the display 816. In some embodiments, as described above, the endoscope processor 808 can process and identify common features between the secondary image and the primary image such that the secondary image can be aligned and displayed as an overlay on top of the primary image on the display 816. It should be understood that other suitable methods may be used to align the secondary image and to overlay the secondary image on top of the primary image. In other embodiments, the secondary image may be displayed adjacent to the primary image in an overlapping or non - overlapping manner.

[0202] In process 1018, the image processing system 800 determines whether the surgeon wishes to continue the assisted imaging process and / or continue viewing the assisted images. If continued assisted imaging is requested by the surgeon, regardless of whether the images are at different positions or of different modalities, method 1000 returns to process 1008 to obtain new instructions from the surgeon. If the image processing system 800 determines that the surgeon wishes to end the assisted imaging process and / or stop viewing the assisted images, the assisted imaging process can be ended and / or the assisted images can be hidden (e.g., toggled off) in process 1020. Processes 1012 - 1020 of method 1000 are substantially similar to processes 912 - 920 of method 900 of FIG. 8A.

[0203] FIG. 10A shows a method 1100 for displaying a plurality of images (e.g., images 1150 - 1154, 1160 - 1164, and 1170 - 1174 shown in FIG. 10B) that are captured when the imaging module follows the surgeon's line of sight and scans across the area represented by the primary image on the image display 816. FIG. 10B shows a plurality of exemplary images that are captured and displayed when the imaging module follows the instructions conveyed by the surgeon's line of sight to scan across the primary image on the image display 816.

[0204] In process 1102, the image processing system 800 captures and displays a primary image using a primary imaging module (e.g., the primary imaging module 714 of FIG. 6A or the primary imaging module 720 of FIG. 6B). As described above, the primary imaging module can be a stereoscopic camera configured to capture a stereoscopic image of the surgical area. The primary imaging module can also be actuated and controlled by a surgeon at the surgeon console 120 (as shown in FIG. 1C). The primary imaging module can also be adjusted by other personnel in the operating room. In some embodiments, the primary image can be an overall image of the surgical area, such as the patient's abdominal cavity at zero or low magnification. The captured primary image can be processed by an endoscope processor 808, which can be coupled to the primary imaging module. The endoscope processor 808 can receive the primary image and digitize various types of primary image data, such as the position, boundaries, and / or magnification level of the primary image. The digitized information can be saved in one or more memories coupled to the endoscope processor 808. The endoscope processor 808 can also process the imaging information of the primary image to display the primary image on the display 816.

[0205] In process 1104, the visual target tracking system 200 of FIG. 2C tracks the surgeon's point of gaze as the surgeon's eyes scan across the primary image obtained in process 1102. The surgeon's line of sight can be tracked using the eye trackers 204a and 204b of the visual target tracking system 200. The line of sight processor 206 can process the line of sight information provided by the eye trackers 204a and 204b and digitize data regarding the surgeon's point of gaze, such as the 3D position of the surgeon's point of gaze with respect to the image display 202 and / or the surgical field. The digitized information can be saved in one or more memories 208 coupled to the line of sight processor 206. The line of sight processor 206 can also provide information to the control unit 802 when the surgeon sends an instruction based on the surgeon's line of sight.

[0206] In process 1106, the image processing system 800 receives support from a surgeon. In some embodiments, the surgeon may identify one or more regions of interest while scanning a temporary image for further investigation. The surgeon may request a modified view of the region of interest (e.g., by way of non-limiting example, an enlarged field of view, a field of view of a different imaging modality, an enhanced or sharpened field of view, a brightened or colorized field of view, or a graphically adjusted field of view in some other way). For example, the surgeon may also request a field of view of the region of interest in a different imaging modality to further investigate and / or characterize the structure of interest. In some embodiments, the surgeon may request an enlarged field of view of the region of interest to view a more detailed image of the structure within the region of interest.

[0207] When the surgeon gazes at the region of interest, the surgeon may enter a confirmation instruction using any of a variety of suitable input methods. For example, the surgeon may tap a foot pedal that cues the corresponding button on the surgeon console to enter the desired instruction, send a voice message, or wink in a particular pattern to enter the desired instruction. The confirmation instruction can be sent to a control unit 802, which may or may not be the same as the primary imaging module and is coupled to an imaging module. When the imaging module is different from the primary imaging module 714 or the primary imaging module 720, the auxiliary imaging module 715 of FIG. 6A or the auxiliary imaging module 722 of FIG. 6B can be activated by the control unit 802. In some embodiments, the confirmation instruction may also include the surgeon's desired investigation mode information, such as a preferred imaging modality, a desired magnification level for a "zoomed-in" image, and / or other types of visual / graphic effects. In addition, the 3D position information of the surgeon's point of fixation provided by the gaze processor 206 is sent to the control unit 802. In some embodiments, prior to sending the surgeon's confirmation instruction to the control unit 802, the image processing system 800 (e.g., the processor of the control unit 802 and / or the gaze processor 206) may perform one or more decision processes similar to processes 910 and 911 of method 900.

[0208] In process 1108, the image processing system 800 processes a confirmation instruction from the surgeon. The control unit 802 receives the position information of the surgeon's gaze point and the confirmation instruction (including the desired investigation mode information of the surgeon) from the gaze processor 206. The control unit 802 may send the position data and the confirmation instruction to the instrument interface 150 connected to the appropriate imaging module. In some embodiments, the instrument interface 150 may send the position data and the confirmation instruction to the actuator 806, and the actuator 806 may move the imaging module to a physical position where the imaging module can obtain a desired image of the region of interest presented by the surgeon's gaze point. For example, when the surgeon requests an enlarged image of the region of interest identified by the surgeon's gaze point, the imaging module may be moved by the actuator 806 to extend along the axis of the shaft to capture the enlarged image at the desired level of magnification.

[0209] In process 1110, the imaging module captures a secondary image of the region of interest indicated by the surgeon's gaze point (e.g., based on the surgeon's instruction transmitted in process 1106). In particular, in some embodiments, the actuator 806 adjusts the position and orientation of the imaging module to the region of interest indicated by the surgeon's gaze point, and the imaging module captures a secondary image of the region of interest for the surgeon's investigation. In some embodiments, the imaging module may also capture an image during the movement of the imaging module across the surgical field towards the region of interest before the imaging module reaches the region of interest reflected by the surgeon's gaze point. In some embodiments, the imaging module may include a stereoscopic camera as shown in FIG. 6C that can capture stereoscopic images.

[0210] In process 1112, the image processing system 800 processes the captured secondary image of the region of interest reflected by the surgeon's line of sight and / or the image captured when the imaging module moves towards the region of interest. The endoscope processor 808 may digitize the secondary image before presenting the image on the display 816. For example, in some embodiments, the endoscope processor 808 may create a digitally magnified secondary image from the primary image and / or apply various graphic effects to the captured secondary image. The endoscope processor 808 may also generate digitized information of the secondary image, such as position information regarding the surgeon's line of sight and position and orientation information regarding the imaging module used to capture the image. This digitized data may be stored in a memory, such as the memory 208 of FIG. 2C and / or the memory 812 of FIG. 7A, as a non-limiting example.

[0211] In process 1114, the image processing system 800 displays the captured secondary image of the region of interest indicated by the surgeon's line of sight on the display 816. In some embodiments, the endoscope processor 808 may compare and identify common features between the secondary image and the primary image, and then spatially align the captured image to correctly align and overlay the secondary image on top of the temporary image using the common features. It should be understood that any other suitable method may be used to align the captured image and the primary image together. In some embodiments, when the imaging module includes a stereo camera as shown in FIG. 6C, the stereo camera may capture a stereo image, and the endoscope processor 808 may generate left and right stereo images to be displayed on the display 816 so that the surgeon can perceive a 3D stereo image of the region indicated by the line of sight.

[0212] In process 1116, the image processing system 800 queries whether the surgeon has finished scanning the primary image. If the surgeon has not finished scanning the primary image, or if the surgeon wishes to double-check one or more previously scanned spots, method 1100 can proceed back to process 1104 to continue tracking the surgeon's line of sight. As shown in FIG. 10B, if the surgeon instructs system 10 to capture multiple secondary images of his or her points of fixation at different positions while the surgeon scans the primary image, multiple images 1150-1154, 1160-1164, and 1170-1174 of the regions of interest indicated by the points of fixation are processed and shown on display 816. Images 1150-1154, 1160-1164, and 1170-1174 can be shown on display 816 as overlaid images on the primary image or adjacent to a temporary image.

[0213] During the procedure, the surgeon may at any time wish to stop scanning the primary image for any reason, and the surgeon can send an instruction to system 10 using any suitable method. In process 1116, when the image processing system 800 determines that the surgeon has finished scanning the primary image, the line of sight processor 206 can confirm the end of the scanning process by informing the surgeon of the end of the capture of secondary images. In some embodiments, the image processing system 800 can also prompt the surgeon to review the multiple images and select one image for further evaluation.

[0214] In process 1118, the surgeon may review a plurality of images 1150 - 1154, 1160 - 1164, and 1170 - 1174 that are tracked and captured using the surgeon's point of gaze as the surgeon scans across the primary image. The surgeon may select one or more images to be presented on display 816 (e.g., central image 1162 of display 816 shown in FIG. 10C) for further investigation. The surgeon may select an image by using his or her point of gaze or by entering his or her selection using any other suitable method.

[0215] In process 1120, image processing system 800 processes the surgeon's selection. The surgeon may request a particular imaging effect or type of imaging modality for further investigation of the selected image. For example, the surgeon may instruct system 800 to display a magnified image 1176 of the selected image 1162, and the magnified image 1176 may be overlaid on image 1162 as shown in FIG. 10C. Image processing system 800, which is coupled to an imaging module that performs the requested imaging function, may be configured to have any suitable technology in any suitable arrangement to meet the surgeon's request. The received selection and image processing instructions may be sent to control unit 802 to be processed. In some embodiments, control unit 802 may also exchange information with memory 208 and / or 812 to read out position data of the newly selected region of interest and position and orientation data of the imaging module that captured the image of the point of interest. In some embodiments, the read data may be used to adjust actuator 806 to reposition the imaging module to capture an image of the region of interest corresponding to the selected image based on the surgeon's request.

[0216] In process 1122, the requested image of the selected image is either captured or digitally created and then displayed on display 816 for review by the surgeon. FIG. 10C shows an example of displaying an enlarged image 1176 of image 1162 based on a request to the surgeon from among the plurality of images of FIG. 10B. In the illustrated embodiment, the enlarged image 1176 is shown superimposed on the selected image 1162. In some embodiments, the requested image (e.g., enlarged image 1176) can be shown in the central portion of display 816 regardless of the original position of the selected image on display 816. In some embodiments, a stereoscopic camera can be used as an imaging module to capture the enlarged image and the rendered image can be displayed such that it is perceived by the surgeon as a 3D enlargement of the selected image 1162. In some embodiments, newly created or captured images can be co-registered so as to be on top of previously captured images using substantially the same methods discussed previously in connection with process 1114. Processes 1118-1122 can be optional in some surgeon scans.

[0217] A first embodiment of a medical system includes an image display configured to display a primary image of an operative field to a user, a gaze tracking system configured to measure user gaze point data and to determine a viewing location in the primary image at which the user's gaze is directed based on the gaze point data, and an image processing system coupled to the image display and configured to adjust the primary image of the image display based on the viewing location.

[0218] In the first medical system, the image processing system is configured to receive an instruction from the user and to adjust the primary image based on the instruction.

[0219] In the first medical system, the image processing system is configured to adjust the primary image by displaying a secondary image of the viewing location on the image display based on the gaze point data.

[0220] In the first medical system, the image processing system is configured to align a secondary image with a primary image such that the secondary image at the viewing position is displayed as an overlay on the primary image on the image display.

[0221] In the first medical system, the image processing system is configured to align a secondary image with a primary image such that the secondary image at the viewing position is displayed as an adjacent primary image on the image display.

[0222] The image processing system further includes a first imaging module and a second imaging module. The first imaging module is configured to capture a secondary image based on fixation point data, and the first imaging module and the second imaging module are independently controlled.

[0223] In the first medical system, the first and second imaging modules are components of the first imaging device.

[0224] In the first medical system, the first imaging module has a first imaging device, and the second imaging module has a second imaging device.

[0225] In the first medical system, at least one of the first and second imaging modules includes a stereo camera.

[0226] In the first medical system, the first imaging module operates in a first imaging modality, and the second imaging module operates in a second imaging modality.

[0227] In the first medical system, the first imaging module operates in a first wavelength range, and the second imaging module operates in a second wavelength range, and the first wavelength range is different from the second wavelength range.

[0228] The first medical system includes a fixation tracking threshold region in the secondary image, the fixation tracking threshold region being smaller than the secondary image, and the image processing system is configured to maintain the primary image and the secondary image in response to changes in the fixation point data reflecting the movement of the user's line of sight within the fixation tracking threshold region.

[0229] The first medical system further has an endoscope processor coupled to the first imaging module and the second imaging module, and the endoscope processor is configured to adjust at least one of the primary image and the secondary image according to the determined viewpoint and the instruction input from the user.

[0230] In the first medical system, the endoscope processor is configured to independently control at least one function of the first imaging module and at least one function of the second imaging module based on the determined viewpoint in the primary image.

[0231] In the first medical system, the endoscope processor is configured to orient and position the secondary imaging module in the surgical field to capture the secondary image based on the instruction input from the user and the determined viewpoint in the primary image.

[0232] The first medical system further has a first articulation list section for adjusting the position and orientation of the first imaging module, and a second articulation list section for adjusting the position and orientation of the second imaging module, and the first and second articulation list sections move in response to a command signal from the endoscope processor.

[0233] The first medical system further has a first shaft coupled to the second imaging module, and the second imaging module is configured to move along the longitudinal axis of the first shaft in response to a command signal from the endoscope processor.

[0234] In the first medical system, the endoscope processor is configured to orient and position the first imaging module in the surgical field in order to maintain the secondary image within the primary image based on the fixation point data and the determined viewpoint in the primary image.

[0235] In the first medical system, the endoscope processor is configured to adjust the secondary image acquired by the second imaging module based on an instruction input from the user.

[0236] In the first medical system, the endoscope processor is configured to emphasize the secondary image relative to the primary image on the image display.

[0237] In the first medical system, the endoscope processor is configured to digitally modify the secondary image relative to the primary image on the image display.

[0238] In the first medical system, the endoscope processor is configured to digitally enlarge the secondary image relative to the primary image on the image display.

[0239] In the first medical system, the target aiming system has at least two eye trackers.

[0240] In the first medical system, the image display is included in the target tracking system and has a left-eye image display and a right-eye image display.

[0241] In the first medical system, the image display is a 3D image display configured to display a 3D image of the primary image to the user.

[0242] The first method of operating the surgical system includes the steps of displaying the primary image on the image display, tracking the user's fixation point data using the target tracking system when the user views the primary image on the image display, and adjusting the primary image based on the fixation point data.

[0243] The first method of operating the surgical system further includes the step of receiving an instruction input from a user.

[0244] The first method of operating the surgical system includes the step of displaying a secondary image of a region of interest identified by the fixation point data based on the instruction input from the user.

[0245] In the first method of operating the surgical system, the secondary image includes a virtual adjustment of a portion of the primary image based on the fixation point data.

[0246] In the first method of operating the surgical system, the secondary image includes an enlarged portion of the primary image based on the fixation point data.

[0247] In the first method of operating the surgical system, the secondary image includes a sharpened portion of the primary image based on the fixation point data.

[0248] In the first method of operating the surgical system, the secondary image includes a highlighted portion of the primary image based on the fixation point data.

[0249] In the first method of operating the surgical system, it is an imaging modality different from the primary image.

[0250] The first method of operating the surgical system further includes the step of capturing a secondary image using an imaging module based on the instruction input from the user and the fixation point data.

[0251] In the first method of operating the surgical system, the steps of determining a viewpoint in the primary image towards which the user's fixation point is directed based on the fixation point data, determining whether the viewpoint is outside a predetermined visual target tracking threshold region, and updating the secondary image based on the viewpoint when the viewpoint is outside the predetermined visual target tracking threshold region are included.

[0252] In a first method of operating the surgical system, the position and orientation of the imaging module are adjusted based on the viewpoint and the input indication.

[0253] In a first method of operating the surgical system, the imaging module is configured to capture a plurality of secondary images when the user scans the primary image.

[0254] In a first method of operating the surgical system, the primary image is captured using a primary imaging module, and the secondary images are captured using a secondary imaging module.

[0255] In a first method of operating the surgical system, the primary imaging module is disposed in a first imaging device, and the secondary imaging module is disposed in a second imaging device different from the first imaging device.

[0256] In a first method of operating the surgical system, the step of determining whether the viewpoint is outside a predetermined fiducial tracking threshold region includes comparing an average viewpoint over a predetermined number of image frames with the predetermined fiducial tracking threshold region.

[0257] In a first method of operating the surgical system, the step of determining whether the viewpoint is outside a predetermined fiducial tracking threshold region includes comparing an average viewpoint over a predetermined period with the predetermined fiducial tracking threshold region.

[0258] Although exemplary embodiments have been described and shown, a wide range of modifications, changes, and substitutions are contemplated in the foregoing description, and in some instances, some features of the embodiments may be used without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the following claims, and the claims should be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

[0259] The following appendix is noted. (Appendix 1) A teleoperation medical system for performing medical procedures in the surgical field, an image display configured to display an image of the surgical field to the user, at least one eye tracker configured to measure data regarding the user's point of gaze, a processor configured to process the data to determine a viewpoint in the displayed image towards which the user's point of gaze is directed, and a control unit configured to control at least one function of the teleoperation medical system based on the determined viewpoint, having a fixation point tracking unit, A teleoperation medical system. (Appendix 2) The image display is a 3D image display configured to display a 3D image of the surgical field to the user, The teleoperation medical system according to Appendix 1. (Appendix 3) Further including a surgical instrument, the control unit is configured to control at least one function of the surgical instrument in the surgical field based on the determined viewpoint in the displayed image, The teleoperation medical system according to Appendix 1. (Appendix 4) The surgical instrument is displayed in the image displayed at the viewpoint, The teleoperation medical system according to Appendix 3. (Appendix 5) The control unit is configured to calibrate the 3D position of the surgical instrument in the surgical field based on the determined viewpoint in the displayed image, The teleoperation medical system according to Appendix 3. (Appendix 6) The surgical instrument is an energy application instrument, and the control unit is configured to operate the energy application instrument to supply energy into the surgical field based on the determined viewpoint, The teleoperation medical system according to Appendix 3. (Appendix 7) The surgical instrument is a fastener application instrument, and the control unit is configured to operate the fastener application instrument to supply a fastener into the surgical field based on the determined viewpoint. The remote medical system according to Appendix 3. (Appendix 8) The surgical instrument is a fluid management instrument, and the control unit is configured to operate the fluid management instrument to carry fluid into the surgical field based on the determined viewpoint. The remote medical system according to Appendix 3. (Appendix 9) The control unit is configured to activate the surgical instrument when the processor determines that the viewpoint coincides with the position of the surgical instrument on the image display for a predetermined period. The remote medical system according to Appendix 3. (Appendix 10) The control unit is configured to stop the operation of the surgical instrument when the processor determines that the viewpoint does not coincide with the position of the surgical instrument on the image display for a predetermined period. The remote medical system according to Appendix 3. (Appendix 11) The control unit is configured to move the surgical instrument in the surgical field based on the determined viewpoint in the displayed image. The remote medical system according to Appendix 3. (Appendix 12) The control unit is configured to transfer the control of the surgical instrument from the user to the second user when the determined viewpoint in the image displayed by the second user coincides with the position of the surgical instrument on the image display for a predetermined period. The remote medical system according to Appendix 3. (Appendix 13) Further having a marking application, the control unit is configured to control the marking application to attach a virtual mark visible on the image display to the surgical field based on the determined viewpoint. The remote medical system according to Appendix 1. (Appended Note 14) Further comprising an imaging device, wherein the control unit is configured to control at least one function of the imaging device in the surgical field based on the determined viewpoint. The remote-controlled medical system according to Appended Note 1. (Appended Note 15) The at least one function of the imaging device includes the movement of the imaging device. The remote-controlled medical system according to Appended Note 14. (Appended Note 16) The at least one function of the imaging device includes the zoom operation of the imaging device. The remote-controlled medical system according to Appended Note 14. (Appended Note 17) The at least one function of the imaging device includes the focusing operation of the imaging device. The remote-controlled medical system according to Appended Note 14. (Appended Note 18) The at least one function of the imaging device includes the mode change operation of the imaging device. The remote-controlled medical system according to Appended Note 14. (Appended Note 19) The at least one function controlled by the control unit based on the determined viewpoint includes the user login function. The remote-controlled medical system according to Appended Note 1. (Appended Note 20) The control unit is configured to expel the user from the remote-controlled medical system when the processor fails to determine the viewpoint on the image display. The remote-controlled medical system according to Appended Note 1. (Appended Note 21) The control unit is configured to expel the user from the remote-controlled medical system when the at least one eye tracker fails to detect the user's eyes. The remote-controlled medical system according to Appended Note 1. (Appended Note 22) The image display is configured to display an image of a user interface having a plurality of function options to the user. The remote-controlled medical system according to Appended Note 1. (Supplementary Note 23) The control unit is configured to start at least one of the plurality of function options when the determined viewpoint coincides with the position of at least one of the function options in the displayed image of the user interface. The remote operation medical system according to Supplementary Note 22. (Supplementary Note 24) The control unit is configured to adjust the positional depth of the user interface in the displayed image based on the determined viewpoint. The remote operation medical system according to Supplementary Note 22. (Supplementary Note 25) The processor is configured to generate an evaluation image of the viewpoint towards which the user's fixation point is directed on the evaluation image display. The remote operation medical system according to Supplementary Note 1. (Supplementary Note 26) The image display is configured to display the fixation points of other users looking at the surgical field to the user. The remote operation medical system according to Supplementary Note 1. (Supplementary Note 27) The processor is configured to receive an instruction input from the user and adjust the displayed image based on the instruction. The remote operation medical system according to Supplementary Note 1. (Supplementary Note 28) The processor is configured to adjust the displayed image by executing a zoom function. The remote operation medical system according to Supplementary Note 27. (Supplementary Note 29) The processor is configured to adjust the displayed image by displaying a secondary image of the viewpoint as an overlay on the image display based on the fixation point data. The remote operation medical system according to Supplementary Note 27. (Appendix 30) It further has a first imaging module and a second imaging module, the first imaging module is configured to acquire the displayed image, the second imaging module is configured to capture the secondary image based on the fixation point data, and the first imaging module and the second imaging module are independently controlled. The remote operation medical system according to Appendix 29. (Appendix 31) An operation method for operating a remote operation medical system, displaying an image including an operative field image on an image display; measuring a user's fixation point on the image display; determining a viewpoint in the displayed image toward which the user's fixation point is directed; and controlling at least one function of the remote operation medical system based on the determined viewpoint. Method. (Appendix 32) The at least one function of the remote operation medical system is a three-dimensional configuration function including comparing a predetermined target in the displayed image having known three-dimensional position parameters with the viewpoint. The method according to Appendix 31. (Appendix 33) The at least one function of the remote operation medical system is an evaluation function including displaying the viewpoint to other users. The method according to Appendix 31. (Appendix 34) It further includes providing an instruction position in the displayed image toward which the fixation point of another user is directed. The method according to Appendix 31. (Appendix 35) The displayed operative field image is a 3D image. The method according to Appendix 31. (Appendix 36) The at least one function of the remote operation medical system is a surgical instrument operation function including a step of controlling at least one function of a surgical instrument in the surgical field based on the determined viewpoint in the displayed image. The method according to Appendix 31. (Appendix 37) The surgical instrument is displayed in the surgical field image at the viewpoint. The method according to Appendix 36. (Appendix 38) The at least one function of the remote operation medical system is an instrument operation function. The method according to Appendix 35. (Appendix 39) The instrument operation function includes operating an energy application instrument to supply energy at a certain position in the surgical field based on the determined viewpoint. The method according to Appendix 38. (Appendix 40) The instrument operation function includes operating a fastener application instrument to supply a fastener at a certain position in the surgical field based on the determined viewpoint. The method according to Appendix 38. (Appendix 41) The instrument operation function includes operating a fluid management instrument to carry fluid at a certain position in the surgical field based on the determined viewpoint. The method according to Appendix 38. (Appendix 42) The instrument operation function includes activating the surgical instrument when the viewpoint coincides with the position of the surgical instrument in the image where the viewpoint is displayed for a predetermined period. The method according to Appendix 36. (Appendix 43) The instrument operation function includes transferring control of the surgical instrument from the user to the second user when the viewpoint of the second user coincides with the position of the surgical instrument in the image displayed for a predetermined period. The method according to Appendix 32. (Appendix 44) The instrument operation function includes stopping the operation of the surgical instrument when the viewpoint does not coincide with the position of the surgical instrument in the image where the viewpoint is displayed for a predetermined period. The method according to Appendix 32. (Appendix 45) The instrument operation function includes moving the surgical instrument within the surgical field based on the determined viewpoint. The method according to Appendix 32. (Appendix 46) The at least one function of the teleoperation medical system is a marking function that includes creating a virtual mark visible on the image display in the surgical field based on the determined viewpoint. The method according to Appendix 31. (Appendix 47) The at least one function of the teleoperation medical system is an imaging device operation function that includes controlling at least one function of the imaging device in the surgical field based on the determined viewpoint. The method according to Appendix 31. (Appendix 48) The at least one function of the imaging device includes the movement of the imaging device. The method according to Appendix 47. (Appendix 49) The at least one function of the imaging device includes the zoom operation of the imaging device. The method according to Appendix 47. (Appendix 50) The at least one function of the imaging device includes the focusing operation of the imaging device. The method according to Appendix 47. (Appendix 51) The at least one function of the teleoperation medical system is a user login function. The method according to Appendix 31. (Appendix 52) The at least one function of the teleoperation medical system is a lockout function for expelling the user from the teleoperation medical system when the user's gaze point is directed away from the displayed image. The method according to Appendix 31. (Appendix 53) The displayed image includes a user interface having a plurality of function options. The method according to Appendix 31. (Appendix 54) When the determined viewpoint coincides with the position of at least one of the function options in the displayed image of the user interface, further including the step of starting at least one of the plurality of function options The method according to Appendix 53. (Appendix 55) Further including the step of adjusting the positional depth of the user interface in the displayed image based on the determined viewpoint The method according to Appendix 54. (Appendix 56) Further including the step of receiving an instruction input from the user and adjusting the displayed image based on the instruction The method according to Appendix 31. (Appendix 57) The displayed image is adjusted by executing a zoom function The method according to Appendix 56. (Appendix 58) The displayed image is adjusted by displaying an overlay image on the image display based on the fixation point data The method according to Appendix 56.

Claims

1. A teleoperation medical system for performing a medical procedure in an operative field, the teleoperation medical system comprising: A fixation point tracking unit comprising: An image display configured to display an image of the operative field to a user; At least one eye tracker configured to measure data regarding a fixation point of the user; A processor configured to process the data to determine a first determined fixation point in the displayed image toward which the fixation point of the user is directed and a second determined fixation point in the displayed image toward which the fixation point of the user is directed; A fixation point tracking unit including the above; and A control unit comprising: When the first determined fixation point is within a predetermined area that coincides with a range of the shape of a surgical instrument in the displayed image and the processor determines that the first determined fixation point has been within the predetermined area for a first predetermined period, activating at least one function of the surgical instrument displayed in the displayed image; After activation of the surgical instrument, when the processor determines that the second determined fixation point is outside the predetermined area for a second predetermined period, deactivating the at least one function of the surgical instrument. A control unit configured as above; Having A teleoperation medical system.

2. The image display is a 3D image display configured to display a 3D image of the operative field to the user, The teleoperation medical system according to claim 1.

3. Further including the surgical instrument, The teleoperation medical system according to claim 1 or 2.

4. The control unit is configured to calibrate a 3D position of the surgical instrument in the displayed image of the operative field based on the first determined fixation point in the displayed image, The teleoperation medical system according to any one of claims 1 to 3.

5. The surgical instrument is an energy application instrument, and the at least one function of the surgical instrument includes supplying energy into the operative field, The teleoperation medical system according to any one of claims 1 to 3.

6. The surgical instrument is a fastener application instrument, and the at least one function of the surgical instrument includes supplying a fastener into the operative field, The teleoperation medical system according to any one of claims 1 to 3.

7. The surgical instrument is a fluid management instrument, and at least one function of the surgical instrument includes transporting fluid into the surgical field. The teleoperation medical system according to any one of claims 1 to 3.

8. The control unit is configured to operate the surgical instrument in the surgical field based on the first determined viewpoint in the displayed image. The teleoperation medical system according to any one of claims 1 to 3.

9. A method of operating a teleoperation medical system, the method comprising: a step in which a visual target tracking unit of the teleoperation medical system displays an image including a surgical field on an image display; a step in which the visual target tracking unit measures a user's gaze point on the image display; a step in which the visual target tracking unit determines a first determined viewpoint in the displayed image toward which the user's gaze point is directed and a second determined viewpoint in the displayed image toward which the user's gaze point is directed; a step in which the control unit of the teleoperation medical system determines to generate a command to activate at least one function of the surgical instrument displayed in the displayed image when the first determined viewpoint is within a predetermined area that matches the shape range of the surgical instrument in the displayed image and when the first determined viewpoint is within the predetermined area for a first predetermined period; a step in which, after activation of the surgical instrument, the control unit determines to generate a command to stop operation of the at least one function of the surgical instrument when the second determined viewpoint is outside the predetermined area for a second predetermined period; including method.

10. The method according to claim 9, further comprising a step in which the control unit calibrates the 3D position of the surgical instrument in the displayed image of the surgical field based on the first determined viewpoint in the displayed image. The method according to claim 9.

11. The surgical instrument is an energy application instrument, and at least one function of the surgical instrument includes supplying energy into the surgical field. The method according to claim 9.

12. The surgical instrument is a fastener application instrument, and at least one function of the surgical instrument includes supplying a fastener into the surgical field. The method according to claim 9.

13. The surgical instrument is a fluid management instrument, and the at least one function of the surgical instrument includes carrying fluid into the surgical field. The method according to claim 9.

Images