Systems and methods for constraining virtual reality surgical systems

Abstract

This application relates to systems and methods for constraining a virtual reality surgical system. This application discloses a method of operating a surgical control system, the method comprising displaying an image of the surgical environment from the field of view of an imaging instrument on a display system. The display system is configured to be mounted on a user's head. The method further includes detecting imaging control input from the user, and in response to the detection of the imaging control input, enabling an imaging control mode of the surgical control system. The method also includes detecting movement of the user's head. When the surgical control system is in imaging control mode and in response to the user's head movement, an image of the surgical environment with a changed field of view from the imaging instrument is displayed. The changed field of view corresponds to the detected movement of the user's head.

Description

[0001] This application is a divisional application of Chinese Patent Application 201880023943.2 (PCT / US2018 / 028376), filed on April 19, 2018, entitled "System and Method for Restraining Virtual Reality Surgical System".

[0002] Cross-reference of related applications

[0003] This application claims the benefit of U.S. Provisional Application 62 / 487,833, filed April 20, 2017, which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure relates to systems and methods for performing medical procedures or training medical procedures using a virtual reality display system, and more specifically to systems and methods for providing constraints within a surgical system or surgical training system when using a virtual reality display system. Background Technology

[0005] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during invasive medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. These techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, clinicians can insert medical instruments to reach target tissue locations. Minimally invasive medical instruments include instruments such as therapeutic, diagnostic, and surgical instruments. They can also include imaging instruments such as endoscopic instruments, which provide the user with a field of view within the patient's anatomy. Some minimally invasive medical instruments and imaging instruments can be remotely operated or otherwise computer-assisted. Head-mounted display systems can be used to provide clinicians with immersive augmented reality or virtual reality experiences when performing remote procedures or providing training simulations for them. When using immersive display systems, system control is required to provide safe and reliable operation of the remote operating system. Summary of the Invention

[0006] Embodiments of the invention are summarized in the appended claims.

[0007] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the disclosure without limiting its scope. In this regard, other aspects, features, and advantages of the disclosure will be apparent to those skilled in the art from the following detailed description.

[0008] In one embodiment, a method of operating a surgical control system includes displaying an image of the surgical environment from the field of view of an imaging instrument on a display system. The display system is configured to be mounted on the user's head. The method also includes detecting imaging control input from the user and, in response to the detection of the imaging control input, enabling an imaging control mode of the surgical control system. The method further includes detecting movement of the user's head. When the surgical control system is in imaging control mode and in response to movement of the user's head, an image of the surgical environment with a changed field of view from the imaging instrument is displayed. The changed field of view corresponds to the detected movement of the user's head.

[0009] In another embodiment, the method of operating a surgical control system includes generating an image of the surgical environment from a viewpoint of an imaging tool having a field of view. Surgical instruments are positioned within the surgical environment. The method also includes displaying the image of the surgical environment on a display system. The display system is configured to be mounted on a user's head. The method further includes detecting movement of the user's head and determining whether the movement of the user's head is within a boundary corresponding to the field of view of the imaging tool. If the movement of the user's head is within the boundary, the image of the surgical environment on the display system is changed by generating a changed viewpoint in the field of view of the imaging tool corresponding to the detected movement of the user's head.

[0010] In another embodiment, a method of operating a surgical control system includes generating an image of a surgical environment. Surgical instruments are positioned within the surgical environment and coupled to a manipulator arm having a range of motion. The method also includes displaying the image of the surgical environment on a display system configured to be mounted on a user's head. The method further includes detecting manipulator arm input motion from the user and determining whether the manipulator arm input motion corresponds to movement of the manipulator arm within its range of motion. If the movement of the manipulator arm input motion corresponds to movement outside the range of motion of the manipulator arm, a ghost image of the surgical instruments is generated, wherein the ghost image moves corresponding to the manipulator arm input motion. Attached Figure Description

[0011] The various aspects of this disclosure can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that, in accordance with standard industry practice, the various features are not drawn to scale. In fact, the dimensions of the various features can be arbitrarily increased or decreased for clarity of discussion. Furthermore, reference numerals and / or letters may be repeated in various examples within this disclosure. Such repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or configurations discussed.

[0012] Figure 1a Provides a view of the surgical environment including a remotely operated surgical system, which includes an immersive display system.

[0013] Figure 1b It provides a view of a simulated surgical environment including a remotely operated surgical system, which includes an immersive display system.

[0014] Figure 2 Provides field-of-view images from the imaging instrument.

[0015] Figure 3 The manual controller of the remote surgical system is shown.

[0016] Figure 4 This is a flowchart of a method for entering and exiting a virtual exploration mode of a remote surgical system.

[0017] Figures 5 to 8 It shows images of the surgical environment, the external environment, and the immersive display system.

[0018] Figure 9 This is a flowchart providing a method for operating within the imaging control mode of a remotely operated surgical system.

[0019] Figure 10 It shows images of the surgical environment, the external environment, and the immersive display system.

[0020] Figure 11 This demonstrates a method for entering and operating within a virtual manipulator arm mode of a remote operating system. Detailed Implementation

[0021] To facilitate an understanding of the principles of this disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings, and these embodiments will be described using specific language. However, it should be understood that this is not intended to limit the scope of the disclosure. Numerous specific details are set forth in the following detailed description of various aspects of the invention to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that embodiments of this disclosure can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention.

[0022] As would be readily apparent to those skilled in the art from the present disclosure, any changes and modifications to the described apparatus, instruments, methods, and any further application of the principles of this disclosure are fully contemplated. In particular, it is fully anticipated that features, components, and / or steps described with respect to one embodiment may be combined with features, components, and / or steps described with respect to other embodiments of this disclosure. Furthermore, the dimensions provided herein are for specific examples, and it is contemplated that different sizes, dimensions, and / or ratios may be used to implement the concepts of this disclosure. To avoid unnecessary descriptive repetition, one or more components or actions described according to one illustrative embodiment may be used or omitted from other illustrative embodiments where applicable. For brevity, multiple iterations of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used in all figures to denote the same or similar parts.

[0023] The following embodiments will describe various instruments and parts thereof in three-dimensional space. As used herein, the term "orientation" refers to the position of an object or part thereof in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part thereof (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "attitude" refers to the orientation of an object or part thereof in at least one translational degree of freedom and the orientation of an object or part thereof in at least one rotational degree of freedom (up to six degrees of freedom in total).

[0024] Refer to the attached diagram. Figure 1a The surgical environment 10 includes a remotely operated medical system 11 used in medical procedures, such as diagnostic procedures, treatment procedures, or surgical procedures. The remotely operated medical system typically includes a remotely operated component mounted on or near the operating table O, with the patient P positioned on the operating table O. The remotely operated component may include one or more modular manipulator arms 12. A medical device system 14 or an imaging system 15 may be operatively coupled to the remotely operated manipulator (e.g., arm) of the remotely operated component. The imaging system may be, for example, a stereoscopic endoscopic imaging system. An operator input system 16 allows the surgeon S or other type of clinician to control the operation of the medical device system 14 and / or the imaging system 15. One or more assistant surgeons, anesthesiologists, or support personnel may also be present in the surgical environment.

[0025] The remote-operated medical system also includes a display system 17, which can present images captured by the imaging system 15, surgical navigation and guidance images, and / or alphanumeric or symbolic information to assist the surgeon or assistant in performing surgical procedures. The display system can be, for example, an immersive display system worn by the surgeon S. More specifically, the immersive display system can be a head-mounted display system used to present images of the internal anatomical environment from the imaging system 15, enhanced or virtual images of the internal anatomical environment, images from outside the patient's anatomical structures, interactive user interfaces, or other image-based, graphical, or text-based information to the surgeon S. The head-mounted display system can be worn as a pair of glasses or goggles to cover the user's eyes. Figure 2 This image shows a field-of-view image 30 from an imaging instrument 15 positioned within the internal anatomical structures of patient P. Image 30 can be displayed on a display system 17. Using stereoscopic image data, multiple viewpoints can be generated from the stereoscopic image data of the field of view while the imaging instrument remains in a stationary position. Therefore, using an immersive display system, the field of view can present slightly different viewpoint images as the user's head moves within the display system. This can provide the user with a more immersive experience when viewing the field of view.

[0026] The remote-operated medical system also includes a control system 20 that communicates with the operator input system 16, the remote operation component, and the display system 17, as described below.

[0027] In this embodiment, the operator input system 16 includes one or more operator manual controllers 18 for controlling one or more medical device systems 14 or endoscopic imaging systems 15. Figure 3 The input system may also include other types of user input, including pedal input, gaze tracking, voice command recognition, and head pose recognition. In various alternatives, the operator manual controller 18 may be tethered via power cable and / or signal transmission cable, or it may be untethered / wireless. Figure 3As shown, the operator controller 18 may include one or more of any number of various input devices, such as the grip input device 22 and the trigger switch 24. The input devices may be used, for example, to move the distal end of an endoscopic imaging system within the patient P, close the end effector of a gripper, apply a potential to electrodes, deliver medication, or the like. In various alternatives, the operator input system may additionally or alternatively include joysticks, trackballs, data gloves, trigger guns, hand or foot-operated controllers, voice recognition and control devices, touchscreens, body motion or presence sensors, and the like. In some embodiments, the manual controller 18 will be provided with the same degrees of freedom as the medical device with the remotely operated components to provide the surgeon with a telepresence / presence, i.e., the perception that the control device(s) and the instrument(s) are integrated, so that the surgeon has a strong feeling as if directly controlling the instrument at the surgical site. In other embodiments, the manual controller 18 may have more or fewer degrees of freedom than the associated medical device and still provide telepresence to the surgeon.

[0028] When the surgeon S performs a procedure from the patient side or another location within the surgical environment, the manipulator arm 12 supports and manipulates the medical device system 14 and / or imaging system 15. The number of medical device systems 14 used at one time typically depends on the diagnostic or surgical procedure and space constraints within the operating room, among other factors. Each arm 12 of the teleoperation component may include a kinematic structure of one or more servo or non-servo controlled links. The teleoperation arm 12 may also include multiple motors that drive input devices on the medical device system 14 or imaging system 15. These motors move in response to commands from the control system 20. The motors include drive systems that, when coupled to the medical device system 14 or imaging system 15, can move the system into or out of natural or surgically created anatomical openings. Other electrically driven systems may move the distal end of the system in multiple degrees of freedom, which may include three linear degrees of motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of motion (e.g., rotation about the X, Y, Z Cartesian coordinate axes). Additionally, the motor can be used to actuate an articulated end effector of an instrument used to grasp tissue in the grippers of a biopsy device or similar apparatus. Instrument 14 may include an end effector having a single working component (such as a scalpel, blunt blade, optical fiber, or electrode). Other end effectors may include, for example, forceps, grippers, scissors, or applicators.

[0029] Control system 20 includes at least one memory and at least one processor, and the at least one processor is typically one of multiple processors used to implement control between the medical device system, imaging system 15, operator input system 16, display system 17, and other auxiliary systems, which may include, for example, additional imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, flushing systems, and / or suction systems. Control system 28 also includes programming instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described according to the aspects disclosed herein. Although control system 20 is in Figure 1a The simplified schematic is shown as a single block, but the system may include two or more data processing circuits, with some processing optionally performed on or near the remote operating component, and other processing performed at the operator input system 16 and the like. The control system 20 can employ any of a variety of centralized or distributed data processing architectures. Similarly, programming instructions can be implemented as multiple separate programs or subroutines, or integrated into many other aspects of the remote operating system described herein. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF (Home Wireless Connectivity Protocol), IEEE 802.11, DECT (Digital Enhanced Wireless Communication), and wireless telemetry.

[0030] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical device system or imaging system. In response to the feedback, the servo controllers transmit signals to the operator input system 16. The servo controllers may also transmit signals instructing the remote manipulation component to move the medical device system 14 and / or the endoscopic imaging system 15, which extends through an opening in the body to an internal surgical site within the patient. Any suitable conventional or specialized servo controller may be used. The servo controller may be separate from or integrated with the remote manipulation component. In some embodiments, the servo controller and the remote manipulation component are provided as part of a remote manipulation arm positioned adjacent to the patient's body.

[0031] The control system 20 may be communicatively coupled to the imaging system 15 and may include a processor to process the captured images for subsequent display, such as displaying them to a surgeon on a surgical console or on another suitable display located locally and / or remotely. For example, in the case of using a stereoscopic endoscope, the control system 20 may process the captured images to present the surgeon with a coordinated stereoscopic image of the surgical site. This coordination may include alignment between relative images and may include adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0032] like Figure 1b As shown, the control system 20, operator input system 16, and immersive display system 17 can also be used to perform procedures using the simulated surgical environment 40. The simulated surgical environment 40 can be observed by the user S on the immersive display system 17. In one embodiment, the simulated surgical environment is a three-dimensional virtual reality environment representing a surgical environment that can be used to design and provide training for clinicians. The field of view of the simulated imaging system can look similar to the real-time field of view image 30. The simulated surgical environment can be a pre-recorded image of the patient's anatomy, a computer-generated diagram of the patient's anatomy, or a combination of both. In the simulator environment, a manual controller 18 can be used to manipulate virtual instruments in the field of view image. Similarly, the manual controller can be used to adjust the simulated field of view by moving the simulated imaging system (e.g., a virtual endoscope). In various embodiments, the simulated surgical environment can be a complete virtual reality environment in which the user S's movements (such as head movements or inputs to the input system 16) are bound to parameters of virtual components (e.g., virtual instruments, virtual imaging systems) in the virtual reality environment. In these fully virtual simulation environments, user movements are not bound to physical components in the physical surgical environment. In some embodiments, user input can be provided by tracking gaze or head movement without the need for a manual controller.

[0033] In alternative embodiments, the simulated surgical environment can be a physical, artificial surgical environment designed to mimic natural anatomy or provide specialized procedural training. In these alternative embodiments, user input can be mapped to physical components operating within the artificial environment. In other alternative embodiments, the simulated surgical environment can be a hybrid environment comprising both virtual reality elements and physical components.

[0034] Immersive display systems allow users to experience a live or simulated surgical environment as if they were actually in it. Head-mounted immersive display systems physically allow users a full range of translational and rotational movements, which may not correspond to physical constraints imposed by the kinematic and safety features of a remote operating system. When using immersive display systems in live or simulated surgical environments, the systems and methods described below can be implemented to constrain or enhance the user experience.

[0035] In traditional remote surgical systems, the user is positioned on a fixed operator console. The system enters engagement mode (also known as "head-in" mode) when sensors detect the user's head is positioned in the viewfinder and the user can visualize the surgical environment displayed in the viewfinder via images captured by the imaging system. The "follow mode" of a remote medical system is the system's operating state, in which the movement of the operator-input system (such as a manual controller) affects the movement of instruments positioned within the patient's anatomy. Typically, the system must first be in engagement mode before it can enter follow mode to perform a surgical procedure. To safely perform remote procedures using a conventional system, movement of instruments within the patient's anatomy can be paused when the operator looks away or otherwise exits engagement mode, indicating that the system is no longer visible to the operator.

[0036] When the display system is not in a fixed position within the operator's console, such as in the case of an immersive display system installed by the operator, the remote operating system can be constrained or enhanced to provide a virtual engagement mode that instructs the operator to visualize the surgical environment in the immersive display before entering instrument follow mode. Figure 4 This is a flowchart of a method 100 for providing virtual engagement and virtual exploration modes for entering and exiting a remote surgical system. The remote surgical system can be, for example, a remote surgical system 11 for a live surgical environment 10 or a simulator environment 40. Method 100 in... Figure 4 The diagram shows a set of operations or processes 102 to 114. Not all of the shown processes 102 to 114 are performed in all embodiments of method 100. Additionally, Figure 4 One or more processes not explicitly shown may be included before, after, between, or as part of processes 102 to 114. In some embodiments, one or more processes may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., a processor of a control system), may cause one or more processors to execute one or more processes.

[0037] At step 102, an image of the surgical environment visible within the imaging instrument's field of view is displayed. The image is displayed on a user-installed immersive display system. (Reference) Figure 5 An imaging device 204, having a current field of view 206 and a potential field of view 208, can visualize the surgical environment 202 (e.g., the internal anatomy of a living patient or a simulated surgical environment). The potential field of view can be defined by the range of motion of the imaging device 204. The environment 214 is outside the surgical environment 202. A user 201 (e.g., a surgeon S) wearing a head-mounted immersive display system 203 (e.g., display system 17) has an observation orientation 210 toward the current field of view 206. A visible image 212 is presented on the display system 203. In this orientation, the user can be in a virtual engagement mode. In virtual engagement mode, the user visualizes the current field of view. From virtual engagement mode, an instrument follow mode can be entered, in which instruments in the field of view can be moved under the control of the operator input system.

[0038] At process 104, movement of the user's head is detected. At process 106, it is determined whether the user's head (or a part of the head, such as the eyes) is within the boundary of the field of view that corresponds to and allows the user to view the field of view. The orientation of the user's head and the direction of observation can be evaluated to determine whether the current field of view is visible to the user.

[0039] At process 108, if the head movement has not yet exceeded the boundary and / or the user's head remains pointing towards the current field of view, the image on the display system remains the image of field of view 206. If the image data is three-dimensional (e.g., from a stereoscopic endoscope), the image of the field of view can be presented from a slightly different viewpoint corresponding to the detected movement of the user's head. Figure 6 As shown, user 201's head has moved, but the user's eye's viewing direction 210 points to the current field of view 206, and the user's head and eyes are within the boundary 205 corresponding to the field of view 206. Therefore, the visible image 212 is an image of the field of view 206 from a slightly offset viewpoint corresponding to the movement of the user's head.

[0040] At process 110, if the head movement has exceeded the boundary and / or the user's head is not pointing towards the current field of view, the image on the display system changes to notify the user that the field of view is no longer visible on the display system. For example, at optional process 110, the remote operating system can enter virtual exploration mode. In another example at optional process 112, the remote operating system can provide an alert. Figure 7 and Figure 8In an example embodiment, the user's head movement causes the user's eyes to now be outside boundary 205, and the viewing direction 210 is no longer pointing towards the current field of view 206. Optionally, a message 212b may be displayed on display system 203 to warn the user that they have moved outside the current field of view. This message may include text, graphics, or other images to warn the user or guide the user's head back into the field of view. The remote operating system may pause follow mode so that movement of the manual controller does not move surgical instruments in the surgical environment.

[0041] Optionally, the remote operating system can enter a virtual exploration mode, where image 212a on the display system is a virtual user interface, which may include images of the environment 214 outside the surgical environment (such as the external environment around the patient or another remote viewpoint separate from the viewpoint of the imaging instruments). Alternatively, the virtual user interface may allow the user to access, for example, interactive control menus for controlling features of the remote operating system, status information about the patient or components of the remote operating system, or a perspective image of the environment in front of the user's head.

[0042] At step 114, guidance can be provided to direct the user's gaze back into the boundary, making field of view 206 visible again. For example, guidance can take the form of: an audio recording providing verbal instructions for head movement, a sound that changes with head movement, a set of text instructions, a graphic movement diagram, a visual repositioning beacon, or a field of view scene that changes color or focus as the viewing direction moves toward the field of view. When the user's head aligns with the final field of view direction, the remote operating system can exit virtual exploration mode and return to follow mode or engage mode. Audio, visual, or tactical instructions can warn the user that they have exited virtual exploration mode.

[0043] Alternatively, the user can exit virtual exploration mode and re-enter engagement mode, follow mode, or imaging control mode at a newly calibrated head orientation. The user can provide an indication that they are working from a new orientation, or the system can receive an indication that a large translational movement has been detected (exceeding a threshold movement value and indicating that the user is working from a new orientation). Based on this indication, the system can establish a new set of orientation, orientation, and head boundaries for the user's new position. From this new position, the user's head orientation and viewing direction can be assessed to determine the user's current field of view. For example, if the user starts standing on one side of the patient and then moves to the opposite side of the table, the system will first enter virtual exploration mode when the user's head movement exceeds the boundaries. The system can exit virtual exploration mode and re-enter a mode such as follow mode or imaging control mode when on the opposite side of the table. When a new head orientation and orientation are detected, the imaging system view displayed to the user is based on the user's new orientation and orientation.

[0044] The remote operating system can be constrained to provide a virtual imaging control mode that requests user input before moving actual or simulated imaging instruments within the surgical environment. Figure 9 This is a flowchart 300 providing a method for entering and operating within a virtual imaging control mode of a remotely operated surgical system. The remotely operated surgical system can be, for example, a remotely operated surgical system 11 for a field surgical environment 10 or a simulator environment 40. Method 300 in... Figure 9 The diagram shows a set of operations or processes 302 to 312. Not all of the shown processes 302 to 312 are performed in all embodiments of method 300. Additionally, Figure 9 One or more processes not explicitly shown may be included before, after, between, or as part of processes 302 to 312. In some embodiments, one or more processes may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., a processor of a control system), may cause one or more processors to execute one or more processes.

[0045] At step 302, an image of the surgical environment, visible within the field of view of the imaging instrument, is displayed. The image is displayed on an immersive display system installed by the user. (Reference) Figure 10 User 201 may wish to change the current field of view 206 within the potential field of view 208 defined by the motion range of the imaging system 204.

[0046] At process 304, an imaging control input is detected. The imaging control input can be a clutching input, which switches the remote operating system to either engagement or follow mode. For example, the image control input could be an input at manual controller 18, such as pressing button 24. Other types of user input (including switches, gestures, pedal movements, or other intentional user inputs) can be suitable image control inputs for initiating the image control mode.

[0047] At step 306, in response to the detection of the imaging control input, the remote operating system enters the imaging control mode. In imaging control mode, the field of view of the imaging instrument can be changed by moving the imaging instrument within its range of motion. In imaging control mode, the surgical instruments are locked in place, and movements of the operator input system do not cause corresponding movements of the surgical instruments. In imaging control mode, the imaging instrument can move in response to user movements (such as movement of the user's head or movement of the operator input system).

[0048] At process 308, movement of the user's head is detected. Alternatively, another type of user input can be detected (such as movement of the manual controller 18). For example, refer to... Figure 10 It can detect the movement 211 of the head 201. The viewing direction 210 changes as the user's head moves.

[0049] At process 310, the field of view of the imaging device changes from field of view 206 to field of view 207. Field of view 207 is the field of view of the imaging system 204 after it has moved in response to the detected head movement 211.

[0050] At step 312, an image of the surgical environment from the changed field of view is displayed on the display system. For example, refer to... Figure 10 The image 215a of the field of view 206 on the display system 203 is changed to the image 215b of the field of view 207.

[0051] When the remote operating system is in imaging control mode, movement of the manual controller 18 can cause movement of the instrument 204 within its range of motion, and thus alter the field of view of the instrument 204. Alternatively, movement of the manual controller 18 can cause the ghost image of the surgical instruments in the field of view to move accordingly.

[0052] If the detected movement of the user's head does not correspond to movement within the range of motion of the imaging device, the remote operating system can enter virtual exploration mode and provide images, user interfaces, or information of other environments as previously described. As mentioned earlier, virtual exploration mode can be exited by matching the user's head with the final orientation and orientation of the imaging system.

[0053] When additional imaging control input is detected (e.g., pressing a button on controller 18), the imaging control mode can be exited. In various alternative embodiments, the imaging control mode can request continuous activation of imaging control input (e.g., continuous pressing of a button on controller 18), such that interrupting the control input switches the remote operating system from imaging control mode to engaged mode or follow mode.

[0054] The remote operating system can be constrained to provide a virtual manipulator arm mode that indicates when a user's arm movement is outside the range of motion corresponding to the permissible range of motion of the manipulator arm (e.g., arm 12). Figure 11 This is a flowchart 400 providing a method for entering and operating within a virtual manipulator arm mode of a remote operating system. The remote surgical system can be, for example, a remote surgical system 11 for a live surgical environment 10 or a simulator environment 40. Method 400 in... Figure 11The diagram shows a set of operations or processes 402 to 408. Not all of the shown processes 402 to 408 are performed in all embodiments of method 400. Additionally, Figure 11 One or more processes not explicitly shown may be included before, after, between, or as part of processes 402 to 408. In some embodiments, one or more processes may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., a processor of a control system), may cause one or more processors to execute one or more processes.

[0055] At process 402, an image of the surgical environment is displayed. The image is displayed on an immersive display system installed by the user. The surgical environment image may include images of surgical instruments and / or portions of the manipulator arm to which the instruments are coupled.

[0056] At process 404, a manipulator arm input motion is detected. For example, movement of the user's arm or hand can be detected as a manipulator arm input motion. At process 406, the remote operating system determines whether the manipulator arm input motion corresponds to a movement outside the range of motion of the physical manipulator arm (e.g., arm 12). If the input motion corresponds to a movement outside the range of motion of the manipulator arm, the remote operating system enters virtual manipulator arm mode. In virtual manipulator arm mode, movement of the user's arm does not cause a corresponding movement of the physical manipulator arm, and a ghost image of the surgical instrument or manipulator arm is displayed on the display system, where the movement corresponds to the manipulator arm input motion. To exit virtual manipulator arm mode, the user can be guided back to the last permissible orientation of the surgical instrument or manipulator arm. The guidance to return to the matching orientation can be similar to the guidance provided to the user to exit virtual exploration mode, as previously described.

[0057] While certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that these embodiments are merely illustrative and not limiting. Furthermore, it should be understood that various other modifications will arise in those skilled in the art, and therefore the embodiments of the invention are not limited to the specific structures and arrangements shown and described.

[0058] Furthermore, numerous specific details have been set forth in the detailed description of the embodiments of the present invention to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that the embodiments of this disclosure can be practiced without these specific details. In some cases, well-known methods, procedures, and components have not been described in detail to avoid unnecessarily obscuring aspects of the embodiments of the present invention.

Claims

1. A medical system comprising: processor; and A memory having computer-readable instructions stored thereon, which, when executed by the processor, cause the system to: An image of the surgical environment from the imaging field of view is displayed on a display system, which is configured to move with the user's head; Enter the imaging control mode of the surgical control system; Detect the movement of the user's head; When the surgical control system is in the imaging control mode and responds to the movement of the user's head, it displays an image of the surgical environment from a changed imaging field of view, the changed imaging field of view corresponding to the detected movement of the user's head; and When the virtual exploration mode of the surgical control system is enabled, an interactive virtual user interface is displayed when the surgical control system is in the virtual exploration mode and responds to the movement of the user's head.

2. The medical system of claim 1, wherein the computer-readable instructions, when executed by the processor, further cause the system to: Determine whether the movement of the user's head is within the boundary corresponding to the range of motion of the imaging instrument having the imaging field of view; and If the user's head movement is not within the boundary corresponding to the range of motion of the imaging device, then the virtual exploration mode is enabled.

3. The medical system of claim 2, wherein the computer-readable instructions, when executed by the processor, further cause the system to: When the surgical control system is in the virtual exploration mode and responds to the movement of the user's head, it displays an image of the environment outside the patient's anatomical structures.

4. The medical system of claim 2, wherein the computer-readable instructions, when executed by the processor, further cause the system to: Instructions are provided for exiting the virtual exploration mode, wherein the instructions guide the user's head movement in a direction that matches the imaging field of view.

5. A medical system comprising: A display that is configured to be mounted on the user's head; Imaging devices; and The control system is configured to: Generate an image of the surgical environment from the viewpoint of the imaging device, the image including the field of view; The image of the surgical environment is displayed on the monitor; Detect the movement of the user's head; Determine whether the movement of the user's head is within the boundary corresponding to the field of view; and If the movement of the user's head is determined to be within the boundary, then: Generate an image of the changed surgical environment from a changed viewpoint of the imaging device, the changed image including a changed field of view corresponding to the detected movement of the user's head; and An image of the changes in the surgical environment is displayed on the monitor.

6. The medical system of claim 5, wherein the control system is further configured to: If the user's head movement exceeds the boundary, the virtual exploration mode of the medical system is activated.

7. The medical system of claim 6, wherein the control system is further configured to: When the medical system is in the virtual exploration mode and responds to the movement of the user's head, it displays an image of the surgical environment on the display from a remote viewpoint separate from the viewpoint of the imaging device.

8. The medical system of claim 6, wherein the control system is further configured to: When the medical system is in the virtual exploration mode and responds to the movement of the user's head, it displays an interactive virtual user interface.

9. The medical system of claim 6, wherein the control system is further configured to: Instructions are provided for exiting the virtual exploration mode, wherein the instructions guide the user's head movement in a direction that matches the viewpoint of the imaging device.

10. The medical system of claim 9, wherein the control system is further configured to: provide the user with an indication that the medical system has exited the virtual exploration mode.

11. The medical system of claim 6, wherein the control system is further configured to: display an image of the environment outside the patient's anatomy when the medical system is in the virtual exploration mode and in response to the movement of the user's head.

12. The medical system of claim 6, wherein the control system is further configured to: Provide instructions for exiting the virtual exploration mode; and When the medical system exits the virtual exploration mode, it enters the imaging control mode, the integration mode, or the follow mode.

13. The medical system of claim 6, wherein the control system is further configured to: provide guidance for exiting the virtual exploration mode; and When a user input is received instructing the user's head orientation to match the second changed viewpoint of the imaging device, the virtual exploration mode is exited.

Images