System and method for remote guidance in a robot assisted medical system
By generating and projecting the stereoscopic endoscopic image data of the medical field, the problem of the lack of three-dimensional visual information in remote operation robot-assisted medical system is solved, and accurate three-dimensional visual guidance is achieved in the remote environment, which is improved. Training and execution quality.
Patent Information
- Application Number
- CN202380028198.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-02-02
- Filing Date
- 2023-01-25
- Publication Date
- 2024-11-01
AI Technical Summary
The existing remote-operated robot-assisted medical system lacks complete three-dimensional visual information when providing remote guidance, which makes the instructor unable to generate visual guidance with three-dimensional accuracy, limiting the effectiveness of training, guidance or recommendations.
By generating stereoscopic endoscopic image data of the medical field, defining the medical field volume, and projecting its 3D representation into an extended reality device in the remote volume, generating visual guidance, enhancing the stereoscopic image, and mapping it to the medical field In the field volume, an enhanced image display is provided.
It achieves the provision of complete three-dimensional experience and accurate visual guidance in remote environments, and improves the training and execution quality of medical procedures such as surgery.
Smart Images

Figure CN118891017A_ABST
Abstract
Description
[0001] Cross-referenced applications
[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 305,915, filed on February 2, 2022, and entitled “Systems and Methods for Remote Mentoring in a Robot Assisted Medical System,” which is incorporated herein by reference in its entirety. Technical Field
[0003] The present disclosure relates to systems and methods for remote guidance in which an extended reality user interface device is used to display endoscopic image information and generate visual guidance. Background Art
[0004] A teleoperated robot or robotic-assisted system includes a manipulation component that can be remotely controlled from a primary interface system. Systems and methods for training, instructing, or advising an operator of a primary interface system may be limited if the visual information provided to the instructor does not provide complete three-dimensional information and the instructor guidance may not be generated with three-dimensional accuracy. Therefore, it would be advantageous to provide improved methods and systems for providing remote guidance in which the instructor has a complete three-dimensional experience of the surgical environment and is able to provide visual guidance with three-dimensional accuracy. Summary of the invention
[0005] Embodiments of the invention are best summarized by the claims appended to the description.
[0006] Consistent with some embodiments, a non-transitory machine-readable medium may store instructions that, when executed by one or more processors, cause the one or more processors to generate stereoscopic endoscopic image data of a medical field and define a medical field volume based on the stereoscopic endoscopic image data. The processor may also project a 3D representation of the medical field volume to an extended reality display device in a remote volume. The 3D representation may include a stereoscopic image and a 3D scene generated from the stereoscopic endoscopic image data. The processor may also generate visual guidance in the remote volume to enhance the stereoscopic endoscopic image, map the visual guidance from the remote volume to the medical field volume to generate an enhanced image of the medical field volume, and project the enhanced image of the medical field volume to a display device viewed by an operator of an instrument in the medical field.
[0007] It should be understood that the above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 is an illustration of a teleguidance system according to some embodiments.
[0009] Figure 2A is a perspective view of a teleoperation assembly according to some embodiments.
[0010] Figure 2B A patient environment is shown in accordance with some embodiments.
[0011] Figure 3 Medical field volumes are shown in accordance with some embodiments.
[0012] Figure 4 A primary interface system is shown according to some embodiments.
[0013] Figure 5 An environment of a secondary interface system according to some embodiments is shown.
[0014] Figure 6 is a flow chart illustrating a method for providing guidance to an operator (eg, surgeon S) of a primary interface system according to some embodiments.
[0015] Embodiments of the present disclosure and their advantages are best understood by referring to the following detailed description. It should be understood that like reference numerals are used to identify like elements shown in one or more of the accompanying drawings, wherein the illustrations therein are for the purpose of illustrating embodiments of the present disclosure, rather than for the purpose of limiting the present disclosure. DETAILED DESCRIPTION
[0016] Figure 1A teleguidance system 100 is shown, which is used to guide, train, evaluate, assist, guide, advise, or otherwise monitor an operator during a teleoperation procedure (including a live patient procedure, a training procedure, a simulation procedure, or other guided procedure). The teleguidance medical system 100 can be used for medical procedures including, for example, diagnostic, therapeutic, or surgical procedures. In one or more embodiments, the system 100 can be a teleoperated or robotic-assisted medical system under the teleoperation control of a surgeon S. In an alternative embodiment, the system 100 can be under partial control of a computer programmed to perform a medical procedure or subroutine. In yet other alternative embodiments, the system 100 can be a fully automated medical system that is under the full control of a computer that is programmed to perform a medical procedure or subroutine using the system 100. An example of a system 100 that can be used to implement the systems and techniques described in the present disclosure is a da Surgical systems.
[0017] System 100 includes a manipulator assembly 102, a primary interface system 104, a secondary interface system 106, a primary control system 108, and a secondary control system 109. Manipulator assembly 102 may be mounted to or positioned proximate to an operating table O where a patient P is positioned. Assembly 102 may be referred to as a patient side cart, a surgical cart, a surgical robot, a manipulator system, and / or a teleoperated arm cart.
[0018] Manipulator assembly 102 may be located in environment 112. Primary interface system 104 may be located in environment 114. Secondary interface system may be located in environment 116. Primary control system 108 may be located in environment 118. Secondary control system 109 may be located in environment 119. In some embodiments, environment 112 may be a medical environment such as an operating room. In other embodiments, the medical environment may be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medical procedures or medical training procedures may be performed. Environment 114 may be in environment 112, in another room in a facility shared with environment 112, or in another geographic location. Environment 116 may be in environment 112 or environment 114, in another room in a facility shared with environment 112 or 114, or in another geographic location. Environment 118 may be in environment 112 or 114; in another room in a facility shared with environment 112 or 114; or in another geographic location. Environment 119 may be in environment 112 or 116; in another room in a facility shared with environment 112 or 116; or in another geographic location. In some embodiments, primary and secondary control systems 108, 109 may be a single control system located near the primary interface system, near the secondary interface system, or at a location remote from the primary and secondary interface systems. In some embodiments, control system components used to enable communication, control, and image data transfer between primary interface system 104, secondary interface system 106, and manipulator assembly 102 may be distributed over one or more locations.
[0019] Further references Figure 2A, one or more medical device systems 124 and endoscopic imaging systems 125 are operably coupled to assembly 102. Medical device system 124 may include one or more medical devices. In embodiments where medical device system 124 includes multiple medical devices, the multiple medical devices may include multiple identical medical devices and / or multiple different medical devices. Similarly, endoscopic imaging system 125 may include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes may include multiple identical endoscopes and / or multiple different endoscopes. Assembly 102 supports and manipulates medical device system 124 while surgeon S views the surgical site through primary interface system 104. Endoscopic image data of the surgical site may be obtained by endoscopic imaging system 125, which may be manipulated by assembly 102. The number of medical device systems 124 used at one time will typically depend on the diagnostic or surgical procedure to be performed and the space constraints within the operating room (among other factors). The assembly 102 may include one or more non-servo controlled links (e.g., one or more links that can be manually positioned and locked in place, commonly referred to as manipulator support structures) and a kinematic structure of the manipulator. The assembly 102 includes a plurality of motors that drive inputs on the medical device system 124. In an embodiment, these motors move in response to commands from a control system (e.g., the main control system 108). The motors include drive systems that, when coupled to the medical device system 124, can advance the medical device into a natural or surgically created anatomical hole. Other motorized drive systems can move the distal end of the medical device in multiple degrees of freedom, which may include three linear motions (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational motions (e.g., rotation around the X, Y, Z Cartesian axes). Additionally, the motor can be used to actuate the articulated end effector of the medical device to grasp tissue in the jaws of a biopsy device, etc. The medical device of the medical device system 124 may include an end effector having a single working member (e.g., a scalpel, a blunt blade, an optical fiber, or an electrode). Other end effectors may include, for example, forceps, graspers, scissors, or clip appliers.
[0020] The illustrated assembly 102 provides for manipulation of three medical instruments 124a, 124b, and 124c and an endoscopic imaging device 125 (e.g., a stereo endoscope for capturing images of the site of the procedure). The imaging device 125 can transmit signals to the main control system 108 via a cable 127. The imaging device 125 and the medical instruments 124a-124c can be positioned and manipulated through an incision in the patient's body so that a kinematic telecenter is maintained at the incision to minimize the size of the incision. When the distal ends of the medical instruments 124a-124c are positioned within the field of view of the imaging device 125, an image of the surgical site can include an image of the distal ends of the medical instruments 124a-124c. The imaging device 125 and the medical instruments 124a-124c can each be a therapeutic, diagnostic, or imaging instrument.
[0021] The assembly 102 includes a drivable base 126. The drivable base 126 is connected to a telescopic column 128 that allows the height of the arm 130 to be adjusted. The arm 130 may include a swivel joint 132 that both rotates and moves up and down. Each arm 130 may be connected to an orientation platform 134 that is capable of 360 degree rotation. The assembly 102 may also include a telescopic horizontal boom 136 for moving the orientation platform 134 in a horizontal direction.
[0022] In this example, each arm 130 is connected to a manipulator arm 138. The manipulator arm 138 can be directly connected to the medical instrument 124a-124c. The manipulator arm 138 can be teleoperable. In some examples, the arms 138 connected to the orientation platform 134 may not be teleoperable. Instead, the arms 138 can be positioned as needed before the surgeon S begins to operate using the teleoperated components. Throughout the surgical procedure, medical instruments may be removed and replaced with other instruments, so that the association of instruments with arms may change during the procedure.
[0023] The endoscopic imaging system 125 can be provided in various configurations including rigid or flexible endoscopes. A rigid endoscope includes a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end of the endoscope. A flexible endoscope uses one or more flexible optical fibers to transmit images. An endoscope based on digital images has a "chip on the tip" design in which a distal digital sensor (such as one or more charge coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices) stores image data. An endoscopic imaging system can provide a viewer with a two-dimensional or three-dimensional image. A two-dimensional image may provide limited depth perception. A three-dimensional stereoscopic endoscopic image may provide a viewer with a more accurate depth perception. A stereoscopic endoscopic instrument employs a stereo camera to capture stereoscopic images of a patient's anatomical structure. An endoscopic instrument may be a fully sterilizable assembly in which the endoscope cable, handle, and shaft are all rigidly coupled and sealed.
[0024] Figure 2B A patient environment 140 is shown with the distal end portion of an endoscopic imaging system 125 and the distal end portions of instrument systems 124a and 124c inserted into an anatomical structure 142 of a patient P. The area within the anatomical structure 142 visible using the imaging system 125 is an imaging field 144, which includes patient tissue 145 and the distal ends of the instrument systems 124a, 124c. The interaction between the instrument systems 124a, 124c and the tissue 145 in the imaging field 144 during a medical procedure can be captured by stereoscopic image data generated by the imaging system 125.
[0025] Figure 3 The reference frame (X I , Y I , Z I ) of the imaging field 144. The medical field volume 146 can be generated based on the stereoscopic image data of the imaging field 144. The medical field volume 146 can include images of the instrument systems 124a, 124c and tissue 145 from the imaging field 144. The medical field volume 146 can also include virtual user interface elements 148, such as user interactive menus, virtual instrument markers, off-screen instrument indicators, or other graphical, numerical, and / or textual elements that provide information to the surgeon S during the medical procedure. The user interface elements 148 can be overlaid on the stereoscopic image of the medical field volume 146, or can be incorporated at selected depth locations within the medical field volume 146.
[0026] refer to Figure 4The primary interface system 104 allows the surgeon S or other types of clinicians to view images of the surgical site or images representing the surgical site and control the operation of the medical instrument system 124 and / or the endoscopic imaging system 125. The primary interface system 104 can be located at the surgeon's console 150. In one or more embodiments, the primary interface system 104 can be referred to as an operator interface system, an operator input system, a user control system, a user input system, etc. The primary interface system 104 can include a primary display system 156 that displays image data for performing a teleoperation procedure, including endoscopic images from within the patient's anatomy, guidance information from the secondary interface system, patient information, and procedure planning information. The primary interface system 104 includes a left eye display 152 and a right eye display 154 for presenting a coordinated stereoscopic view of the surgical environment capable of depth perception to the surgeon S. The left eye display 152 and the right eye display 154 can be components of a display system 156. In other embodiments, the display system 156 can include one or more other types of displays. In some examples, display system 156 can display stereoscopic image 147 (which includes user interface elements 148) of medical field volume 146. In some examples, stereoscopic image 147 can be enhanced with visual guide 170 generated by secondary interface system 106, as described below.
[0027] The primary interface system 104 may also include an audio system that allows the surgeon S to communicate with personnel in the patient environment 112 and / or personnel in the secondary interface environment 116 (eg, a mentor M).
[0028] The primary interface system 104 may also include one or more primary input devices 158, which in turn cause the assembly 102 to manipulate one or more instruments of the endoscopic imaging system 125 and / or the medical instrument system 124. The control device(s) may include one or more of any number of various input devices, such as hand-held manipulation devices, joysticks, trackballs, data gloves, trigger guns, foot pedals, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and other types of input devices.
[0029] Each primary input device 158 may be movable within the environment 114 with multiple degrees of freedom, typically with six degrees of freedom (three rotational degrees of freedom and three translational degrees of freedom). This allows the primary input device 158 to be moved to any position and any orientation within its range of motion. The control device(s) may include one or more of any number of various input devices, such as hand-held manipulation devices, joysticks, trackballs, data gloves, trigger guns, foot pedals, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and other types of input devices.
[0030] The kinematic ranges and kinematic constraints of motion associated with the medical instrument system 124, the imaging system 125, and the components can be provided by the primary input device 158. The primary input device 158 can provide the same Cartesian degrees of freedom as their associated instruments to provide telepresence to the surgeon S, or to provide a sense that the primary input device 158 is integral with the instrument, so that the surgeon has a strong sense of direct control of the instrument. Therefore, the degrees of freedom of each primary input device 158 are mapped to the degrees of freedom of the associated instrument of each primary input device 158 (e.g., one or more instruments of the endoscopic imaging system 125 and / or the medical instrument system 124). To this end, position, force, and tactile feedback sensors (not shown) can be used to transmit position, force, and tactile sensations from medical instruments (e.g., medical instruments 124a-124c or imaging devices 125) back to the surgeon's hands through the primary input device 158. Additionally, the arrangement of the medical instruments 124a-124c can be mapped to the arrangement of the surgeon's hands and the view in the surgeon's eyes, so that the surgeon has a strong sense of direct control of the instrument. The input control device 160 is a foot pedal that receives input from a user's foot.
[0031] Reference again Figure 1 , the system 100 also includes a primary control system 108. The primary control system 108 includes at least one memory 120 and at least one processor 122 for enabling communication, control, and image data transmission between the medical device system 124, the primary interface system 104, the secondary interface system 106, and other auxiliary systems, which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, irrigation systems, and / or suction systems. Although depicted as being external to the component 102 and the primary interface 104, in some embodiments, the primary control system 108 may be completely contained within the component 102 or the primary interface 104. The primary control system 108 also includes programmed instructions (e.g., stored on a non-transitory computer-readable medium) to implement some or all of the methods described according to the aspects disclosed herein. Although the primary control system 108 is Figure 1Although shown as a single block in the simplified schematic diagram of , the main control system 104 may include two or more data processing circuits, with a portion of the processing optionally being performed on or near the component 102, another portion of the processing being performed at the main interface system 104, and so on. Any of a variety of centralized or distributed data processing architectures may be employed in the main control system 108. Similarly, the programmed instructions may be implemented as a plurality of separate programs or subroutines, or they may be integrated into a variety of other aspects of the systems described herein (including teleoperation systems). In one embodiment, the control system 108 may support any of a variety of wired or wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry).
[0032] The system 100 also includes a secondary control system 109. The secondary control system 109 includes at least one memory 121 and at least one processor 123 for implementing control between the medical device system 124, the secondary interface system 106, the primary interface system 104 and / or other auxiliary systems, which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, irrigation systems and / or suction systems. In some embodiments, the secondary control system 109 may be a component of the secondary interface system 106. The secondary control system 109 also includes programmed instructions (e.g., stored on a non-transitory computer readable medium) to implement some or all of the methods described according to the aspects disclosed herein. Although the secondary control system 109 is Figure 1 104, another portion of the processing is performed at the secondary interface system 106, and so on. Any of a variety of centralized or distributed data processing architectures may be employed in the control system 109. Similarly, the programmed instructions may be implemented as a plurality of separate programs or subroutines, or they may be integrated into a variety of other aspects of the system described herein (including the teleoperation system). In one embodiment, the control system 109 may support any of a variety of wired or wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry). In some embodiments, the primary control system and the secondary control system may be a single control system located near the primary interface system, near the secondary interface system, or at a location remote from the primary and secondary interface systems. In some embodiments, the control system components used to implement communication, control, and image data transmission between the primary interface system 104, the secondary interface system 106, and the component 102 may be distributed at one or more locations.
[0033] The secondary interface system 106 (also referred to as the instructor interface system) allows an instructor, proctor, instructor, consultant, or other user M at the secondary interface system 106 to receive visual and auditory information and generate guidance for the surgeon S at the primary interface system 104. The secondary interface system 106 can be operated by the instructor M to guide, train, assist, guide, or otherwise advise the operator of the primary interface system 104 when performing a patient medical procedure, a simulation procedure, a training procedure, or other operations performed by the surgeon S via the primary interface system 104. The secondary interface system 106 may include a display system and an input system. In some examples, the interface system 106 may include an extended reality system. For example, the extended reality system may include a wearable interface device that includes a head-mounted extended reality display device, an auditory device, and a sensor system that can track hand movements, eye gaze, head movements, voice, or other instructor actions related to information presented by the display device or the auditory device. In some examples, the auditory device may be integrated into the head-mounted extended reality display device, but in other examples, it may be a separate headset or speaker. In some examples, the sensor system can be integrated into a head-mounted extended reality display device, but in other examples, the sensor system can be part of a handheld control device or an optical sensor system that uses a camera to track the user's movements. The extended reality system can include a mixed reality device, an augmented reality device, or any combination thereof, which presents a combination of virtual and real environments. In some examples, an augmented reality device can provide virtual information and / or virtual objects overlaid on a real-world environment. In some examples, a mixed reality device can provide virtual objects and the ability to interact with these objects in a real-world environment. In some examples, a mixed reality interface device can include, for example, Microsoft Hololens or Oculus Quest. In some examples, a secondary interface system can include a three-dimensional display device in the form of a portable or desktop device.
[0034] refer to Figure 5 , the instructor M can be located in a reference frame (X M , Y M , Z M ) in the environment 116. The remote volume 201 within the environment 116 can be determined based on the minimum, maximum, and comfortable ranges of the instructor M's head movement, eye gaze, and hand extension. The remote volume 201 can have a remote reference frame (X R , Y R , Z R). The remote volume 201 may be egocentric about the instructor M and may physically move or rotate as the instructor moves. In some examples, the remote volume 201 may be the entire environment 116. Through the extended reality device 106, the instructor M can view the three-dimensional representation 204 of the medical field volume 146. Through the extended reality device 106, the instructor M can also have a field of view 200 in the remote volume 201. The field of view 200 can include objects 202 in the remote volume 201 (e.g., furniture, other people, walls, and floors). The field of view 200 can be visible around, outside, in front of, and through the three-dimensional representation 204 of the medical field volume.
[0035] The three-dimensional representation 204 of the medical field volume 146 may include a virtual user interface element 148, a stereoscopic image 147 of the medical field volume 146, and one or more three-dimensional scenes 206 representing the medical field. The scene 206 may be generated at least in part based on the stereoscopic image data of the imaging field 144. Additionally or alternatively, the scene 206 may be generated based on a separate depth measurement camera or other depth assessment system. The stereoscopic image 147 may be a three-dimensional image from the viewpoint of the imaging system 125, but the three-dimensional scene 206 is three-dimensional from any viewpoint. The three-dimensional scene may be selectively constructed for discrete regions of the stereoscopic image (e.g., the region where the instructor's gaze intersects the medical field volume, the region around the distal end of the instrument visible in the medical field volume, the known or interesting landmarks determined based on the type of medical procedure, and / or the region where the surgeon S is viewing the medical field volume). The three-dimensional scene 206 may be generated using intrinsic and extrinsic parameters of the imaging system and by determining object depth and segmenting anatomical objects. For example, the object depth may be determined by matching corresponding pixels in the right and left stereoscopic images based on an approximation based on mutual information and a global smoothness constraint. The disparity between matching points provides a measure of depth. Anatomical object segmentation can use graph analysis techniques to distinguish tissue types (e.g., connective tissue and organs). Construction of a three-dimensional scene can be computationally complex, expensive, and time consuming, and therefore may be limited to discrete regions. In an alternative example, a three-dimensional scene with object depth determination and anatomical segmentation can be calculated for each pixel in the entire stereo image 147 to generate a three-dimensional scene of the entire medical field volume 146.
[0036] The extended reality device 106 may also be used to generate visual guides 170 in the remote volume 201 to augment the stereoscopic image 147 of the medical field volume 146 displayed on the display system 156 of the primary interface system 104. The generated visual guides may be mapped from the remote volume 201 to the medical field volume 146 to generate an augmented stereoscopic image 147 of the medical field volume 146 displayed on the display system 156. In some examples, the visual guides may be located near relevant structures at a selected depth within the three-dimensional scene 206. For example, the sensor system of the extended reality device 106 may track the freehand movements of the instructor in the remote volume 201 to create visual guides in the form of articulated ghost hand images 172 that track the movements of the instructor's hands 174 at different depths within the scene 206. The user's hands may make gestures, point, or provide other indications that provide guidance to the surgeon S during the medical procedure. The visual guides in the form of ghost hand images 172 may be placed at the tracking depth within the medical field volume 146. The visual guidance provided by the ghost hand image 172 can be visible in the stereoscopic image of the medical field volume displayed on the display system 156 of the main interface system 104. The ghost hand image 172 may include joint positions and orientations. In some examples, more than twenty joint positions and / or orientations of each hand may be used to generate the ghost hand image 172. In other examples, the visual guidance may be generated by a physical expression felt by the instructor, which includes any combination of tracked eye gaze, head movement, body movement, speech and / or sensors or input channels. In some examples, tracked sensor information or other inputs may be used to generate visual guidance 170 in the form of arrows, pointers, ghost tools, gaze marks, and text, numbers, or graphical annotations. In some examples, a three-dimensional teleportation may be generated based on the motion mapped from the instructor's palm. Gestures may be used to start and terminate the generation of teleportation. In some examples, the visual guides 170, 172 may be visible in a three-dimensional representation 204 of the medical field volume 146 visible to the instructor M, so that the instructor can view the generated visual guides, thereby providing closed-loop feedback to the instructor. The visual guides 170, 172 may be mapped in the remote volume 201 with corresponding three-dimensional parameters (e.g., depth, position, orientation, and velocity) in the medical field volume 146. The mapping may include scaling parameters to generate the visual guides. The mapping may change adaptively with movement of the endoscope 125 or movement of the instructor.
[0037] In some examples, the visual guide may have spatiotemporal properties that affect its behavior or display life. For example, when the surgeon S moves the endoscope 125, the arrow placed by the instructor M (e.g., an arrow generated in the direction from the instructor's palm toward the instructor's fingertips) may disappear because the guidance provided by the arrow may not be applicable to the changed endoscope's view. In some examples, if the arrow is marked to an anatomical landmark or instrument in the imaging field 144, the arrow may persist even if the surgeon moves the endoscope 125. The arrow can be deleted by the surgeon S's affirmative action. In some examples, visual guidance can only be generated when the instructor's attention is on certain areas or types of real or virtual objects in the three-dimensional representation 204 of the medical field volume 146. In some examples, guidance is generated only when the instructor's attention (e.g., gaze intersects with the scene 206) is on the three-dimensional scene 206. If the instructor's attention is elsewhere and the instructor moves his hand, guidance may not be generated, and therefore the surgeon S will not see the representation of the movement. This may prevent unintentional guidance generation. In some examples, the guidance can be in the form of a virtual three-dimensional anatomical model that can be manipulated by the instructor in the three-dimensional representation 204 of the medical field volume 146. The surgeon S can view the manipulated model on the display 156. In some examples, the instructor M can manipulate the user interface elements 148 to demonstrate various techniques, menu selections, or other virtual object manipulations. In some examples, the instructor M can be prevented from controlling the instruments 124a, 124c from the remote environment 116, but in other examples, the instructor can control the instruments in the patient environment 112 from the remote environment 116. In some examples, the instructor M in the remote environment 116 can also or alternatively generate audio or tactile guidance that can be presented to the surgeon S in the surgeon environment 114.
[0038] In some examples, a real-time model of the manipulator assembly 102 may be projected to the extended reality device 106 for viewing either alone or in the context of the three-dimensional representation 204 with the medical field volume 146. The model manipulator assembly may be generated based on measured kinematic pose data, sensor data, tracked light devices, detected arrows, etc. In some examples, the model manipulator assembly may be generated based on camera images from the environment 112. By viewing the model of the manipulator assembly, the instructor M may be able to view manipulator arm collisions, warning indicators on the manipulator assembly, or other information about the manipulator assembly that may inform the instructor's understanding of conditions in the environment 112. In some examples, the model of the manipulator assembly may be fused with the augmented medical field volume 146 and displayed on the extended reality device 106 so that the instructor M can see the instrument extending from the patient and coupled to the manipulator assembly.
[0039] In some examples, a real-time model of the operator console 104 may be projected to the extended reality device 106. The model operator console may be generated based on measured kinematic pose data, sensor data, tracked light devices, detected arrows, etc. In some examples, the model operator console may be generated based on camera images from the environment 114. By viewing the model of the operator console, the instructor M may be able to view the position, orientation, collision of input devices, or other information about the operator console, which may inform the instructor's understanding of conditions in the environment 114.
[0040] In some examples, the medical field volume may be a mosaic generated from multiple images taken of the patient's anatomy, and the instructor M may generate a visual guide in an area of the mosaic that is outside the view of the medical field volume currently being viewed by the surgeon S. The surgeon S may access the visual guide by moving the imaging instrument 125 to view the area of the medical field volume where the visual guide has been placed.
[0041] Figure 6 is a flowchart showing a method 300 for providing guidance to an operator (e.g., surgeon S) of a primary interface system according to some embodiments. The methods described herein are shown as a set of operations or processes and continue to be described with reference to the previous figures. Not all of the processes shown can be performed in all embodiments of the method. Additionally, one or more processes not explicitly shown may be included before, after, between, or as part of the processes shown. In some embodiments, one or more processes may be implemented at least in part in the form of executable code stored on a non-temporary, tangible, machine-readable medium, which, when run by one or more processors (e.g., a processor of a control system), may cause one or more processors to perform one or more processes. In one or more embodiments, these processes may be performed by any one of the control systems 108, 109.
[0042] Method 300 can be performed using a teleguidance system (e.g., teleguidance system 100) that allows a mentor M at a secondary interface system (e.g., secondary interface system 14, 106) to provide visual guidance to an operator of a primary interface system. In process 302, stereoscopic endoscopic image data of a medical field in a patient's anatomy can be generated. For example, the stereoscopic image data can be generated by an imaging system 125 of an imaging field 144 in an anatomical structure 142.
[0043] At process 304, a medical field volume may be defined based on the stereoscopic image data. For example, the medical field may be generated based on the stereoscopic image data of the imaging field 144. The medical field volume 146 may include images of the instrument systems 124a, 124c and tissue 145 from the imaging field 144. The medical field volume 146 may also include virtual user interface elements 148 (e.g., user interactive menus, virtual instrument markers, off-screen instrument indicators, or other graphical, numerical, and / or textual elements that provide information to the surgeon S during the medical procedure).
[0044] In process 306, a three-dimensional representation of the medical field volume can be projected to an extended reality user interface device. The three-dimensional representation can include a stereoscopic image generated from the stereoscopic endoscopic image data and a three-dimensional scene generated from the stereoscopic endoscopic image data. For example, a three-dimensional representation 204 of the medical field volume 146 can be projected to an extended reality device 106 in a remote volume 201, allowing the instructor M to view the imaging field 144 and any virtual interface elements 148. The three-dimensional representation 204 of the medical field volume 146 can include virtual user interface elements 148, a stereoscopic image 147 of the medical field volume 146, and one or more three-dimensional scenes 206 representing the medical field. In some examples, the instructor M can also view the field of view 200 in the remote volume 201 through the extended reality device 106. The field of view 200 can include objects 202 (e.g., furniture, other people, walls, and floors) in the remote volume 201. The field of view 200 can be visible around, outside, in front of, and through the three-dimensional representation 204 of the medical field volume. In some examples, the instructor may not be able to view field of view 200 in remote volume 201 through extended reality device 106 .
[0045] At process 308 , a visual guide may be generated in the remote volume. For example, the extended reality device 106 may also be used to generate a visual guide 170 in the remote volume 201 to augment the stereoscopic image 147 of the medical field volume 146 displayed on the display system 156 of the primary interface system 104 .
[0046] In process 310, visual guidance can be mapped from the remote volume to the medical field volume. For example, the generated visual guidance 170, 172 can be mapped from the remote volume 201 to the medical field volume 146. The extended reality device 106 can track the instructor's freehand movements, eye gaze, head movement, body movement, speech and / or any combination of sensors or input channels in the remote volume 201 to create visual guidance in the form of articulated ghost hand images, arrows, pointers, ghost tools, gaze marks and text, numbers or graphical annotations. In some examples, the visual guidance 170, 172 can be visible in the three-dimensional representation 204 of the medical field volume 146 visible to the instructor M, so that the instructor can view the generated visual guidance, thereby providing closed-loop feedback to the instructor. The visual guidance 170, 172 can be mapped in the remote volume 201 with corresponding parameters (e.g., depth, position, orientation and speed) in the medical field volume 146. The mapping may include scaling parameters to generate visual guidance. The mapping can change adaptively with the movement of the endoscope 125 or the movement of the instructor.
[0047] At process 312, the enhanced image of the medical field volume may be projected to an operator display device. For example, the enhanced stereoscopic image 147 of the medical field volume 146 is projected on the display system 156 of the primary interface system 104 located in the environment 114. Thus, the enhanced image of the medical field volume provides the surgeon S with instructions and suggestions for performing the medical procedure in the environment 114.
[0048] Whenever feasible, elements described in detail with reference to one embodiment, implementation or application may be optionally included in other embodiments, implementations or applications where they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment, but not described with reference to a second embodiment, the element may still be required to be included in the second embodiment. Therefore, in order to avoid unnecessary repetition in the following description, unless otherwise specifically described, unless one or more elements will make an embodiment or implementation inoperative, or unless two or more than two elements provide conflicting functions, one or more elements shown and described in association with an embodiment, implementation or application may be incorporated into other embodiments, implementations or aspects. In addition, although the subject matter of some claims may be recited in a dependent form from a specific type of claim (e.g., system, device, method, computer-readable medium), it should be understood that such a subject matter may also be required to be another type of claim (e.g., system, device, method, computer-readable medium).
[0049] As would normally occur to a person skilled in the art to which the present disclosure relates, any changes and further modifications to the described devices, systems, apparatuses, methods, and any further applications of the principles of the present disclosure are fully anticipated. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for specific examples, and it is contemplated that different sizes, dimensions, and / or ratios may be utilized to implement the concepts of the present disclosure. In order to avoid unnecessary descriptive repetitions, one or more components or actions described according to an illustrative embodiment may be used or omitted from other illustrative embodiments when applicable. For the sake of brevity, multiple iterations of these combinations will not be described separately.
[0050] Various systems and parts of systems have been described in terms of their state in three-dimensional space. As used herein, the term "position" refers to the position of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three rotational degrees of freedom - such as roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or a portion of an object in at least one translational degree of freedom, and the orientation of the object or a portion of an object in at least one rotational degree of freedom (up to six total degrees of freedom).
[0051] Although some examples described herein relate to surgical procedures or instruments, or medical procedures and medical instruments, the disclosed technology is optionally applicable to non-medical procedures and non-medical instruments. For example, the instruments, systems and methods described herein can be used for non-medical purposes (including industrial uses, general robotic uses, and sensing or manipulating non-tissue artifacts). Other example applications relate to cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, and training medical or non-medical personnel. Additional example applications include uses for the following: performing procedures on tissue removed from human or animal anatomical structures (not returning to human or animal anatomical structures), and performing procedures on human or animal corpses. In addition, these technologies can also be used for surgical and non-surgical medical treatment or diagnostic procedures.
[0052] A computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information to produce processed output information. A computer includes a logic unit that performs the mathematical or logical functions, and a memory that stores the programmed instructions, input information, and output information. The term "computer" and similar terms such as "processor" or "controller" or "control system" are similar.
[0053] Although certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it should be understood that such embodiments are only broad illustrations of the present invention rather than limitations on the scope of the present invention, and that the embodiments of the present invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may be conceived by ordinary technicians in this field.
Claims
1. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: generating stereoscopic endoscopic image data of a medical field; defining a medical field volume based on the stereoscopic endoscopic image data; projecting a three-dimensional representation of the medical field volume onto an extended reality display device in a remote volume, the three-dimensional representation comprising a stereoscopic image generated from the stereoscopic endoscopic image data and a three-dimensional scene representing the medical field; generating a visual guide in the remote volume to enhance the stereoscopic endoscopic image; mapping the visual guide from the remote volume to the medical field volume to generate an enhanced image of the medical field volume; as well as The enhanced image of the medical field volume is projected to a first display device for viewing by an operator of one or more instruments in the medical field.
2. The non-transitory machine-readable medium of claim 1, wherein: The stereoscopic endoscopic image data is generated by an endoscopic imaging device controlled by a robot-assisted manipulator.
3. The non-transitory machine-readable medium of claim 2, wherein: The medical field includes a tool controlled by the robotic-assisted manipulator.
4. The non-transitory machine-readable medium of claim 1, wherein: The medical field volume includes stereoscopic endoscopic image data visible to the patient's anatomy and instruments of the medical field.
5. The non-transitory machine-readable medium of claim 1, wherein: The medical field volume includes a user interface element.
6. The non-transitory machine-readable medium of claim 1 storing instructions that, when executed by one or more processors, further cause the one or more processors to receive audio input from an operator at an operator console.
7. The non-transitory machine-readable medium of claim 1, wherein: The extended reality display device is head-mounted.
8. The non-transitory machine-readable medium of claim 1, wherein: The remote volume is visible outside the three-dimensional representation of the medical field volume.
9. The non-transitory machine-readable medium of claim 1, wherein: The three-dimensional scene is generated by determining object depths of objects in the stereoscopic images.
10. The non-transitory machine-readable medium of claim 1, wherein: The three-dimensional scene is generated for a portion of the stereoscopic image.
11. The non-transitory machine-readable medium of claim 1, wherein: The remote volume moves with the augmented reality display device.
12. The non-transitory machine-readable medium of claim 1, wherein: The extended reality display device tracks a user's body expressions in the remote volume.
13. The non-transitory machine-readable medium of claim 1, wherein: The visual guidance is generated by at least one of a freehand gesture, eye gaze, or head movement of a user in the remote volume.
14. The non-transitory machine-readable medium of claim 1, wherein: The visual guide includes at least one of an arrow, a pointer, a ghost hand, a ghost tool, a gaze marker, a virtual anatomical model, or an annotation.
15. The non-transitory machine-readable medium of claim 1 storing instructions that, when executed by one or more processors, further cause the one or more processors to generate an audio guide.
16. The non-transitory machine-readable medium of claim 1, wherein: Mapping the visual guide includes determining a three-dimensional position of the visual guide in the medical field volume.
17. The non-transitory machine-readable medium of claim 1, wherein: Mapping the visual guide includes scaling parameters from the remote volume to the medical field volume.
18. The non-transitory machine-readable medium of claim 1 storing instructions that, when executed by one or more processors, further cause the one or more processors to adaptively change the mapping as an endoscopic imaging system moves.
19. The non-transitory machine-readable medium of claim 1 storing instructions that, when executed by one or more processors, further cause the one or more processors to project a model of a manipulator assembly from the augmented reality display device.
20. The non-transitory machine-readable medium of claim 19 storing instructions that, when executed by one or more processors, further cause the one or more processors to fuse the model of the manipulator assembly with the augmented image of the medical field volume for display on the extended reality display device.
21. The non-transitory machine-readable medium of claim 1 storing instructions that, when executed by one or more processors, further cause the one or more processors to project a model of an operator console from the augmented reality display device.
22. The non-transitory machine-readable medium of claim 1, wherein: The medical field volume can be a composite comprising a plurality of images, and the visual guide can be visible in a region of the medical field volume visible by a mobile endoscopic imaging instrument.
23. A system comprising: processor; as well as a memory including computer readable instructions stored thereon that, when executed by the processor, cause the system to: generating stereoscopic endoscopic image data of a medical field; defining a medical field volume based on the stereoscopic endoscopic image data; projecting a three-dimensional representation of the medical field volume onto an extended reality display device in a remote volume, the three-dimensional representation comprising a stereoscopic image generated from the stereoscopic endoscopic image data and a three-dimensional scene representing the medical field; generating a visual guide in the remote volume to enhance the stereoscopic endoscopic image; mapping the visual guide from the remote volume to the medical field volume to generate an enhanced image of the medical field volume; as well as The enhanced image of the medical field volume is projected to a first display device for viewing by an operator of one or more instruments in the medical field.
24. The system of claim 23, wherein: The stereoscopic endoscopic image data is generated by an endoscopic imaging device controlled by a robot-assisted manipulator.
25. The system of claim 24, wherein: The medical field includes a tool controlled by the robotic-assisted manipulator.
26. The system of claim 23, wherein: The medical field volume includes stereoscopic endoscopic image data visible to the patient's anatomy and instruments of the medical field.
27. The system of claim 23, wherein: The medical field volume includes a user interface element.
28. A method comprising: generating stereoscopic endoscopic image data of a medical field; defining a medical field volume based on the stereoscopic endoscopic image data; projecting a three-dimensional representation of the medical field volume onto an extended reality display device in a remote volume, the three-dimensional representation comprising a stereoscopic image generated from the stereoscopic endoscopic image data and a three-dimensional scene representing the medical field; generating a visual guide in the remote volume to enhance the stereoscopic endoscopic image; mapping the visual guide from the remote volume to the medical field volume to generate an enhanced image of the medical field volume; as well as The enhanced image of the medical field volume is projected to a first display device for viewing by an operator of one or more instruments in the medical field.
29. The method according to claim 28, wherein: The stereoscopic endoscopic image data is generated by an endoscopic imaging device controlled by a robot-assisted manipulator.
30. The method of claim 29, wherein: The medical field includes a tool controlled by the robotic-assisted manipulator.
31. The method of claim 28, wherein: The medical field volume includes stereoscopic endoscopic image data visible to the patient's anatomy and instruments of the medical field.
32. The method of claim 28, wherein: The medical field volume includes a user interface element.
