For positioning patients and guiding surgical robots

Abstract

According to the present disclosure, a system, method and computer-readable medium for providing guidance for positioning a body and a surgical robot are provided. An exemplary system includes: an image capture device, the image capture device being configured to capture an image of a surgical training environment and generate image data of the surgical training environment; a head-mounted display; and a computing device, the computing device being configured to receive surgical training parameters, receive the image data of the surgical training environment, detect the posture of the workbench and the surgical robot, generate a virtual training environment based on the surgical training environment, determine the posture of a user relative to the surgical training environment, generate guidance for positioning a body on the workbench or positioning the surgical robot around the body based on the surgical training parameters and the determined posture of the user, and cause the head-mounted display to display the generated guidance.

Description

Background Art

[0001] Robotic surgery enables surgeons to perform a variety of surgical procedures (such as open and / or laparoscopic surgical procedures) that cannot be performed or are less efficient using manual surgical techniques. However, in order for a surgeon to be comfortable using a surgical robot for a robotic surgical procedure, a significant amount of training and preparation is typically required. A key aspect of preparing for robotic surgery involves preparing the surgical environment, such as placing the patient on the operating table and positioning the various arms of the surgical robot around the patient. However, conventional training methods for teaching surgeons how to position a patient for a particular surgical procedure, and thereafter position a surgical robot around the patient to ensure optimal access to the surgical area, are very time consuming and, in some cases, even prevent the adoption of a robotic surgical system. Disclosed below are improved systems, devices, methods, and computer-readable media for providing guidance and / or training for positioning a patient and a surgical robot for robotic surgery. Summary of the invention

[0002] The present disclosure relates to providing guidance for positioning a patient and / or a surgical robot, and more particularly to systems, methods, and computer-readable media for generating and displaying virtual or augmented reality visual guidance to guide and / or train clinicians how to position a patient and / or one or more components of a surgical robot for various types of surgical robotic procedures.

[0003] According to an embodiment of the present disclosure, a system for providing guidance for positioning a body and a surgical robot is provided. In one aspect of the present disclosure, the system includes: an image capture device, the image capture device is configured to capture an image of a surgical training environment including a workbench and a surgical robot, and generate image data of the surgical training environment based on the captured image; a head-mounted display (HMD), the head-mounted display is worn by a user; and a computing device, the computing device includes a processor and a memory storing instructions, the instructions when executed by the processor cause the computing device to receive surgical training parameters, receive image data of the surgical training environment from the image capture device, detect the posture of the workbench and the surgical robot based on the received image data, generate a virtual training environment based on the image data of the surgical training environment, the virtual environment includes a representation of the workbench and the surgical robot, determine the posture of the user relative to the surgical training environment based on the received image data and the generated virtual training environment, generate guidance for positioning at least a portion of the body on the workbench or positioning at least a portion of the surgical robot around the body based on the surgical training parameters and the determined posture of the user wearing the HMD relative to the surgical training environment, and cause the HMD to display the generated guidance.

[0004] In another aspect of the present disclosure, the body is a mannequin and the surgical training environment additionally includes the body on a work table.

[0005] In another aspect of the present disclosure, the guiding includes commands for positioning at least a portion of the body on the table based on the surgical training parameters.

[0006] In yet another aspect of the disclosure, the command comprises a text command displayed by the HMD.

[0007] In yet another aspect of the present disclosure, the computing device causes the HMD to display an augmented reality image of a location of at least one organ of the body superimposed on the body.

[0008] In another aspect of the disclosure, the command includes an augmented reality image of a recommended location where at least a portion of the body should be placed.

[0009] In another aspect of the present disclosure, the guidance instructs the user when to place at least a portion of the body into a recommended position.

[0010] In another aspect of the present disclosure, the guide includes an augmented reality image of a location of at least one organ of a body.

[0011] In another aspect of the disclosure, the augmented reality image of the position of at least one organ of the body is based on a model of the position of the organ within the patient's body.

[0012] In another aspect of the present disclosure, the command includes an augmented reality image of a recommended location where at least a portion of the surgical robot should be placed.

[0013] In yet another aspect of the present disclosure, the virtual training environment additionally includes a representation of a virtual body.

[0014] In yet another aspect of the disclosure, an image capture device is coupled to the HMD.

[0015] In yet another aspect of the present disclosure, the image capture device is one of a plurality of image capture devices, and wherein the plurality of image capture devices are positioned at a plurality of corresponding locations around the surgical training environment.

[0016] In another aspect of the present disclosure, at least a portion of the surgical robot includes one or more of the following: a base, a first joint, a first arm connected to the first joint, a second joint connected to the first arm, a second arm connected to the second joint, and the base is connected to the second arm through the first joint, the first arm and the second joint.

[0017] In yet another aspect of the present disclosure, receiving the surgical training parameters includes loading pre-configured surgical training parameters from a memory.

[0018] In yet another aspect of the present disclosure, receiving the surgical training parameter includes receiving a selection of the surgical training parameter from an input by a user.

[0019] In yet another aspect of the present disclosure, the surgical training parameters are based on a training procedure.

[0020] In another aspect of the disclosure, the training program is selected from a database of training programs.

[0021] In yet another aspect of the present disclosure, the database of training procedures includes training procedures related to one or more of: a type of surgery, a type of patient, characteristics of the patient, and a type of medical condition being treated.

[0022] In another aspect of the present disclosure, the training program is selected via a graphical user interface.

[0023] In another aspect of the present disclosure, the surgical training parameters include one or more of: the type of surgery, the age of the patient, the weight of the patient, the size of the patient, and the medical condition of the patient.

[0024] In yet another aspect of the present disclosure, instructions for causing a computing device to determine a posture of a user wearing an HMD relative to a surgical training environment include additional instructions that, when executed by a processor, cause the computing device to analyze received image data to identify the user and one or more of a workbench and a surgical robot in the received image data, determine a position of the user relative to one or more of the workbench and the surgical robot, and generate a representation of the user in a virtual environment based on the determined position of the user relative to one or more of the workbench or the surgical robot.

[0025] In yet another aspect of the disclosure, the instructions, when executed by the processor, further cause the computing device to determine whether the user has performed an action indicated by the guidance.

[0026] In another aspect of the present disclosure, the image capture device is configured to capture additional image data of the surgical training environment, and wherein determining whether the user has performed the action indicated by the guidance includes analyzing the additional image data to detect the user's movement, and determining whether the user has performed the action indicated by the guidance based on the detected user's movement.

[0027] In yet another aspect of the disclosure, the instructions, when executed by the processor, additionally cause the computing device to determine a score based on the detected movement of the user, and cause the HMD to display the determined score.

[0028] According to an embodiment of the present disclosure, a method for providing guidance for positioning a body and a surgical robot is provided. In one aspect of the present disclosure, the method includes: receiving a selection of a surgical training parameter, obtaining image data of a surgical training environment including a workbench and a surgical robot, detecting the positions of the workbench and the surgical robot based on the obtained image data, generating a virtual training environment based on the obtained image data of the surgical training environment, the virtual training environment including representations of the workbench and the surgical robot, determining a posture of a user wearing a head mounted display device (HMD) relative to the surgical training environment based on the obtained image data and the generated virtual training environment, generating guidance for positioning at least a portion of the body on the workbench or positioning at least a portion of the surgical robot around the body based on the surgical training parameter and the determined posture of the user relative to the surgical training environment, and causing the HMD to display the generated guidance.

[0029] In another aspect of the present disclosure, the body is a mannequin and the surgical training environment additionally includes the body on a work table.

[0030] In another aspect of the present disclosure, the guiding includes commands for positioning at least a portion of the body on the table based on the surgical training parameters.

[0031] In yet another aspect of the disclosure, the command comprises a text command displayed by the HMD.

[0032] In yet another aspect of the present disclosure, the method additionally includes causing the HMD to display an augmented reality image of a location of at least one organ of the body superimposed on the body.

[0033] In another aspect of the disclosure, the command includes an augmented reality image of a recommended location where at least a portion of the body should be placed.

[0034] In another aspect of the present disclosure, the guidance instructs the user when to place at least a portion of the body into a recommended position.

[0035] In another aspect of the present disclosure, the guide includes an augmented reality image of a location of at least one organ of a body.

[0036] In another aspect of the disclosure, the augmented reality image of the position of at least one organ of the body is based on a model of the position of the organ within the patient's body.

[0037] In another aspect of the present disclosure, the command includes an augmented reality image of a recommended location where at least a portion of the surgical robot should be placed.

[0038] In yet another aspect of the present disclosure, the virtual training environment additionally includes a representation of a virtual body.

[0039] In yet another aspect of the present disclosure, surgical training parameters are based on a training procedure.

[0040] In another aspect of the disclosure, the training program is selected from a database of training programs.

[0041] In yet another aspect of the present disclosure, the database of training procedures includes training procedures related to one or more of: a type of surgery, a type of patient, characteristics of the patient, and a type of medical condition being treated.

[0042] In yet another aspect of the present disclosure, a training program is selected via a graphical user interface.

[0043] In another aspect of the present disclosure, the surgical training parameters include one or more of: the type of surgery, the age of the patient, the weight of the patient, the size of the patient, and the medical condition of the patient.

[0044] In yet another aspect of the present disclosure, determining a posture of a user wearing an HMD relative to a surgical training environment includes analyzing received image data to identify the user and one or more of a workbench and a surgical robot in the received image data, determining a position of the user relative to one or more of the workbench and the surgical robot, and generating a representation of the user in the virtual environment based on the determined position of the user relative to one or more of the workbench or the surgical robot.

[0045] In yet another aspect of the present disclosure, the method further includes determining whether the user has performed an action indicated by the guidance.

[0046] In another aspect of the present disclosure, the method further includes capturing additional image data of the surgical training environment, and determining whether the user has performed the action indicated by the guidance includes analyzing the additional image data to detect the user's movement, and determining whether the user has performed the action indicated by the guidance based on the detected user's movement.

[0047] In yet another aspect of the present disclosure, the method additionally includes determining a score based on the detected movement of the user, and causing the HMD to display the determined score.

[0048] According to an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, comprising instructions, which when executed by a computer cause the computer to receive a selection of surgical training parameters, obtain image data of a surgical training environment including a workbench and a surgical robot, detect positions of the workbench and the surgical robot based on the obtained image data, generate a virtual training environment based on the obtained image data of the surgical training environment, the virtual training environment including representations of the workbench and the surgical robot, determine a posture of a user wearing a head-mounted display device (HMD) relative to the surgical training environment based on the obtained image data and the generated virtual training environment, generate guidance for positioning at least a portion of a body on the workbench and at least a portion of the surgical robot around the body based on the surgical training parameters and the determined posture of the user relative to the surgical training environment, and cause the HMD to display the generated guidance.

[0049] In another aspect of the present disclosure, the body is a mannequin and the surgical training environment additionally includes the body on a work table.

[0050] In another aspect of the present disclosure, the guiding includes commands for positioning at least a portion of the body on the table based on the surgical training parameters.

[0051] In yet another aspect of the disclosure, the command comprises a text command displayed by the HMD.

[0052] In yet another aspect of the disclosure, the instructions, when executed by the processor, cause the computer to cause the HMD to display an augmented reality image of a location of at least one organ of a body superimposed on the body.

[0053] In another aspect of the disclosure, the command includes an augmented reality image of a recommended location where at least a portion of the body should be placed.

[0054] In another aspect of the present disclosure, the guidance instructs the user when to place at least a portion of the body into a recommended position.

[0055] In another aspect of the present disclosure, the guide includes an augmented reality image of a location of at least one organ of a body.

[0056] In another aspect of the disclosure, the augmented reality image of the position of at least one organ of the body is based on a model of the position of the organ within the patient's body.

[0057] In another aspect of the present disclosure, the command includes an augmented reality image of a recommended location where at least a portion of the surgical robot should be placed.

[0058] In yet another aspect of the present disclosure, the virtual training environment additionally includes a representation of a virtual body.

[0059] In yet another aspect of the present disclosure, surgical training parameters are based on a training procedure.

[0060] In another aspect of the disclosure, the training program is selected from a database of training programs.

[0061] In yet another aspect of the present disclosure, the database of training procedures includes training procedures related to one or more of: a type of surgery, a type of patient, characteristics of the patient, and a type of medical condition being treated.

[0062] In yet another aspect of the present disclosure, a training program is selected via a graphical user interface.

[0063] In another aspect of the present disclosure, the surgical training parameters include one or more of: the type of surgery, the age of the patient, the weight of the patient, the size of the patient, and the medical condition of the patient.

[0064] In yet another aspect of the present disclosure, determining a posture of a user wearing an HMD relative to a surgical training environment includes analyzing received image data to identify the user and one or more of a workbench and a surgical robot in the received image data, determining a position of the user relative to one or more of the workbench and the surgical robot, and generating a representation of the user in the virtual environment based on the determined position of the user relative to one or more of the workbench or the surgical robot.

[0065] In another aspect of the disclosure, the instructions, when executed by the processor, further cause the computer to determine whether the user has performed an action indicated by the guidance.

[0066] In another aspect of the present disclosure, the instructions, when executed by the processor, further cause the computer to capture additional image data of the surgical training environment, and determining whether the user has performed the action indicated by the guidance includes analyzing the additional image data to detect the user's movement, and determining whether the user has performed the action indicated by the guidance based on the detected user's movement.

[0067] In yet another aspect of the disclosure, the instructions, when executed by the processor, additionally cause the computer to determine a score based on the detected movement of the user, and cause the HMD to display the determined score.

[0068] Any of the above-described aspects and embodiments of the present disclosure may be combined without departing from the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Various aspects and features of the present disclosure are described below with reference to the drawings, in which:

[0070] Figure 1 is a schematic diagram of a system for providing guidance for positioning a patient and a surgical robot according to an embodiment of the present disclosure;

[0071] Figure 2 To form Figure 1 A block diagram of an exemplary computing device that is part of a system;

[0072] Figure 3A and 3B A flow chart illustrating an exemplary method for providing guidance for positioning a patient and a surgical robot according to an embodiment of the present disclosure;

[0073] Figure 4A It is shown that the embodiment according to the present disclosure can be formed by Figure 1 an exemplary graphical user interface displayed on a head mounted display device as part of the system; and

[0074] Figure 4B It is shown that the embodiment according to the present disclosure can be formed by Figure 1 Another exemplary graphical user interface displayed on a head-mounted display device that is part of a system. DETAILED DESCRIPTION

[0075] The present disclosure relates to systems, methods, and computer-readable media for providing guidance for positioning a patient's body (or, in some instances, a non-human body modeled based on the body of a patient or another test subject—such as a mannequin, a virtual body, etc.) and a surgical robot. More specifically, the present disclosure relates to instructing a user, such as a surgeon, nurse, technician, and / or other operating room staff (hereinafter referred to as a clinician) how to position the body on an operating room table, and how to position a surgical robot around the body to ensure that surgical tools coupled to the surgical robot optimally access a surgical site within the body. Various forms of visual and / or auditory guidance may be displayed and / or emitted by various display devices and / or speakers (including head-mounted displays, such as augmented reality helmets and / or virtual reality helmets, and projectors configured to display visual guidance on the body, the operating room table, and / or elsewhere around the operating room). Guidance may include one or more commands that instruct the clinician to move the body or any part thereof to a specific position on the operating room table, and then position the various arms of the surgical robot around the body in a particular manner. Thus, the systems, methods, and computer-readable media disclosed herein may be used during training for robotic surgery and / or during an actual robotic surgical procedure to assist in positioning a human patient and a surgical robot in an operating room.

[0076] refer to Figure 1, showing a system 100 for providing guidance for positioning a patient and a surgical robot according to an embodiment of the present disclosure. The system 100 may include a workbench 110 supporting a body B, a head-mounted display device (HMD) 120, one or more image capture devices 125a, 125b, 125c, and 125d, a surgical robot 150 including a robotic arm 155, and a computing device 200. The HMD 120 may be any head-mounted display device configured to output augmented and / or virtual reality images. The image capture devices 125a, 125b, 125c, and 125d may be any image capture device known to those skilled in the art, such as a camera, a static camera, a stereo camera, a three-dimensional camera including an RGB-D camera, a LIDAR sensor, and the like, and may be positioned around a surgical training environment. One or more of the image capture devices 125a 125b, 125c, and 125d (e.g., Figure 1 The image capture device 125d) shown may be included in or coupled to the HMD 120. Additionally or alternatively, the image capture devices 125a, 125b, 125c, and 125d may be coupled or otherwise connected to the computing device 200 via a wireless or wired connection. The image capture devices 125a, 125b, 125c, and 125d are collectively referred to as image capture devices 125 hereinafter.

[0077] In addition to the robot arm 155, the surgical robot 150 may further include a base 151, a first joint 152 coupled to the base 151, a first robotic arm such as the robotic arm 155 coupled to the first joint 152, a second joint 153 coupled to the first robotic arm 155, a second arm 154 coupled to the second joint 153, and an instrument driving unit 156 coupled to the second arm 154. Figure 1150 is shown as a single robotic assembly, but multiple robotic assemblies may be included in the surgical training environment, and those skilled in the art will recognize that the methods described below may be applied to surgical robots 150 having single and / or multiple robotic assemblies each including at least one base 151, robotic arms 154 and 155, joints 152 and 153, and instrument drive units 156 without departing from the scope of the present disclosure. Body B may be a human body such as a patient's body, a non-human body such as a mannequin or other simulated body, and / or a virtual body, which in some cases is modeled based on the human body of a patient or test subject. For example, in an embodiment in which body B is a virtual body, body B may be generated based on parameters of a specific patient and / or test subject, and may be stored in the computing device 200 to be loaded during the training process. In an embodiment in which body B is a virtual body, body B will not be physically present on the workbench 110 in the surgical training environment, but will be displayed in the virtual training environment, as described below. Additionally, in embodiments where body B is a virtual body, workbench 110 may also be a virtual workbench generated by computing device 200 and / or HMD 120 and displayed via HMD 120 .

[0078] The computing device 200 may be any computing device known to those skilled in the art that can be configured for use during training of robotic surgery. For example, the computing device 200 may be a desktop computer, a laptop computer, a server and terminal configuration, and / or a control computer for the surgical robot 150, etc. In some embodiments, the computing device 200 may be included in the HMD 120. As further described below, the system 100 may be used during training of robotic surgery, such as training on how to position the body B on the table 110 and how to position the surgical robot 150 around the body B.

[0079] Now go to Figure 2 , showing the formation of an embodiment according to the present disclosure Figure 1 1 is a schematic diagram of a computing device 200 that is a portion of system 100. Computing device 200 includes memory 202, processor 204, input interface 206, communication interface 208, and output interface 210. Memory 202 stores database 240 and application 280. Application 280 includes instructions that, when executed by processor 204, cause computing device 200 to perform various functions, as described below. Application 280 additionally includes graphical user interface (GUI) instructions 285 that, when executed by processor 204, cause computing device 200 to generate one or more GUIs ( Figure 2 4. The database 240 stores various training programs including various surgical training parameters.

[0080] The training program is a set of commands designed to instruct the user to prepare the operating environment for robotic surgery, such as simulated by the surgical training environment. For example, the training program may include commands that instruct the user to place and / or adjust the position of body B on platform 110, place and / or adjust the position of surgical robot 150 relative to body B and / or platform 110, and / or adjust the posture of surgical robot 150 relative to body B. The training program may include various steps and / or stages, each with its own set of commands. For example, each step and / or stage of the training program may include one or more commands that instruct the user to perform different actions to prepare the operating environment for robotic surgery. The first step or stage may be directed to positioning body B on table 110, the second step or stage may be directed to adjusting the position of body B on table 110 to ensure optimal access to the surgical site, the third step or stage may be directed to placing surgical robot 150 around body B and / or table 110, and the fourth step or stage may be directed to adjusting the posture of surgical robot 150 relative to body B.

[0081] As used herein, the term "pose" is defined as the position and orientation of an object. For example, the pose of the surgical robot 150 refers to the position of the surgical robot 150 within the surgical training environment, the orientation of the surgical robot 150 relative to the surgical training environment and other objects in the surgical training environment, and / or the configuration of one or more of the base 151, the robot arms 154 and 155, the joints 152 and 153, and the instrument drive unit 156. Similarly, the poses of other objects in the surgical training environment (such as the workbench 110 and the body B, etc.) can also be determined.

[0082] In some embodiments, multiple users may be trained simultaneously to perform different tasks to prepare the operating environment for robotic surgery. Each user may wear an HMD 120 configured to generate and display guidance for the corresponding portion of their training program. In such embodiments, the training program may include commands for each individual user to instruct each user to perform the various steps and / or tasks assigned to them. The training program may additionally sequence the commands for each user to teach each user how to perform their corresponding tasks and the order in which each task should be performed when one user's task depends on another user completing his or her task first. In other embodiments, some or all of the users may not wear an HMD 120, and the guidance may be displayed instead by a projector that displays the guidance directly to body B, workbench 110, and / or elsewhere around the surgical training environment.

[0083] The surgical training parameters include various types of surgeries, types of patients, characteristics of the patients, types of medical conditions being treated by the surgery, locations of surgical sites within the patient's body, and / or recommended locations of surgical ports on the patient's body for accessing the surgical sites. In some embodiments, one or more of the surgical training parameters may depend on and / or be derived from another surgical parameter. For example, if the user selects "appendectomy" as the type of surgery, the location of the surgical site may be derived as "lower abdomen". Similarly, the recommended location of the surgical port may be determined based on the type of surgery selected and the location of the surgical site. Similarly, the recommended posture of the surgical robot 150 may be determined based on the location of the surgical site and / or the recommended location of the surgical port, and thus the command instructing the user how to position and / or adjust the posture of the surgical robot 150 may depend on and / or be determined based on the location of the surgical site and / or the recommended location of the surgical port on the body B. The characteristics of the patient include height, weight, body shape, pre-existing medical conditions, and / or surgeries previously performed on the patient, and / or other attributes of the patient. In an embodiment, the training program and / or command may be based on the surgical training parameters. Thus, as described further below, the guidance displayed by HMD 120 may include a visual representation of a surgical site and / or a recommended location for a surgical port.

[0084] The memory 202 may include any non-transitory computer-readable storage medium for storing data and / or software that can be executed by the processor 204 and control the operation of the computing device 200. In one embodiment, the memory 202 may include one or more solid-state storage devices, such as flash memory chips. Alternatively or in addition to one or more solid-state storage devices, the memory 202 may also include a large-capacity storage controller ( Figure 2 Not shown) and the communication bus ( Figure 2 200 ). The computer readable storage medium may include one or more mass storage devices (not shown) connected to the processor 204. Although the description of computer readable media included herein refers to solid-state storage devices, it should be understood by those skilled in the art that computer readable storage media can be any available media that can be accessed by the processor 204. That is, computer readable storage media can include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. For example, computer readable storage media can include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-ray or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store the desired information and can be accessed by the computing device 200.

[0085] The input interface 206 may be a mouse, keyboard or other handheld controller, foot pedal, touch screen, voice interface and / or any other device or interface through which a user can interact with the computing device 200. The communication interface 208 may be configured to connect to a network, such as a local area network (LAN) composed of a wired network and / or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network and / or the Internet. The output interface 210 may be a screen or other display device that can be used to output images or data by the computing device 200.

[0086] refer to Figure 3A and 3B , a flowchart of an exemplary method 300 for providing guidance for positioning a body and a surgical robot according to an embodiment of the present disclosure is shown. In some embodiments, some or all of the steps of method 300 may be performed in a surgical operating environment such as an operating room before the start of the surgical procedure. In other embodiments, some or all of the steps of method 300 may be performed in a teaching or surgical training environment such as a training facility for teaching courses. Described below are various examples of using various components of system 100 to perform the steps of method 300. However, those skilled in the art will recognize that some or all of these steps may be performed in an order or sequence different from that described, repeated, omitted, or by using different components, without departing from the scope of the present disclosure. In particular, although the examples provided below describe the use of HMD 120 as a display device, those skilled in the art will recognize that, as described above, various other display devices including one or more projectors may also be used as display devices instead of HMD 120 or in addition to HMD 120, without departing from the scope of the present disclosure.

[0087] Starting from step S302, at least one image capture device 125 captures at least one image of the surgical training environment. As described above, the surgical training environment includes the workbench 110 and the surgical robot 150. In some embodiments, the surgical training environment may further include a body B on the workbench 110. Then, in step S304, the image capture device 125 generates image data of the surgical training environment based on the captured image.

[0088] At step S306, the computing device 200 receives at least one surgical training parameter. The surgical training parameter may be based on a training program. As described above, the database 240 stores various training programs, including various surgical training parameters, including various types of surgeries, types of patients, characteristics of patients, and / or types of medical conditions being treated by surgery. In an embodiment, a user such as a clinician may select a training program and / or a specific surgical training parameter via a GUI (e.g., a GUI generated based on GUI instructions 285), and then provide the training program and / or surgical training parameter to the computing device 200.

[0089] Thereafter, or simultaneously therewith, at step S308, the computing device 200 receives image data of the surgical training environment from the image capture device 125. In some embodiments, the image capture device 125 does not generate image data of the surgical training environment, but instead sends the captured image to the computing device 200, which then generates image data of the surgical training environment based on the captured image. In such embodiments, what is received at step S308 is the captured image, not the image data of the surgical training environment.

[0090] Next, in step S310, the computing device 200 processes the image data to detect the pose of the workbench 110 and / or the surgical robot 150 based on the image data. For example, the computing device 200 may perform various image processing and / or modeling techniques to identify various objects in the image data, including the workbench 110 and the surgical robot 150. Then, in step S312, the computing device 200 generates a virtual training environment based on the image data of the surgical training environment and the detected pose of the workbench 110 and / or the surgical robot 150. In some examples, the virtual training environment may include a virtual representation of the workbench 110 and / or the surgical robot 150.

[0091] Then, in step S314, the computing device 200 determines the posture of the user relative to the surgical training environment. The determination may be based on the received image data and / or the generated virtual training environment. For example, the computing device 200 may utilize various image processing and / or object detection algorithms known to those skilled in the art to further process and analyze the image data received from the image capture device 125 to identify the user in the image data. In an embodiment, the user is identified in the image data relative to the workbench 110 and / or the surgical robot 150. Then, the computing device 200 determines the posture of the user relative to the workbench 110 and / or the surgical robot 150, and generates a representation of the user in the virtual training environment based on the determined posture of the user relative to the workbench 110 and / or the surgical robot 150. The computing device 200 may further enhance the virtual training environment based on the determined posture of the user, thereby including a representation of the user in the virtual training environment.

[0092] Thereafter, in step S316, the computing device 200 generates guidance for positioning body B on the workbench 110 based on the surgical training parameters received in step S306 and the user's posture relative to the surgical training environment determined in step S314. The guidance includes at least one command for positioning body B on the workbench 110. For example, the guidance may include text commands, graphics or other visual commands, such as virtual and / or augmented reality images, and / or auditory commands. In an embodiment, the guidance may include commands that instruct the user to perform specific actions in order to reposition one or more parts of body B to a recommended position. For example, the guidance may include commands displayed via an augmented reality image of a recommended position where body B should be placed. In some embodiments, the guidance may include multiple commands, and the commands may be provided in an ordered order.

[0093] The guidance may additionally include a virtual and / or augmented reality image of body B on table 110 and the location of one or more organs or other internal structures within body B. The location of the one or more organs or other internal structures may be based on a model of the location of the organs and / or internal structures within the body, and / or a scan of an actual patient's body. In some embodiments, the virtual and / or augmented reality image of the location of the one or more organs or other internal structures within body B includes an image of the skin of body B "peeled back" to reveal the one or more organs or other internal structures.

[0094] In an embodiment, the guidance may be based on the surgical training parameters and / or training program received in step S306 and the virtual training environment generated in step S310. For example, the computing device 200 may analyze the posture of the body B and / or the surgical robot 150, and the posture of the user relative to the body B and the surgical robot 150, and generate guidance based on the commands included in the training program, such as modified based on the specific posture of the body B and the surgical robot 150 in the surgical training environment.

[0095] The computing device 200 generates additional guidance for positioning the surgical robot 150 around the body B on the workbench 110 and / or adjusting the posture of the surgical robot 150 to a recommended posture based on the surgical training parameters received in step S306 and the user's posture relative to the surgical training environment determined in step S314. Similar to the guidance for positioning the body B on the workbench 110 as described above, the guidance for positioning the surgical robot 150 around the body B and / or adjusting the posture of the surgical robot 150 may include various commands that instruct the user to perform one or more actions. For example, the guidance may include text commands, graphics or other visual commands, such as virtual and / or augmented reality images, and / or auditory commands. In an embodiment, the guidance may include commands that are displayed via an augmented reality image of a recommended position to which the surgical robot 150 should be placed and / or a recommended posture to which the surgical robot 150 should be adjusted.

[0096] Then, in step S318, computing device 200 causes HMD 120 to display the generated guide. Computing device 200 may additionally cause HMD 120 to display an augmented reality image of the location of one or more organs or other internal structures of body B. The augmented reality image may be superimposed on body B.

[0097] After displaying the guidance, at step S320, computing device 200 receives additional image data of the surgical training environment from image capture device 125. For example, as the user acts on the displayed guidance by performing one or more actions required to satisfy the command, image capture device 125 may capture at least one additional image and process the additional image to generate additional image data, which is then sent to computing device 200.

[0098] Then, in step S322, the computing device 200 processes and / or analyzes the additional image data to identify the actions performed by the user, and determines whether the user has performed all actions necessary to satisfy the commands included in the guidance. For example, the action may be a movement performed by the user to move or reposition all or one or more parts of the surgical robot 150 (e.g., the robot arm 155) and / or all or one or more parts of the body B (e.g., arms, legs, torso, etc.). For example, the computing device 200 may analyze the additional image data to detect the user's movement relative to the surgical training environment, and in particular relative to the workbench 110 and / or the surgical robot 150, such as the movement of the user's arms and / or hands, to determine whether the user has performed a specific action indicated by the guidance. If the computing device 200 determines that the user has performed the action ("yes" at S322), then the process proceeds to step S324. Alternatively, if the computing device 200 cannot detect the user's movement ("no" at S322), then the process returns to step S318.

[0099] In step S324, the computing device 200 determines whether the action performed by the user and identified in S322 corresponds to the guidance displayed in S318. For example, the computing device 200 may then determine whether the user has performed the action indicated by the guidance displayed in step S318 to satisfy the command based on the detected movement of the user. The computing device 200 may additionally determine a training score evaluated based on the user's performance of the action indicated by the command. For example, the training score evaluation may be based on the accuracy, speed, consistency, etc. of the user's performance of the action indicated by the command. If the computing device 200 determines that the action performed by the user does not correspond to the guidance ("No" in step S324), then the process returns to step S318.

[0100] Alternatively, if the computing device 200 determines that the action performed by the user corresponds to the guidance ("Yes" at step S324), the process proceeds to step S326, where the computing device 200 generates and causes the HMD 120 to display updated guidance indicating whether the user has satisfied the command. For example, the computing device may cause the HMD 120 to display an augmented reality image indicating the status of the command, such as whether the command has been satisfied, the accuracy of the user's action satisfying the command, and / or a training score evaluated based on the user's execution of the action indicated by the command.

[0101] Thereafter, at step S328, the computing device 200 determines whether the training program has been completed. For example, if all actions required by the commands included in the guide have been performed, the computing device 200 may determine that the training program has been completed. If the computing device 200 determines that the training program has been completed ("Yes" at step S328), the computing device 200 causes the HMD 120 to display the user's final score, and then the process ends. Alternatively, if the computing device 200 determines that the training program has not been completed ("No" at step S328), the process proceeds to step S330.

[0102] In step S330, the computing device 200 generates new guidance for positioning the body B on the workbench 110, positioning the surgical robot 150 around the surgical training environment, and / or adjusting the posture of the surgical robot 150. For example, the new guidance may include additional actions performed by the user after the actions required by the commands included in the guidance generated in step S316 have been performed. In an embodiment, as described above, the new guidance may include one or more commands that instruct the user to perform the next step or stage of the training procedure, as described above. For example, the new guidance may include one or more commands that instruct the user to reposition different parts of the body B and / or different parts of the surgical robot 150.

[0103] Thereafter, the process proceeds to step S332, where the computing device 200 causes the HMD 120 to display the generated new guide. The computing device 200 additionally causes the HMD 120 to also display an augmented reality image of the location of one or more organs or other internal structures of the body B.

[0104] After displaying the guidance, in step S334, the computing device 200 receives additional image data of the surgical training environment from the image capture device 125. Then, in step S336, the computing device 200 processes and / or analyzes the additional image data to identify the action performed by the user, and determines whether the action performed by the user satisfies the command included in the new guidance. For example, the computing device 200 may analyze the additional image data to detect the user's movement relative to the surgical training environment, and in particular relative to the workbench 110 and / or the surgical robot 150, such as the movement of the user's arm and / or hand. If the computing device 200 determines that the user has performed an action ("yes" in step S336), then the process proceeds to step S338. Alternatively, if the computing device cannot detect the user's movement ("no" in step S336), then the process returns to step S332.

[0105] In step S338, the computing device 200 determines whether the action performed by the user and identified in step S334 corresponds to the command included in the new guide. For example, the computing device 200 may determine whether the user has performed the action indicated by the guide to satisfy the command based on the detected movement of the user. The computing device 200 further determines a training score evaluated based on the user's execution of the action indicated by the command. If the computing device 200 determines that the action performed by the user does not correspond to the guide ("No" in step S338), then the process returns to step S332. Alternatively, if the computing device 200 determines that the action performed by the user and identified in step S334 corresponds to the new guide displayed in step S332 ("Yes" in step S338), then the process returns to step S328, where the computing device 200 again determines whether the training program has been completed, and the method 300 continues as described above with reference to step S328.

[0106] Although the above description of method 300 with respect to FIG. 3 refers to functions performed by computing device 200, those skilled in the art will appreciate that such functions may be performed by computing device 200 based on the execution of one or more applications (such as application 280) and / or based on dedicated hardware and / or other software included in computing device 200. Therefore, although described as being performed by computing device 200, the description of method 300 should not be construed as being limited to hardware embodiments.

[0107] Now turn to Figure 4A and 4B, showing an exemplary graphical user interface (GUI) 400 according to an embodiment of the present disclosure, which may be displayed by the HMD 120. The GUI 400 may be displayed by the HMD 120 at various points during the surgical preparation and / or training process, as described above with reference to Figure 3A and 3B The process of method 300 is described. Figure 4A In the embodiment shown in , the workbench 110 and the surgical robot 150 are physical objects visible to the user in the surgical training environment. Body B is shown lying on the workbench 110. In this example, body B is a virtual body, and the guide instructs the user to move the left arm of body B three inches in a specific direction. The position to which the left arm of body B should be moved is indicated by a ghost overlay 410 displayed on body B. A direction indicator (such as an arrow 420) shows the direction to which the left arm of body B should be moved. The command is also displayed as a text instruction 430. In addition, an augmented reality image of one or more organs 440 is shown as overlaid and / or superimposed on body B. In addition to the augmented reality images of one or more organs 440, the GUI 400 may similarly include a graphical representation of the surgical site (not shown) and / or a graphical representation of a recommended location for a surgical port (not shown), as further described in commonly owned U.S. patent application Ser. No. 62 / 615,481 (Attorney Docket No. C00014520.USP1(203-11482)), filed by Covidien LP on Jan. 10, 2018, the entire contents of which are incorporated herein by reference.

[0108] After the user moves the left arm of body B to the indicated position, the GUI 400 may be updated, such as Figure 4B . The GUI 400 may then show a confirmation image 450 indicating that body B has been correctly positioned. An indicator 455 may also be shown to notify the user that the action that satisfies the command has been successfully performed. The GUI 400 may also show the next command in the training program, for example, a command to reposition the surgical robot 150. The GUI 400 may show a ghost image 415 of the surgical robot 150 in the position and pose where the surgical robot 150 should be positioned. Direction indicators such as arrows 425 may also be shown to indicate the direction to which the surgical robot 150 and / or the robotic arm 155 should be moved. The text instructions 430 may also be updated based on the new command.

[0109] Detailed embodiments of the device, the system incorporating such a device, and the method of using the device are described herein. However, these detailed embodiments are only examples of the present disclosure, which can be implemented in various forms. Therefore, the specific structural and functional details disclosed herein should not be understood as limiting, but should be understood as merely serving as the basis for the claims and as a representative basis for allowing those skilled in the art to use the present disclosure differently with appropriate detailed structures.

Claims

1. A system for providing guidance for positioning a virtual body and a surgical robot, the system comprising: An image capture device, the image capture device being configured to: capturing images of a surgical training environment including a workbench and a surgical robot; and generating image data of the surgical training environment based on the captured images; a head mounted display (HMD), the head mounted display being worn by a user; and A computing device comprising a processor and a memory storing instructions which, when executed by the processor, cause the computing device to: receiving surgical training parameters; receiving the image data of the surgical training environment from the image capture device; Detecting the positions and postures of the workbench and the surgical robot based on the received image data; generating a virtual training environment based on the image data of the surgical training environment, the virtual environment including the workbench, a ghosted image of the surgical robot, and a representation of the virtual body; Determining a posture of the user relative to the surgical training environment based on the received image data and the generated virtual training environment; generating guidance for at least one of positioning at least a portion of the virtual body on the workbench or positioning at least a portion of the surgical robot around the virtual body based on the surgical training parameters and the determined pose of the user wearing the HMD relative to the surgical training environment; and The HMD is caused to display the generated guidance. 2 . The system of claim 1 , wherein the guidance comprises commands for positioning at least a portion of the virtual body on the workbench based on the surgical training parameters.

3. The system of claim 2, wherein the command comprises a text command displayed by the HMD. 4 . The system of claim 3 , wherein the computing device causes the HMD to display an augmented reality image of a position of at least one organ of the virtual body superimposed on the virtual body.

5. The system of claim 2, wherein the command includes an augmented reality image of a recommended location where at least a portion of the virtual body should be placed.

6. The system of claim 5, wherein the guidance instructs the user when to place at least a portion of the virtual body into the recommended position.

7. The system of claim 2, wherein the guidance comprises an augmented reality image of a location of at least one organ of the virtual body.

8. The system of claim 7, wherein the augmented reality image of the position of the at least one organ of the virtual body is based on a model of the position of the organ within the patient's body.

9. The system of claim 2, wherein the command includes an augmented reality image of a recommended location where at least a portion of the surgical robot should be placed.

10. The system of claim 1, wherein the image capture device is coupled to the HMD.

11. The system of claim 1, wherein the image capture device is one of a plurality of image capture devices, and wherein the plurality of image capture devices are positioned at a plurality of corresponding locations around the surgical training environment.

12. The system of claim 1, wherein the at least a portion of the surgical robot comprises one or more of: Pedestal; First joint; a first arm coupled to the first joint; a second joint coupled to the first arm; a second arm coupled to the second joint, Wherein the base is coupled to the second arm via the first joint, the first arm, and the second joint.

13. The system of claim 1, wherein receiving the surgical training parameters comprises loading preconfigured surgical training parameters from the memory.

14. The system of claim 1, wherein receiving the surgical training parameter comprises receiving a selection of a surgical training parameter from an input by a user.

15. The system of claim 1, wherein the surgical training parameters are based on a training procedure.

16. The system of claim 1, wherein the surgical training parameters include one or more of the following: Type of surgery; The patient's age; the patient's weight; The patient's body size; and The patient's medical condition.

17. The system of claim 1, wherein the instructions for causing the computing device to determine a pose of the user wearing the HMD relative to the surgical training environment include further instructions that, when executed by the processor, cause the computing device to: analyzing the received image data to identify the user and one or more of the table and the surgical robot in the received image data; determining a position of the user relative to one or more of the table and the surgical robot; and A representation of the user in the virtual environment is generated based on the determined position of the user relative to one or more of the work table or the surgical robot.

18. The system of claim 1, wherein the instructions, when executed by the processor, additionally cause the computing device to determine whether the user has performed an action indicated by the guidance.

Images