Surgical robot setup using an enhanced mirror display

Abstract

The present invention is entitled "SETTING UP A SURGICAL ROBOTS USING AUGMENTED REFLECTIVE MIRROR DISPLAY". Using augmented reality to assist in setting up a robotic arm in a surgical robotic system may include capturing real-time video of a user setting up a robotic arm in a surgical robotic system. Visual guidance representing a target pose of the robotic arm may be rendered onto the real-time video, thereby generating an augmented real-time video for guiding the setting up of the arm. The augmented real-time video may be displayed to the user as the user follows the visual guidance to set up the robotic arm. The captured real-time video may be continuously processed to determine whether the robotic arm has reached the target pose.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] none. Technical Field

[0003] The present invention relates generally to surgical robotic systems, and more particularly to novel and useful systems and methods for providing an augmented reality surgical robotic environment. Background Art

[0004] Minimally invasive surgery (MIS) such as laparoscopic surgery involves techniques designed to reduce tissue damage during surgical procedures. For example, laparoscopic surgery typically involves making many small incisions in the patient's body (e.g., in the abdominal cavity), and introducing one or more tools and at least one camera into the patient's body through these incisions. The surgical procedure is then performed using the introduced tools and the visualization assistance provided by the camera. In general, MIS provides a variety of beneficial effects, such as reducing patient scars, alleviating patient pain, shortening patient recovery periods, and reducing medical expenses associated with patient recovery. For example, a robotic arm can support various devices at its distal end, such as surgical end effectors, imaging devices, cannulas for providing access to a patient's body cavity and organs, and the like.

[0005] MIS can be performed using non-robotic or robotic systems. Conventional robotic systems may include a robotic arm that manipulates tools based on commands from an operator, which can provide many of the benefits of MIS while reducing the need for a surgeon. Control of such a robotic system may require a user (e.g., a surgeon or other operator) to make control inputs via one or more user interface devices, which convert the user's manipulations or commands into control of the robotic system. For example, in response to a user command, a tool driver having one or more motors can actuate one or more degrees of freedom of the surgical tool when the surgical tool is positioned at the surgical site of the patient.

[0006] Robotic surgical systems, such as those used in MIS, may have three or more degrees of freedom. This allows the robotic arm to have a nearly infinite number of poses, and as the degrees of freedom increase, the complexity of setting up the robot also increases. Medical personnel typically pose the robotic arm before starting the surgical procedure. It is important that the robotic arm is in the correct posture relative to the patient; otherwise, complications may arise during the surgical procedure.

[0007] Therefore, the setup and correct configuration of the robotic arm is a complex task. The correct posture of the robotic arm may be difficult to show to the medical staff. Similarly, even when viewing various illustrations of the same posture, it may be difficult for the staff to fully visualize the posture in real space. The spatial relationship between the staff's body, real objects in the surgical environment (e.g., tool racks, stools, and lights) and the target posture is lost.

[0008] Additionally, obstacles may be encountered when moving the surgical robot from one pose to another or to a target pose, such as other surgical robots, lights, tools, trays, or other objects typical of such environments. Again, these obstacles are difficult to visualize because the desired pose of the surgical robot is not obvious, nor is the path between the current pose and the target pose.

[0009] Furthermore, even after the medical staff has completed the setup of the robotic arm, it is not obvious whether the robotic surgical arm is actually in the correct posture. In other words, there should be an objective way to verify that the robotic surgical arm has been set to the correct posture for a given procedure, which may take into account the size and shape of the patient, the type of robotic surgical arm, the type of distal device used, the point or points of entry into the patient's body, and other considerations.

[0010] Therefore, it is desirable to develop a system that can assist in setting up the surgical robot in a reliable and verifiable manner that can be customized for different procedures, patients, and tools. Summary of the invention

[0011] In general, a system for guiding arm setup in a surgical robot system using augmented reality may include a camera and a processor (e.g., an augmented reality (AR) processor). The camera may be configured to capture a real-time video of a user setting up a robotic arm in a surgical robot system. The AR processor may be configured to receive the real-time arm setup video captured by the camera, render a virtual surgical robot arm in a target pose onto the real-time arm setup video, thereby generating an augmented real-time video for guiding arm setup, and stream the augmented real-time video to a display to visually guide the user through the arm setup to achieve the target pose.

[0012] In another aspect, the AR processor is further configured to continuously process the real-time setup video to determine whether the robotic arm is in the target pose represented by the virtual robotic arm, and trigger an indication when the robotic arm is superimposed on the virtual robotic arm in the target pose. This provides feedback on whether the task of moving the robotic arm to the target pose is completed.

[0013] It should be understood that the camera and AR processor capture, process and display the augmented surgical environment video while the user is setting up and viewing the display, so that the user can correctly set up one or more surgical robots. In this way, medical personnel can view the display while arranging the robotic arm for robotic surgery. Medical personnel will better understand the spatial relationship between the real robotic arm and other real objects, and get visual guidance for arranging the robotic arm (e.g., a virtual surgical robotic arm in a target posture). BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 and Figure 2 An embodiment of a system for assisting in setting up a surgical robotic arm is shown.

[0015] Figure 3 A virtual reality mirror is shown in one embodiment.

[0016] Figure 4 The user is shown viewing the augmented reality image through the display.

[0017] Figure 5 Indicator markings according to some embodiments are shown.

[0018] Figures 6 to 8 A method performed by an AR processor in some embodiments is shown.

[0019] Fig. 9 A system for determining a target pose in one embodiment is shown. DETAILED DESCRIPTION

[0020] Examples of various aspects and variations of the invention are described herein and shown in the accompanying drawings.The following description is not intended to limit the invention to these embodiments, but rather to enable those skilled in the art to make and use the invention.

[0021] The following description and drawings are illustrative of the present disclosure and should not be construed as limiting the present disclosure. Many specific details are described to provide a thorough understanding of the various embodiments of the present disclosure. However, in some cases, known or conventional details are not described to make the description of these embodiments of the present disclosure more concise.

[0022] References to "one embodiment" or "embodiment" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0023] "Robotic arm" or "robot arm" or "surgical robot arm" or other variations thereof are used herein to describe a movable, articulated and motorized member having multiple degrees of freedom that can hold various tools or accessories at a distal end. Typical systems include the da Vinci(r) Surgical System that can be used for minimally invasive surgical procedures (e.g., urological surgery, general laparoscopic surgical procedure, gynecological laparoscopic surgery, general non-cardiovascular thoracoscopic surgical procedure, and thoracoscopically assisted cardiotomy).

[0024] As used herein, a "real surgical environment" describes the local physical space of a setup, which may include robotic arms, mounts, operating tables, lights, trays, medical staff, rooms, and objects in the room.

[0025] As used herein, a "target pose" describes the desired pose of a surgical robotic arm. Thus, a virtual robotic arm or an abstract image thereof may be rendered onto an augmented reality real-time video in a "target pose". The AR processor may determine when the real robotic arm is in the same pose as the virtual robotic arm, or when it is within an allowed tolerance (which can be determined and defined by a person skilled in the art) or when it is in a target pose. A "target pose" may also be a series of poses, for example, more than one pose. The robotic arm may include multiple components connected at different types of joints (e.g., rotational, revolving, torsion, orthogonal, or colinear). Each pose may be described as a unique configuration of the relative or absolute position and / or orientation of each component of the robotic arm. Thus, each robotic arm may present multiple poses based on the orientation, position, and extension of the robotic components, similar to how a human model may be posed in multiple different poses by moving (for example) the head, neck, arms, legs, and / or hips. In addition, a pose may include the position of the robotic arm in the room. For example, the position of the robotic arm may be based on the (relative or absolute) position of the base attachment of the robotic arm. The pose may include the position and orientation of each component and the position of the robot arm as a whole. The position may be described as where the component is. The orientation may be described as the direction the component (or the axis of the component) is pointing and the rotation of the component.

[0026] As used herein, a "virtual robotic arm" describes a computer-generated model of a robotic arm rendered on a captured video of a user setup. A "virtual robotic arm" may be a complex 3D model of a real robotic arm. Alternatively or in addition, a "virtual robotic arm" may include visual aids such as arrows, tooltips, or other representations (such as a geometrically simplified version of a real robotic arm) relevant to providing pose information of the robotic arm.

[0027] As used herein, "real-time" video describes video data that is captured and processed while the action in the video is occurring or with the minimum delay allowed by the system software and hardware (e.g., delays caused by transmission, processing, and buffering). Similarly, as used herein, "streaming" video describes the transmission of video data to be processed in real time or to be displayed in real time.

[0028] System and Display

[0029] As in Figure 1 As generally shown in the schematic diagram of , a robotic surgical system 100 for assisting in setting one or more surgical robotic arms 110 in a surgical environment using a display 104 may include a camera 140 configured to capture a video of a user 106 setting up a robotic arm 110 mounted on an operating table 114 in a real surgical environment; an AR processor 150 configured to: receive a real-time arm setting video 102 captured by the camera; and place a virtual surgical robotic arm (e.g., Figure 3 and Figure 4 ) is rendered onto the real-time arm setting video, thereby generating a real-time enhanced surgical environment video 108 for guiding the arm setting; and the real-time enhanced video is streamed to the display 104; wherein the display is configured to display the enhanced surgical environment video.

[0030] Figure 3 A rendered augmented surgical environment video on the display 104 is shown. The user 306 is shown moving one of the real robotic arms 310. The AR processor has rendered the virtual robotic arm 311, providing the user with visual indicators to guide the matching of the real robotic arm 310 with the target pose of the virtual robotic arm. The AR processor renders the virtual robotic arm at the same anchor point / mounting point as the real robotic arm 310, giving the real and virtual robotic arms a common anchor reference point.

[0031] Figure 4The user 106 is shown manipulating a real robotic arm 110 while viewing a real-time augmented surgical environment video on the display 104. In one aspect, the augmented surgical environment can be displayed as a mirror image. For example, the system can flip or invert the video image captured by the camera along its vertical axis, thereby generating a mirror image. The mirror image can then be output to the display 104. In this way, the system can provide a natural reflection like a mirror. In another aspect, the real-time augmented surgical environment is displayed from the camera's viewpoint. Different camera viewing angles can be configured and selected, for example, based on user preferences.

[0032] Robotic Arm and Computer Vision

[0033] See now Figure 2 , the AR processor may receive and process status messages from one or more robotic arms in the real surgical environment via the network 202. Each of the one or more robotic arms may have an associated module or processor 116 having communication capabilities, programmable logic, and memory, capable of collecting and organizing data from sensors 114 (e.g., position sensors, accelerometers, etc.), and passing the data to the AR processor 150. Thus, the AR processor 150 may receive status data from real robotic arms in the real surgical environment, such as the position / orientation (rotation, pitch angle, travel) of different components of the real robotic arms or warning messages for determining whether there is a problem with any of the robotic arms.

[0034] like Figure 2 As shown, the augmented reality processor 150 may include one or more databases 154 or may access such databases via the network 202. The database may include models of different robotic arms, postures of the robotic arms, operating tables, patient data, and data structures that associate different postures of the robotic arms with different surgical procedures, operating tables, and patients. Thus, the processor may be configured to identify the real surgical robotic arm and operating table by comparing and matching objects in the surgical environment with the 3D models of the operating table and surgical robot stored in the computer memory. Configuration files corresponding to different surgical procedures using the robotic arm may be stored in the memory and selected (e.g., via a user interface) prior to or during setup.

[0035] like Figure 2 As shown, the system may include a database 154 stored in electronic storage or such database may be accessed via a network 202. The database may include one or more 3D models of robotic arms, each robotic arm being associated with one or more target poses, and a user interface configured to provide selectable options for the one or more 3D models of the robotic arms and the associated target poses.

[0036] The AR processor may also include a computer vision module 152, which is configured to perform scene reconstruction, event detection, video tracking, object recognition, 3D pose estimation, learning, indexing, motion estimation, image restoration and other known computer vision techniques based on the captured video. It can apply the above-mentioned 3D vision technology to analyze the captured video and identify surgical robot arms, operating tables, patients or human models, medical staff, other objects in the surgical environment and their orientation and position. Alternatively or in addition, the processor can identify and / or track objects (e.g., robotic arms and operating tables) with 2D markers (e.g., labels or stickers placed on the objects).

[0037] Target pose

[0038] The AR processor may be configured to process the captured video to determine whether the pose of the real surgical robotic arm is at a target pose of the virtual surgical robotic arm; and trigger an indication when the pose of the real surgical robotic arm is at the virtual robotic arm. This determination may be made based on 3D object recognition, 3D pose estimation, and other techniques that can be implemented by a person skilled in the art of computer vision to detect the pose of the real robotic arm in the captured video and then compare the pose to the target pose. In this regard, the processor may then determine whether the real robotic arm is at or within a tolerance of the target pose shown by the computer-generated virtual surgical robotic arm.

[0039] See also Fig. 9 In one aspect, the control system of the robotic arm 110 (e.g., the processor 114 and the sensor 116) can sense and calculate pose information of the robotic arm or a component of the robotic arm. The AR processor can receive pose information, such as the angle α of the component of the robotic arm relative to an axis or plane. R The AR processor can receive the posture information (such as angle α R ) and the corresponding angle α of the corresponding component of the virtual robot arm 310 relative to the axis or plane T The components of the robot arm are compared to determine an error ε. If the error ε is less than the determined tolerance, it can be determined that the component (of the robot arm) is in the target pose (as shown by the virtual robot arm) or sufficiently overlaps the target pose. Depending on the application (e.g., the type of robot arm used and the type of surgical procedure), the process of comparing posture information can be repeated for multiple components or all components of the robot arm to determine whether each component is within the error of the target pose. When making such a determination, the AR processor generates an indication (e.g., a visual indication rendered onto the real-time arm setup video) and displays the augmented real-time video on the display 104.

[0040] Additionally or alternatively, the AR processor may also determine the pose information of the robot arm based on the captured real-time arm setup video received from the camera 140. The AR processor may use suitable techniques (e.g., through computer vision and / or depth sensors) to determine the pose information of the robot arm in the captured real-time arm setup video.

[0041] Although Fig. 9 A comparison of angles is shown, but determining whether the robot arm is in a target pose may include comparing additional pose information of the robot components and combinations thereof. For example, the pose information may also include rotation orientation (e.g., for a rotation or torsion component) and stroke length (e.g., for an extension or telescoping component). The pose information may be based on position sensors, linear position sensors, displacement sensors, magnetic sensors, controller data, encoders, and other equivalent robot sensing technologies.

[0042] Indicator

[0043] For example, Figure 3 The user 306 is shown manipulating the robotic arm 310 to match the target pose of the virtual robot 311. Once the AR processor determines that the real robotic arm is at the target pose or within tolerance of the target pose, the processor can respond to the user with an indication / notification.

[0044] With regard to such instructions, Figure 5 The display 104 is shown showing virtual surgical robotic arms 511 and 512 (note that the real robotic arms are omitted here to simplify the illustration, but in practice there will be real robotic arms). In this case, the processor has detected that the posture of the real robotic arms is at the target posture of the virtual surgical robotic arms 512. In response, the processor may trigger an indication, such as a visual indication or an audio indication, that the posture of the real surgical robotic arms is at the target posture.

[0045] For example, the AR processor may visually indicate that the real robotic arm is in the target pose by changing the color, brightness, transparency, or other visual attributes of the virtual robotic arm 512 (e.g., rendering a bright outline around the virtual robotic arm).

[0046] In another embodiment, the indication may be a visual or text-based message on the display generated by the AR processor indicating that the target pose has been established. In another embodiment, the indication may be audible feedback, such as a noise (such as a beep) or speech, emitted from the speaker 560. In another embodiment, such as Figure 2As shown, the indication can be a tactile indication (e.g., vibration) generated by the vibration actuator 112 of the robot arm in a real surgical environment. In another embodiment, the indication can be any combination of the above-mentioned visual indication, sound indication, or tactile indication.

[0047] Re-reference Figure 1 In one embodiment, the AR processor may be configured to determine an area 120 (eg, a light or other robotic arm) to be cleared of all objects (eg, lights or other robotic arms). Figure 1 The processor may be configured to display an indication of the zone (e.g., by rendering an outline of the zone) into the augmented surgical environment video.

[0048] camera

[0049] exist Figure 1 , the camera is shown facing substantially the same direction as the display. Advantageously, this provides a natural perspective and direct view and line of sight 107 to visually guide the user 106 manipulating the surgical robotic arm 110. Alternatively, the camera may be strategically placed in other locations and orientations, which can be determined based on routine experimentation and iteration.

[0050] The camera 140 may be a 2D camera, such as a standard video camera or a standard digital video camera. Alternatively, the camera may be a 3D camera with depth sensing, such as infrared, laser projection, multi-lens (range imaging or stereo camera), or a combination thereof. The AR processor is configured to track the real surgical robot arm and operating table and other objects within the field of view based on the depth data received from the depth sensor and by comparing and matching the objects in the surgical environment with the 3D model of the operating table and robot in the computer memory. This helps to perform collision detection and determine the posture of the real robot arm.

[0051] When it is determined that a real object is located between the virtual robot arm and the camera, the AR processor may determine the relative depth of the real object and occlude the virtual robot arm with the real object. Figure 5 As shown in space 520 , the processor may determine that the depth of the patient / mannequin 521 is in the foreground compared to a portion of the virtual robotic arm 511 , and may occlude that portion of the virtual robotic arm accordingly.

[0052] The system may include one or more additional cameras arranged and configured to capture different perspectives of a user setting up the robotic arm in a real surgical environment. In this case, the AR processor may be configured to receive the setup video captured by the one or more additional cameras, render additional perspectives of an augmented surgical environment video for guiding the arm setup, and display one or more of the additional perspectives of the augmented surgical environment video.

[0053] Advantageously, by including multiple cameras, the AR processor can perform triangulation at multiple perspectives to determine the depth of an object in the surgical environment. In addition, the user can view the real and augmented robotic arms from multiple perspectives, thereby providing an improved understanding of the visual space. The processor may be configured to display all of the one or more additional perspectives of the augmented surgical environment video on the display. Alternatively, the processor may select which perspective to display based on, for example, the posture and / or position of the robotic arm in the real surgical environment, the position of the user, or the position of the user relative to the posture and / or position of the robotic arm. For example, the processor may employ heuristics such as "not showing the perspective in which the robotic arm is blocked", or "showing the perspective in which the robotic arm or the component of the robotic arm being adjusted is best seen", or "showing the perspective in which the robotic arm has the maximum change posture". The processor may determine and display the best perspective for the augmented surgical environment video, or provide a user interface for the user to select which perspective to display on, for example, a separate display or on the same display as the display showing the augmented video. The augmented video may be shown, for example, on a display with a touch screen interface.

[0054] Collision and path detection

[0055] See now Figure 3 , the AR processor may be configured to determine the current pose of the real robotic arm 301, and calculate a path 304 between the current pose of the real robotic arm and the target pose of the virtual robotic arm 302; detect real objects 303 in the calculated path; and project a visual indicator (e.g., a message, "X" or other symbol) on the captured surgical environment, the visual indicator showing one or more possible collisions between the real robotic arm and the real objects in the calculated path. For example, when the processor detects that the edge of the lamp will collide with the real robotic arm or may collide with the real robotic arm, the processor may render an X on the edge of the lamp.

[0056] Posture Adjustment

[0057] In one embodiment, the AR processor may be configured to adjust the saved pose of the virtual robotic arm based on the Figure 2 As shown, the processor can automatically adjust the target posture based on the patient's body shape or based on the user's input to calculate the new target posture of the virtual robot arm.

[0058] See again Figure 5 , the AR processor can be configured to determine (e.g., via computer vision) the orientation of a real surgical table 550 or other robotic mounting hardware 552, and adjust the orientation or pose (e.g., a target pose) of the virtual robotic arms (511 and 512) based on the orientation of the surgical table or other robotic mounting hardware.

[0059] method

[0060] In one embodiment, a method for using augmented reality to assist in setting up a robotic arm in a surgical robotic system may include capturing real-time video of a user setting up a robotic arm in a surgical robotic system. The method may include rendering a visual guide representing a target pose of the robotic arm onto the real-time video, thereby generating an augmented real-time video for guiding the arm setting, and displaying the augmented real-time video to the user. While the user is following the visual guide to set up the robotic arm, the method may include continuously processing the captured real-time video to determine whether the robotic arm has reached the target pose, and notifying the user when the robotic arm has been set to the target pose.

[0061] See now Figure 6 In one embodiment, the AR processor may be configured to receive 610 an arm setup video captured by a camera of a user setting up a robotic arm in a real surgical environment. The processor may render 620 a virtual surgical robotic arm in a target pose onto the arm setup video, thereby generating an augmented surgical environment video for guiding the arm setup. The processor may then stream 630 the augmented video to a display configured to display the augmented surgical environment video.

[0062] See now Figure 7 (shown as an additional Figure 6 The AR processor may also be configured to determine 710 whether the posture of the real surgical robot arm is at the target posture of the virtual surgical robot arm. When the posture of the real surgical robot arm is at the virtual robot arm or overlaps the virtual robot arm, the processor may trigger 720 an indication.

[0063] See now Figure 8 (shown as an example of an Figure 6The AR processor may also be configured to determine 810 a current pose of the real robot arm, calculate 820 a path between the current pose of the robot arm and a target pose of the virtual robot arm, detect 830 a real object in the calculated path, and project / render 850 an indicator on the real-time arm setup video. The visual indicator may include, for example, a rendered arrow and "X" or other suitable graphics to illustrate a possible collision between the robot arm and a real object in the calculated path.

[0064] In one embodiment, the processor of the system (e.g., an AR processor, a processor in a camera, a display, and a robot arm) may include a microprocessor and a memory. Each processor may include a single processor or multiple processors including a single processor core or multiple processor cores. Each processor may represent one or more general-purpose processors, such as a microprocessor, a central processing unit (CPU), etc. More specifically, each processor may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. Each processor may also be one or more special-purpose processors, such as an application-specific integrated circuit (ASIC), a cellular or baseband processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, a graphics processor, a communication processor, a cryptographic processor, a coprocessor, an embedded processor, or any other type of logic capable of processing instructions.

[0065] Modules (e.g., computer vision module 152), components, and other features (such as the algorithms or method steps described herein) may be implemented by a microprocessor, discrete hardware components, or integrated into the functionality of hardware components such as ASICs, FPGAs, DSPs, or similar devices. In addition, such features and components may be implemented as firmware or functional circuits within a hardware device.

[0066] It should be noted that although system 100 is shown as having various components of a data processing system, it is not intended to represent any particular architecture or manner of interconnecting the components; therefore, these details are not relevant to the embodiments of the present disclosure. It should also be understood that network computers, handheld computers, mobile computing devices, servers, and / or other data processing systems with fewer or possibly more components may also be used with embodiments of the present disclosure.

[0067] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulations of physical quantities.

[0068] It should be remembered, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically noted, it should be apparent from the foregoing description that discussions throughout this specification using terms such as those listed in the following claims refer to the behavior and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0069] Embodiments of the present disclosure also relate to devices for performing the operations herein. Such computer programs are stored in non-transitory computer-readable media. Machine-readable media include any mechanism for storing information in a form that can be read by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium (e.g., a read-only memory ("ROM"), a random access memory ("RAM"), a disk storage medium, an optical storage medium, a flash memory device).

[0070] The processes or methods shown in the foregoing figures may be performed by processing logic, which includes hardware (e.g., circuits, dedicated logic, etc.), software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. Although the process or method is described above according to some sequential operations, it should be understood that some of the operations may be performed in a different order. In addition, some operations may be performed in parallel rather than sequentially.

[0071] The embodiments of the present disclosure are not described with reference to any particular programming language. It should be appreciated that a variety of programming languages ​​may be used to implement the teachings of the embodiments of the present disclosure as described herein.

[0072] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments of the present disclosure. It is apparent that various modifications may be made thereto without departing from the broader spirit and scope of the present disclosure as described in the following claims. Therefore, the description and the accompanying drawings should be considered in an exemplary rather than a restrictive sense. For ease of explanation, the above description uses specific nomenclature to provide a thorough understanding of the present invention. However, it is apparent to those skilled in the art that specific details are not required in order to implement the present invention. Therefore, the above description of specific embodiments of the present invention is provided for the purpose of illustration and description. They are not intended to be exhaustive or to limit the present invention to the precise form disclosed; obviously, in view of the above teachings, many modifications and variations are possible. The embodiments are selected and described so as to best explain the principles of the present invention and its practical application, and they thereby enable other persons skilled in the art to best utilize the present invention and various embodiments with various modifications suitable for the intended specific use. It is intended that the following claims and their equivalents define the scope of the present invention.

Claims

1. A system for guiding arm setup in a surgical robotic system using augmented reality, the system comprising: a camera configured to capture real-time arm setup video of a user setting up a robotic arm in a surgical robotic system; monitor; and A processor, the processor being configured to: receiving the real-time arm setup video captured by the camera; rendering a virtual surgical robotic arm at a target pose for each of a plurality of components including the virtual robotic arm onto the real-time arm setup video, thereby generating an augmented real-time video for guiding the arm setup; as well as streaming the augmented real-time video as a mirror image to the display facing substantially the same direction as the camera to visually guide the user through setting up the arm to the target pose; as well as When each of the plurality of components of the robotic arm matches a target pose of the virtual robotic arm within a tolerance, at least one of a color change or a transparency change of the virtual robotic arm is indicated on the display.

2. The system of claim 1 , wherein the processor is further configured to continuously process the real-time arm setup video to determine whether the robotic arm is in the target pose as represented by the virtual robotic arm; and An indication is triggered when the robot arm is superimposed on the virtual robot arm in the target pose.

3. The system according to claim 1, further comprising: An electronic memory including stored therein: one or more 3D models of robotic arms, each robotic arm being associated with one or more target poses, and A user interface is configured to provide selectable options of one or more 3D models of the robotic arm and associated target poses.

4. The system of claim 1 , wherein the processor is configured to determine an orientation or position of a real surgical table or other robotic mounting hardware, and to adjust the target pose of the virtual robotic arm based on the orientation or the position of the surgical table or other robotic mounting hardware.

5. The system of claim 1, wherein the processor is further configured to determining a current pose of the robot arm; calculating a path between the current pose of the robot arm and the target pose of the virtual robot arm; Detecting real objects in the calculated path; and Visual indicators are projected into the real-time arm setup video, the visual indicators showing possible collisions between the robotic arm and the real object in the calculated path.

6. The system of claim 1, wherein the processor is configured to calculate a second target pose of the virtual robotic arm based on adjusting the saved pose of the virtual robotic arm relative to a body size of a patient.

7. The system of claim 1, wherein the processor is configured to identify the robotic arm and the surgical table by comparing and matching objects in the surgical environment with 3D models of the surgical table and the robot in computer memory.

8. The system of claim 1, wherein the camera is a 2D camera.

9. The system of claim 8, wherein the processor is configured to track a real surgical robotic arm and operating table with 2D markers.

10. The system of claim 1, wherein the camera is a 3D camera with a depth sensor.

11. The system of claim 10, wherein the processor is configured to track a real surgical robotic arm and operating table and other objects within a field of view based on depth data received from the depth sensor and by comparing and matching objects in the surgical environment with 3D models of the operating table and robot in computer memory.

12. The system of claim 1, wherein the processor is configured to receive status from a real robotic arm in a real surgical environment.

13. The system of claim 1, wherein the processor is configured to determine a relative depth of the real object and occlude the virtual robotic arm with the real object when it is determined that the real object is located between the camera and the virtual robotic arm.

14. The system according to claim 1, comprising: one or more additional cameras arranged to capture different perspectives of the user setting up the robotic arm in a real surgical environment; wherein the processor is configured to receiving one or more setup videos captured by the one or more additional cameras, rendering one or more additional augmented surgical environment videos for guiding the arm setup, and One or more of the one or more additional augmented surgical environment videos are displayed to show different perspectives of the setup of the robotic arm.

15. The system of claim 14, wherein the processor is configured to fully display the one or more additional perspectives of the augmented surgical environment video.

16. The system of claim 14, wherein the processor is configured to display an optimal viewing angle of the augmented surgical environment video.

17. The system of claim 16, wherein the processor is configured to select which of the perspectives to display based on a position of the robotic arm, a position of the user, or a position of the user relative to the position of the robotic arm in the real surgical environment.

18. A method for assisting robotic arm setup in a surgical robotic system using augmented reality, the method comprising: using a camera to capture real-time video of a user setting up a robotic arm in a surgical robotic system; rendering visual guidance representing a target pose for each of a plurality of components comprising a virtual robotic arm onto the real-time video, thereby generating an augmented real-time video for guiding setup of the arm; displaying the augmented real-time video as a mirror image to the user on a display facing substantially the same direction as the camera; while the user is following the visual guidance to set up the robotic arm, continuously processing the captured real-time video to determine whether the robotic arm has reached the target pose; and When each of the plurality of components of the robotic arm matches a target pose of the virtual robotic arm within a tolerance, an indication including at least one of a color change or a transparency change of the virtual robotic arm is triggered on the display.

Images